传统文件上传的痛点

在我们的日常开发工作中，经常会遇到这样的文件上传难题：

• 用户上传几个G的视频文件，网络中断导致上传失败，需要重新开始
• 大文件上传占用服务器大量带宽，影响其他用户访问
• 相同文件重复上传，浪费存储空间和带宽
• 上传进度无法实时显示，用户体验差
• 服务器内存被大量上传请求占满，导致服务不稳定

传统的单文件上传方式在面对大文件时显得力不从心。今天我们就来聊聊如何构建一个高效的大文件上传系统。

解决方案核心思路

1. 文件分片上传

将大文件切分成多个小片段，分别上传，降低单次请求的压力。

2. 断点续传

记录上传进度，网络中断后可以从断点继续上传，避免重新上传。

3. MD5校验秒传

通过MD5校验判断文件是否已存在，实现秒传功能。

4. 并发控制

合理控制并发上传的分片数量，平衡上传效率和服务器压力。

核心实现方案

1. 文件分片处理

@Servicepublicclass FileChunkService {        public List<FileChunk> splitFile(MultipartFile file, int chunkSize) {        List<FileChunk> chunks = new ArrayList<>();        long fileSize = file.getSize();        int chunkCount = (int) Math.ceil((double) fileSize / chunkSize);                try {            InputStream inputStream = file.getInputStream();            byte[] buffer = newbyte[chunkSize];                        for (int i = 0; i < chunkCount; i++) {                int bytesRead = inputStream.read(buffer);                if (bytesRead == -1) break;                                byte[] chunkData = Arrays.copyOf(buffer, bytesRead);                FileChunk chunk = new FileChunk();                chunk.setIndex(i);                chunk.setData(chunkData);                chunk.setTotalChunks(chunkCount);                chunk.setSize(bytesRead);                                chunks.add(chunk);            }        } catch (IOException e) {            thrownew RuntimeException("文件分片失败", e);        }                return chunks;    }}

2. MD5校验与秒传

@Servicepublicclass FileMd5Service {        public String calculateFileMd5(byte[] fileData) {        try {            MessageDigest md = MessageDigest.getInstance("MD5");            byte[] hashBytes = md.digest(fileData);            StringBuilder sb = new StringBuilder();            for (byte b : hashBytes) {                sb.append(String.format("x", b));            }            return sb.toString();        } catch (NoSuchAlgorithmException e) {            thrownew RuntimeException("MD5算法不可用", e);        }    }        public boolean isFileExists(String md5) {        // 检查文件是否已存在于数据库        return fileRepository.existsByMd5(md5);    }        public boolean isChunkExists(String md5, int chunkIndex) {        // 检查分片是否已存在        return fileChunkRepository.existsByMd5AndChunkIndex(md5, chunkIndex);    }}

3. 上传进度管理

@Servicepublicclass UploadProgressService {        privatefinal Map<String, UploadProgress> progressMap = new ConcurrentHashMap<>();        public void updateProgress(String uploadId, int currentChunk, int totalChunks) {        UploadProgress progress = progressMap.computeIfAbsent(uploadId, k -> new UploadProgress());        progress.setUploadId(uploadId);        progress.setCurrentChunk(currentChunk);        progress.setTotalChunks(totalChunks);        progress.setPercentage((currentChunk * 100) / totalChunks);        progress.setLastUpdateTime(LocalDateTime.now());    }        public UploadProgress getProgress(String uploadId) {        return progressMap.get(uploadId);    }        public void removeProgress(String uploadId) {        progressMap.remove(uploadId);    }}

4. 分片上传接口

@RestController@RequestMapping("/api/upload")publicclass FileUploadController {        @Autowired    private FileChunkService fileChunkService;        @Autowired    private FileMd5Service fileMd5Service;        @Autowired    private UploadProgressService uploadProgressService;        @PostMapping("/chunk")    public ResponseEntity<UploadResponse> uploadChunk(            @RequestParam("file") MultipartFile file,            @RequestParam("md5") String fileMd5,            @RequestParam("chunkIndex") int chunkIndex,            @RequestParam("totalChunks") int totalChunks) {                // 1. 检查是否已存在该分片        if (fileMd5Service.isChunkExists(fileMd5, chunkIndex)) {            // 分片已存在，跳过上传            uploadProgressService.updateProgress(fileMd5, chunkIndex + 1, totalChunks);            return ResponseEntity.ok(new UploadResponse("SUCCESS", "分片已存在"));        }                // 2. 保存分片        FileChunk chunk = new FileChunk();        chunk.setMd5(fileMd5);        chunk.setChunkIndex(chunkIndex);        chunk.setTotalChunks(totalChunks);        chunk.setData(file.getBytes());        chunk.setFileSize(file.getSize());                fileChunkRepository.save(chunk);                // 3. 更新上传进度        uploadProgressService.updateProgress(fileMd5, chunkIndex + 1, totalChunks);                return ResponseEntity.ok(new UploadResponse("SUCCESS", "分片上传成功"));    }        @PostMapping("/complete")    public ResponseEntity<UploadResponse> completeUpload(            @RequestParam("md5") String fileMd5,            @RequestParam("fileName") String fileName,            @RequestParam("fileSize") long fileSize) {                // 1. 检查所有分片是否上传完成        int uploadedChunks = fileChunkRepository.countByMd5(fileMd5);        Optional<FileChunk> firstChunk = fileChunkRepository.findFirstByMd5(fileMd5);                if (firstChunk.isPresent() && uploadedChunks == firstChunk.get().getTotalChunks()) {            // 2. 合并分片            mergeChunks(fileMd5, fileName);                        // 3. 记录文件信息            FileInfo fileInfo = new FileInfo();            fileInfo.setMd5(fileMd5);            fileInfo.setFileName(fileName);            fileInfo.setFileSize(fileSize);            fileInfo.setFilePath(generateFilePath(fileMd5, fileName));            fileInfo.setUploadTime(LocalDateTime.now());                        fileRepository.save(fileInfo);                        // 4. 清理临时分片            cleanupTempChunks(fileMd5);                        // 5. 清理进度信息            uploadProgressService.removeProgress(fileMd5);                        return ResponseEntity.ok(new UploadResponse("SUCCESS", "文件合并完成"));        } else {            return ResponseEntity.badRequest()                    .body(new UploadResponse("ERROR", "分片上传不完整"));        }    }}

前端配合实现

1. 文件分片上传

// 前端文件分片处理function uploadFile(file) {    const chunkSize = 2 * 1024 * 1024; // 2MB    const chunks = [];    let start = 0;        // 计算文件MD5    const fileReader = new FileReader();    fileReader.onload = function(e) {        const md5 = SparkMD5.ArrayBuffer.hash(e.target.result);                // 检查是否秒传        checkFileExists(md5).then(exists => {            if (exists) {                console.log('文件已存在，秒传');                return;            }                        // 分片上传            while (start < file.size) {                const chunk = file.slice(start, start + chunkSize);                chunks.push({                    index: chunks.length,                    data: chunk                });                start += chunkSize;            }                        uploadChunks(chunks, md5);        });    };        fileReader.readAsArrayBuffer(file);}

2. 上传进度展示

function uploadChunks(chunks, fileMd5) {    let uploadedChunks = 0;        // 并发上传分片，限制并发数    const concurrentLimit = 3;    const uploadingQueue = [...chunks];        const uploadNext = () => {        if (uploadingQueue.length === 0) {            // 所有分片上传完成，合并文件            completeUpload(fileMd5);            return;        }                const chunk = uploadingQueue.shift();        const formData = new FormData();        formData.append('file', chunk.data);        formData.append('md5', fileMd5);        formData.append('chunkIndex', chunk.index);        formData.append('totalChunks', chunks.length);                fetch('/api/upload/chunk', {            method: 'POST',            body: formData        }).then(response => {            uploadedChunks++;            const progress = (uploadedChunks / chunks.length) * 100;            updateProgressBar(progress);        }).finally(() => {            uploadNext(); // 继续上传下一个分片        });    };        // 启动并发上传    for (let i = 0; i < concurrentLimit && i < chunks.length; i++) {        uploadNext();    }}

高级特性实现

1. 断点续传

@PostMapping("/resume-check")public ResponseEntity<ResumeCheckResponse> checkResume(        @RequestParam("md5") String fileMd5,        @RequestParam("totalChunks") int totalChunks) {        // 检查已上传的分片    List<Integer> uploadedChunks = fileChunkRepository.findUploadedChunkIndexes(fileMd5);        ResumeCheckResponse response = new ResumeCheckResponse();    response.setNeedUploadChunks(findMissingChunks(uploadedChunks, totalChunks));    response.setUploadProgress(uploadedChunks.size() * 100 / totalChunks);        return ResponseEntity.ok(response);}

2. 并发控制

@Servicepublic class ChunkUploadThrottler {        private final Semaphore semaphore = new Semaphore(10); // 限制并发数        public void acquire() throws InterruptedException {        semaphore.acquire();    }        public void release() {        semaphore.release();    }}

3. 文件合并优化

private void mergeChunks(String fileMd5, String fileName) {    try {        List<FileChunk> chunks = fileChunkRepository.findByMd5OrderByChunkIndex(fileMd5);                String filePath = generateFilePath(fileMd5, fileName);        Path outputPath = Paths.get(filePath);                try (FileChannel outputChannel = FileChannel.open(outputPath,                 StandardOpenOption.CREATE, StandardOpenOption.WRITE)) {                        for (FileChunk chunk : chunks) {                ByteBuffer buffer = ByteBuffer.wrap(chunk.getData());                outputChannel.write(buffer);            }        }    } catch (IOException e) {        thrownew RuntimeException("文件合并失败", e);    }}

性能优化策略

1. 内存优化

• 使用流式处理，避免将整个文件加载到内存
• 合理设置分片大小，平衡内存使用和网络效率

2. 存储优化

• 及时清理已完成合并的临时分片
• 使用对象存储服务存储最终文件

3. 网络优化

• 合理设置并发上传数量
• 实现分片压缩传输