Uploading a large file to S3 from flutter

We were recently trying to upload a large zip file from flutter browser app and we discovered that when we try to upload files larger than 500 MB it was throwing Buffer Allocation error. We needed to upload files larger than 10 GB.

So we started exploring other options. We finally solved the problem using S3 Multipart upload. We did not want to expose S3 credentials to client due to security reasons. So we implemented 3 APIs on server:

1. InitiateMultipartUpload - This API initiates multipart upload on S3 and return uploadId to client
2. UploadPart - This API receives a part from client and sends it to S3 and returns PartETag to client
3. ComplateMultipartUpload - This API completes multipart upload in S3 using PartETags received from client.

Now for client side support we developed a custom component to upload file to S3 taking inspiration from https://github.com/Taskulu/chunked_uploader/blob/master/lib/chunked_uploader.dart

After these changes I was able to upload 5GB file to S3 in 20 minutes without getting buffer allocation error.

Uploading file to S3 using flutter

Recently were working on uploading a zip file to S3 using presigned url. We exposed an API to client for generating a presigned url over an authenticated API call. After generating presigned URL client is supposed to send content in a PUT call to S3 on the presigned url. But when we tried sending the file using MultipartRequest the file got corrupted. After struggling for a long time we found StreamedRequest using which we were able to upload the file to S3 without getting it corrupted. Following is the code for uploading the file to S3.

Future<http.StreamedResponse?> uploadFileToS3(
  String s3Url,
  file_picker.PlatformFile file,
) async {
  var request = http.StreamedRequest('PUT', Uri.parse(s3Url)); 
  final streamSink = request.sink as StreamSink<List<int>>;
  print('Uploading file '+file.name+' to '+s3Url);
  var resp = request.send();
  await streamSink.addStream(file.readStream!);
  streamSink.close();
  print('returning response for '+file.name);
  return await resp;
}

Please note readStream will be null if you don't pass required arguments to file_picker. The required arguments for file picker are:

FilePickerResult? result = await FilePicker.platform.pickFiles(withData:false, withReadStream: true);

Please let me know if you face any problem in using this approach.