Multipart upload with CSE using RSA key pair #268
Comments
|
Hello Sid22, The Async S3 Encryption Client uses the Java implementation Reactive Streams to send data to S3. This is generally more efficient, but for multipart uploads, there are issues, because Reactive Streams do not have an equivalent to Alternatively, for the most robust multipart upload solution, you can set the same option ( Let us know if you have any further questions, thanks! |
|
@justplaz thanks for the detailed explanation. I modified my code to add However, i still see the same error. On a side note we are using the "low level" multi part upload API ( We have a method which expects an object of S3Client interface and uses it with low level multipart methods to do the upload. If i pass S3Client object directly the multi part upload succeeds so it is not an issue of improper login in the upload method. |
|
I see, that makes sense then. If you're using the the low-level multipart upload API, then setting For low-level multipart upload, you need to ensure that each part is encrypted sequentially and in the correct order. Otherwise, the encryption will fail. Please refer to the low-level multipart upload example for an example of how to upload parts in sequence. Let us know if you are still having issues, thanks! |
|
@justplaz thanks for the example, one key thing we are not doing is
We were uploading parts sequentially in order but were not setting the above anywhere. I will modify the code to do this and then revert back |
|
I added check in our code to manually ensure we tag the last part properly. Still however getting the same error but from a different code part now within the SDK @justplaz is there any check around the type of RSA keys that work ? I generated a fresh sample RSA key and got the above error. Adding the RSA key here for reference. key:- |
Problem:
My code is roughly
Now I create S3EncryptionClient by wrapping above such as
I am able to use this
encObjectto do operations like creating bucket etc. I am also able to upload files to s3 bucket.However, when i try to upload a large file ( say ~200MB ) with multi part upload it fails with following error
If I directly use the
s3ClientObjectit works.Solution:
Is there some limitation on CSE with multi part uploads ?
The text was updated successfully, but these errors were encountered: