Set length of avro input file after memory input file is created #23667
Conversation
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
| if (estimatedFileSize < BUFFER_SIZE.toBytes()) { | ||
| try (TrinoInputStream input = inputFile.newStream()) { | ||
| byte[] data = input.readAllBytes(); | ||
| inputFile = new MemoryInputFile(path, Slices.wrappedBuffer(data)); | ||
| } | ||
| } | ||
| length = min(inputFile.length() - start, length); |
There was a problem hiding this comment.
Can we check if similar change is needed in LinePageSourceFactory and RcFilePageSourceFactory as well ?
There was a problem hiding this comment.
Is it possible to add a test for this ?
E.g. we test filesystem with AWS encryption at io.trino.filesystem.s3.TestS3FileSystemAwsS3WithEncryption
Description
In AWS Athena, we currently support encrypting S3 objects using CSE-KMS. The size of these encrypted object will be greater than the size when unencrypted since padding will be added. The
input.readAllBytes();function strips the padding from an encrypted object so we have to update thelengthvariable as well if we create a new in memory input file.Additional context and related issues
Release notes
(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:
@nineinchnick @anusudarsan