Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Append the hash by default. #47

Open
kurin opened this issue Mar 9, 2018 · 6 comments
Open

Append the hash by default. #47

kurin opened this issue Mar 9, 2018 · 6 comments

Comments

@kurin
Copy link
Owner

kurin commented Mar 9, 2018

By default, blazer will do a pass over the data to compute the hash so that it can be sent at the start of the upload request, as described in https://www.backblaze.com/b2/docs/uploading.html

Blazer can either do this in memory, which is the default, or can copy the input into a temp file, or it can just make two passes over the input if the input supports seeking. Each of these has side effects that might not be obvious to a user (e.g., if you're using blazer to copy files out of NFS, blazer will happily read every file twice).

Blazer should append the hash at the end by default. This is not recommended by blazer, but if there's a functional difference between the two options it's not obvious to me what it is.

@fd0
Copy link
Contributor

fd0 commented Mar 9, 2018

Cool, that'd be nice to have!

@kurin
Copy link
Owner Author

kurin commented Mar 9, 2018

B2 doesn't support chunked transfers, so this just got a lot less exciting.

@AviKav
Copy link

AviKav commented May 9, 2018

Blazer should append the hash at the end by default. This is not recommended by blazer

Why isn't it recommended?

B2 doesn't support chunked transfers

What advantages would chunked transfers provide?

@kurin
Copy link
Owner Author

kurin commented May 9, 2018

I mistyped, I meant it's not recommended by Backblaze. I don't know why, and I still haven't thought of anything.

It's been a while, but I think the reason I wanted chunked transfers was so that I wouldn't have to set the content-length header on upload. For byte streams (which is my use case) I'd still have to buffer the whole file somewhere, so I didn't develop this.

Do you have a use-case for it?

@kurin
Copy link
Owner Author

kurin commented May 9, 2018

it can just make two passes over the input if the input supports seeking

btw this is no longer true, it only makes one pass in this instance.

@kurin
Copy link
Owner Author

kurin commented May 11, 2018

Oh, here's a good reason: if the file changes during upload, appending the hash will happily submit the corrupted file, but a two-pass solution would have the upload be rejected as the hashes wouldn't match.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants