We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rclone currently uses the following parameters:
rclone --transfers 5 --use-mmap --buffer-size 0M --use-json-log --stats 200ms --verbose sync mys3:project/node/workspace/ test/ --links
A few things here are questionable:
--buffer-size 0M
use-mmap
--transfers 5
We have some examples from TN where the dataset has >300k files and total size >170GiB.
rclone --transfers 32 with TN dataset brought the download time from 1 hour 30 to 15 minutes!
rclone --transfers 32
The text was updated successfully, but these errors were encountered:
To change --transfers R_CLONE_OPTION_TRANSFERS=5 is already present on all deployments and can be used to change the number of parallel downloads.
--transfers
R_CLONE_OPTION_TRANSFERS=5
Sorry, something went wrong.
To change --buffer-size R_CLONE_OPTION_BUFFER_SIZE=0M is already present on all deployments.
--buffer-size
R_CLONE_OPTION_BUFFER_SIZE=0M
GitHK
sanderegg
Successfully merging a pull request may close this issue.
Rclone currently uses the following parameters:
A few things here are questionable:
--buffer-size 0M
this one is something that should not be, the default is 16M and is almost never hit. it is a performance hit.use-mmap
this one seems pretty useless and even maybe problematic--transfers 5
if we have many small files, this is very limiting. Nevertheless this is probably also directly linked to the amount of modelsWe have some examples from TN where the dataset has >300k files and total size >170GiB.
tested
rclone --transfers 32
with TN dataset brought the download time from 1 hour 30 to 15 minutes!The text was updated successfully, but these errors were encountered: