Skip to content
This repository has been archived by the owner on Nov 1, 2023. It is now read-only.

The agent should delete output files after they are uploaded to Azure #3447

Open
DrChat opened this issue Aug 24, 2023 · 1 comment
Open
Labels
enhancement New feature or request

Comments

@DrChat
Copy link
Member

DrChat commented Aug 24, 2023

As stated, for any output files uploaded from a task, the 1F agent should delete the source files from disk after the azcopy sync operation has finished.
We need this because we produce some pretty large files from our generic_analysis task, and they have the potential to fill up the OS disk if they aren't cleaned up after they have been uploaded.

The most likely place that this could be implemented is in SyncedDir::sync, adding an additional delete_src: bool parameter.

Also note that it does not appear that the azcopy team will implement this functionality:
Azure/azure-storage-azcopy#990

AB#164737

@DrChat
Copy link
Member Author

DrChat commented Sep 7, 2023

Per discussion with @chkeita, this functionality would have to be opt-in due to the possibility that existing tasks probably expect outputs to still exist after they are uploaded.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants