You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
during the data extraction process, full copies of the data as it passes thorugh different states are generated (e.g. see any of the output folders).
the final dataset is large.
These files trigger the use of Github's LFS system, which may at some point become a financial cost for the project. Whilst it's useful as an assurance and debugging measure to generate lots of tsvs at different points in processing, they're unncessary to keep long term. In all cases, these files can be re-generated from their inputs anyhow.
Some work on the sub-projects to prune these intemediate outputs from the historical repo would be a useful investment in the event of a change in LFS storage costs.
The text was updated successfully, but these errors were encountered:
This project generates a lot of large TSV files:
output
folders).These files trigger the use of Github's LFS system, which may at some point become a financial cost for the project. Whilst it's useful as an assurance and debugging measure to generate lots of tsvs at different points in processing, they're unncessary to keep long term. In all cases, these files can be re-generated from their inputs anyhow.
Some work on the sub-projects to prune these intemediate outputs from the historical repo would be a useful investment in the event of a change in LFS storage costs.
The text was updated successfully, but these errors were encountered: