Skip to content

Commit

Permalink
UserCache: env var to enable agressive cache writing
Browse files Browse the repository at this point in the history
When using oca-port as a Python package to collect migration data where
repositories are cloned on a filesystem with very low IO perf, the
process could take a lot of time to finish (every underlying git operations
to retrieve for instance updated file paths of a commit are very
costly). Such process could be killed before it actually finishes (and
write the data in cache).

By setting the OCA_PORT_AGRESSIVE_CACHE_WRITE environment variable, we
force oca-port to update the cache very often, generating more IO during
the first analysis of a module, but making the next process execution
much faster.

As this is an advanced option, it is exposed through an environment
variable to not clutter the usual parameters.
  • Loading branch information
sebalix committed Oct 14, 2024
1 parent ffedd00 commit d057730
Showing 1 changed file with 12 additions and 0 deletions.
12 changes: 12 additions & 0 deletions oca_port/utils/cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -221,13 +221,25 @@ def set_commit_files(self, commit_sha: str, files: list):
if self.readonly:
return
self._commits_data[commit_sha]["files"] = list(files)
if os.environ.get("OCA_PORT_AGRESSIVE_CACHE_WRITE"):
# IO can be very slow on some filesystems (like checking modified
# paths of a commit), and saving the cache on each analyzed commit
# could help in case current oca-port process is killed before
# writing its cache on disk, so the next call will be faster.
self._save_commits_data()

def save(self):
"""Save cache files."""
if self.readonly:
return
self._save_commits_to_port()
self._save_commits_data()

def _save_commits_to_port(self):
# commits/PRs to port
self._save_cache(self._commits_to_port, self._commits_to_port_path)

def _save_commits_data(self):
# commits data file
self._save_cache(self._commits_data, self._commits_data_path)

Expand Down

0 comments on commit d057730

Please sign in to comment.