Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

src: cache negative results in GetPackageJSON #56834

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

dmichon-msft
Copy link

Fixes #56821

Ensure that GetPackageJSON caches which package.json files do not exist, not just the contents of the ones that do. In packages with at least one subfolder, it is normal for most folders in a package not to contain a package.json file, so this will significantly reduce the number of file system reads performed during GetNearestParentPackageJSON.

This change ensures that GetPackageJSON caches
which folders do *not* contain a package.json, to
prevent excessive file system probing.
@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Jan 30, 2025
@ljharb
Copy link
Member

ljharb commented Jan 30, 2025

What happens if a package.json file is created after this is cached?

@dmichon-msft
Copy link
Author

What happens if a package.json file is created after this is cached?

The same thing that happens if you change the content of an existing package.json file after it gets read (and cached). The state of the world as of the first read of the folder is preserved.

@ljharb
Copy link
Member

ljharb commented Jan 30, 2025

How does one clear the cache, then, after making a change?

@dmichon-msft
Copy link
Author

How does one clear the cache, then, after making a change?

Usually by shutting down and restarting the process, since this cache is used during module loading.

@dmichon-msft
Copy link
Author

dmichon-msft commented Jan 31, 2025

Missing package.json files were cached in Node <= 20:

cache.set(jsonPath, result);

As near as I can tell, this behavior change was simply an oversight when migrating readPackageScope from the JS layer to the native layer.

Node <= 20 also does not provide any means for cache entries regarding package.json to be cleared.

src/node_modules.cc Outdated Show resolved Hide resolved
src/node_modules.cc Outdated Show resolved Hide resolved
Fixed incorrect negation

Co-authored-by: Yagiz Nizipli <yagiz@nizipli.com>
@jasnell
Copy link
Member

jasnell commented Jan 31, 2025

Usually by shutting down and restarting the process, since this cache is used during module loading.

Hmm...as a follow up, should we consider making it possible to clear both the missing and found caches at runtime to allow for hot reloading of these? Would that even make sense? Not something to worry about in this PR but something worth exploring separately.

@nodejs-github-bot
Copy link
Collaborator

@dmichon-msft
Copy link
Author

Usually by shutting down and restarting the process, since this cache is used during module loading.

Hmm...as a follow up, should we consider making it possible to clear both the missing and found caches at runtime to allow for hot reloading of these? Would that even make sense? Not something to worry about in this PR but something worth exploring separately.

The entries in the cache will be there because they impacted the loading of already-loaded modules, so you could get some really weird results if we purge the cache and not all of the modules that were loaded from folders whose package.json files were checked. Ultimately, I think any runtime change that affects resolution of already-loaded code is heavily in the "use at your own peril" category.

I think we'd be better served ensuring the hits and misses also find their way into the metadata for node --watch so that it can restart if a relevant package.json changes, is deleted, or becomes available (assuming they aren't already?).

@aduh95
Copy link
Contributor

aduh95 commented Jan 31, 2025

It looks like build is failing on CI

@jasnell
Copy link
Member

jasnell commented Jan 31, 2025

Appears to need #56846 to land...
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GetPackageJson internal binding should cache missing file results
6 participants