Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add custom error handler #485

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

etnoy
Copy link

@etnoy etnoy commented Feb 2, 2025

What is the purpose of this pull request?

Add the option to suppress EACCES, permission denied errors.

This is an error we're seeing in Immich when fast-glob is crawling a directory that contains items that we lack read permission for. We don't want to error out and stop crawling, and we don't want to suppress all errors. This is a reasonable middle road that adds an option, suppressEaccess

Ref: immich-app/immich#12713

In the above bug, a user has a few files in their NAS that wan't accessible. This throws out the whole crawl process even though only a few items are missing

What changes did you make? (Give an overview)

Add option suppressEacces
Add more conditions in isNonFatalError and _isFatalError
Added documentation in README

@benmccann
Copy link

I wonder if an error handling function might be a better solution to this problem. Then you could choose which types of errors to ignore vs error on, etc. Otherwise you might end up with a new option for each type of error that could be encountered

@etnoy
Copy link
Author

etnoy commented Feb 3, 2025

I wonder if an error handling function might be a better solution to this problem. Then you could choose which types of errors to ignore vs error on, etc. Otherwise you might end up with a new option for each type of error that could be encountered

That could work, but I'm trying to figure out what further errors a user might want to filter out beyond the permission error. I can only think of EMFILE and EINVAL, both of which are pretty fatal to any further globbing.

Maybe we can look at a refactor of this as a follow-up?

Also, maybe we should add a log warning to any file that was skipped due to EACCES?

@benmccann
Copy link

I'm trying to figure out what further errors a user might want to filter out beyond the permission error.

Perhaps ENOENT if a file/directory is deleted during globbing. I'm not sure if it's possible to trigger it not

Maybe we can look at a refactor of this as a follow-up?

I don't think you'd want to change the option in that manner in a follow up as it'd be a breaking change. Better to do it up front

Also, maybe we should add a log warning to any file that was skipped due to EACCES?

That's the beauty of making it a function. Some users can silently proceed while others can log an error with their logging framework of choice

@etnoy etnoy force-pushed the feat/suppress-eacces branch from f7d29d7 to fa21607 Compare February 6, 2025 12:54
@etnoy etnoy changed the title feat: suppress eacces option feat: add custom error handler Feb 6, 2025
@etnoy
Copy link
Author

etnoy commented Feb 6, 2025

Thanks for your feedback. I've pushed some changes that allows the user to supply an error handler function.

In Immich, we'll use it something like this:

image

From my testing it seems to work very well, and also allows us to log warnings if we get ENOENT or EACCES.

Let me know if this seems to be the correct approach and I'll make sure everything is cleaned up and nice

@etnoy etnoy force-pushed the feat/suppress-eacces branch 4 times, most recently from 9d3b44b to 77aaa37 Compare February 6, 2025 22:32
@etnoy etnoy force-pushed the feat/suppress-eacces branch from 77aaa37 to 9cd2bdd Compare February 6, 2025 22:38
@etnoy
Copy link
Author

etnoy commented Feb 21, 2025

Hello @mrmlnc , what is your opinion of this PR so far? Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants