Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to exclude folders from scanning #4360

Open
falco467 opened this issue Feb 8, 2024 · 17 comments
Open

Option to exclude folders from scanning #4360

falco467 opened this issue Feb 8, 2024 · 17 comments
Labels
dependencies Relates to an upstream dependency enhancement New feature or improvement

Comments

@falco467
Copy link

falco467 commented Feb 8, 2024

Your feature request related to a problem? Please describe.

golang-lint-ci --fast is slowed down, because it traverses all directories (even unrelated big directories like a node_modules folder) and there is no option to exclude unrelated folders from scanning/traversal.

Describe the solution you'd like.

Provide an option to ignore/exclude folders from traversal/scanning.

Describe alternatives you've considered.

I tried excluding the folders with the --skip-dirs flag, but golang-lint-ci will still traverse the subfolders and take a lot of time on each run.

Additional context.

I'm using golang-lint-ci as my integrated linter in VS Code (running on file-save), fast execution time is critical. I measured performance with --verbose and Go packages loading at mode 575 takes up the most time (even with caching enabled) it takes almost 9 seconds every run, even though my project has only ~1000 Lines of Go-Code in 6 files.

We have the frontend project in a subfolder of the go project (/frontend) which includes a huge node_modules folder. Deleting this folder speeds up the scan from 12 to 4 seconds. Is there any option to hide this folder from golang-lint-ci ?

@falco467 falco467 added the enhancement New feature or improvement label Feb 8, 2024
Copy link

boring-cyborg bot commented Feb 8, 2024

Hey, thank you for opening your first Issue ! 🙂 If you would like to contribute we have a guide for contributors.

@falco467
Copy link
Author

falco467 commented Feb 8, 2024

This could also probably be fixed by caching. On multiple runs golang-lint-ci could recognize by the modification time on the folder that nothing has changed inside the folder and should not need to scan anything inside the folder. I'm working with go build cache and golang-lint-ci cache. But it is still scanning all folders again even when I don't change any file between runs.

@ldez
Copy link
Member

ldez commented Feb 8, 2024

Hello,

A package, if it's used by another part of the code, cannot be skipped.

Related to #4217
Related to #913

@falco467
Copy link
Author

falco467 commented Feb 8, 2024

This is not about skipping packages, but skipping unrelated Folders without any .go files in it (like a folder with PDFs or JS files)

@falco467
Copy link
Author

falco467 commented Feb 8, 2024

Here is the output which illustrates the problem:

golangci-lint run --verbose
level=info msg="[loader] Go packages loading at mode 575 (types_sizes|files|imports|name|compiled_files|deps|exports_file) took 10.0107529s"
level=info msg="Execution took 12.4308233s"

After deleting the big folder without any .go files:

golangci-lint run --verbose
level=info msg="[loader] Go packages loading at mode 575 (compiled_files|imports|name|types_sizes|deps|exports_file|files) took 2.3652813s"
level=info msg="Execution took 4.488697s"

@ldez
Copy link
Member

ldez commented Mar 10, 2024

Can you provide a reproducible example?

@falco467
Copy link
Author

@ldez sure - here is a simple example:

A project with a single main.go file and a folder "stuff" which contains no files, just empty folders.
golangci-lint --skip-dirs stuff still takes about 30seconds on windows in contrast to 3 seconds if I delete the folder stuff.

package main

import (
	"fmt"
	"os"
)

func main() {
	fmt.Println("generating 10_000 empty folders")
	for i := range 100 {
		for j := range 100 {
			os.MkdirAll(fmt.Sprintf("stuff/f%d/f%d", i, j), 0o777)
		}
	}
}

Here is the whole project as a zip file:
golangtest.zip

@falco467
Copy link
Author

I think I located the problem with how the go module system works. If I rename the folder to _stuff or if I put an empty "go.mod" file into the folder it will be ignored by the go module system and build and I get the fast response.
If golangci-lint cannot solve this, we will use the workaround of putting an empty go.mod file with a comment into the subfolder so it gets ignored.

@ldez
Copy link
Member

ldez commented Mar 13, 2024

Just to explain, golangci-lint uses the Go pieces and those pieces allow only the same pattern as go build.
The scanning will happen on all trees because Go needs to know where Go files are located.
Directories starting with _ are ignored by Go.
Golangci-lint will not be able to do a magical filter.

@skandragon
Copy link

My use case is different. I have some auto-generated code from a Ragel lexer, and it is huge and mostly not my own code. I'm willing to accept slightly broken code there so long as it compiles.

The problem is, this 250k file takes significant time to run, and even if issues are found, it is not likely I will care enough to fix them so long as the code compiles. It is a literal waste of CPU minutes in GitHub Actions to scan this file, and I have to up my time limit to 10m just to get it to be scanned.

If I were to add a //nolint: ignore-whole-file option or something, would this work? I suppose I'd still need to have a syntax scan to get the method fingerprints so return values can be checked in other source.

My use case isn't per-package as I in the same package I have code I do want to be scanned.

@ldez
Copy link
Member

ldez commented Apr 7, 2024

As I already explained: the issues reported for a file can be ignored but the files cannot be ignored by the analysis.

@falco467
Copy link
Author

@skandragon can your code compile without this 250k file? If not than you're probably out of luck, since the whole code-base has to be compiled and linked by the go builder.

But I was thinking - should it not be possible to increase performance with more aggressive caching for excluded files? If the files are in subfolder e.g. other modules, should not build-caching allow the linker to reuse the binary without rebuilding? And the linter could also use cached resources for this file, since it is marked as included?
Or is this something we need to address upstream with the go build team ?

@ldez
Copy link
Member

ldez commented Apr 19, 2024

When the Go tooling supports that, we will be able to effectively ignore packages and files.
But for now, the possibilities are extremely limited.

@silverwind
Copy link
Contributor

silverwind commented Apr 19, 2024

When the Go tooling supports that, we will be able to effectively ignore packages and files. But for now, the possibilities are extremely limited.

Could also just crawl the files without relying on the go tooling which would also solve other issues like having to run golangci multiple times with different build tags or GOOS values to be able to lint everything.

@ldez
Copy link
Member

ldez commented Apr 19, 2024

The Go tooling is the core of golangci-lint, golangci-lint relies on that to have its performance.
We will not reinvent the wheel.

@silverwind

This comment was marked as off-topic.

@coin-au-carre
Copy link

I have also the same problem, I have a persistend database generated by Docker, in one of my subfolder and I would like that golangci does not try to look into it...

golangci-lint run --fix
ERRO [linters_context] typechecking error: pattern ./...: open /home/user/project/docker/persistent-data/db: permission denied 
make: *** [Makefile:83: lint] Error 7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Relates to an upstream dependency enhancement New feature or improvement
Projects
None yet
Development

No branches or pull requests

5 participants