Make tests build cacheable

[martinbonnin]

Tests are not build cacheable. Running them twice in a row re-runs them. To reproduce:

git clone https://github.com/apollographql/apollo-kotlin
cd apollo-kotlin
git checkout 4824ae3bf53025a0b2b6290edb609647834f435c
# run tests a first time
./gradlew :intellij-plugin:test 
# delete outputs
rm -rf intellij-plugin/build
# re-run tests
./gradlew :intellij-plugin:test 
# tests are executed again

The second test invocation should reuse the cached results but instead the tests are re-run.

I'm suspecting it's because some of the input/output directories are shared between tasks or something like this? On some occasions, I have seen logs like this:

Caching disabled for task ':intellij-plugin:test' because:
  Gradle does not know how file 'build/idea-sandbox/system-test/log/idea.log' was created (output property 'System directory'). Task output caching requires exclusive access to output paths to guarantee correctness (i.e. multiple tasks are not allowed to produce output in the same location).

Other times, it was classpath.index being missing in one of the invocations.

Tests currently take ~1min so it's not the end of the world but would be nice to get those minutes back.

[mgroth0]

I stumbled upon this because my gradle configuration cache is getting invalidated for what seems like a similar reason:

Calculating task graph as configuration cache cannot be reused because the file system entry 'ideaPlugin/build/classes/kotlin/main/classpath.index' has been created.

Seems like my config cache gets invalidated whenever some test task is run on the idea plugin module.

[YannCebron]

there could be changes in

• testData files used in tests (e.g., highlighting)
• any external systems (e.g., LSP)

that are impossible to track

[martinbonnin]

testData files used in tests

We can track that manually:

tasks.withType<Test> {
  inputs.dir("testData").withPropertyName("testData").withPathSensitivity(PathSensitivity.RELATIVE)
}

any external systems (e.g., LSP)

Not overly familiar with LSP and/or other things. Is that something IntelliJ downloads at startup? If yes that could be an issue but maybe something that could be pinned/disabled using configuration?

[YannCebron]

testData directories could be located anywhere, or even generated at runtime

LSP server is just a sample for any external system that tests may depend on. Or e.g. some external linter that is called via ExternalAnnotator and many other things.

[martinbonnin]

My superficial understanding of the IntelliJ IDE is that it's mostly a (very sophisticated) executable .jar and therefore self contained/predictable and ultimately cacheable?

I understand IntelliJ is highly customizable and as such could take inputs from "the internet" that are impossible to track but these customizations would not be important for our tests so feels like having a way to run the IDE without them for reproducable tests would be a win?

[YannCebron]

This is about what the actual tests of the plugin project do. We basically can't predict it or set "useful" defaults for caching.

Reference for test data: https://plugins.jetbrains.com/docs/intellij/test-project-and-testdata-directories.html

[martinbonnin]

For the files/inputs that belongs the plugin under test tests (our testData is here), feels like the user could add inputs to the test task manually. In our case, it would be like so:

tasks.withType<Test> {
  inputs.dir("src/test/testData").withPropertyName("testData").withPathSensitivity(PathSensitivity.RELATIVE)
}

Doing this, I would expect the test task to become cacheable, i.e.:

• run tests
• delete the build folder
• re-run tests: the tests results should be taken from the build cache, and not re-run

(This is obviously assuming that the src/test/testData also did not change)

Is that a wrong expectation to have?

[YannCebron]

That's correct, but a breaking change for any plugin relying on the previous behaviour (or having non-default testData location(s)).