Test tasks sometimes miss the build cache due to overlapping outputs

[abrooksv]

Describe the bug
The Test tasks have their build cache disabled when ran:

The task was not up-to-date because of the following reasons: |  
-- | --
Output property 'binaryResultsDirectory' file build/test-results/test/binary has been removed. |  
Output property 'binaryResultsDirectory' file build/test-results/test/binary/output.bin has been removed. |  
Output property 'binaryResultsDirectory' file build/test-results/test/binary/output.bin.idx has been removed. |  
 
Cache key | 2e67aa93bbea2dbfed0525b9a74b6cd0 |  
Not cacheable | Overlapping outputs: Gradle does not know how file 'build/idea-sandbox/system-test/index/stubs/Stubs.storage.values.at' was created (output property 'System directory'). Task output caching requires exclusive access to output paths to guarantee correctness (i.e. multiple tasks are not allowed to produce output in the same location).

To Reproduce

1. Run test task with a project that has build cache enabled (org.gradle.caching=true)
2. delete the outputs of test results to force a re-run (but inputs won't be changed so build cache key should match). I deleted the test-results folder: rm -r build/test-results
3. Run tests again
4. See a build scan output as to why the task was re-run instead of FROM_CACHE

I am trying to build a test case but I can't get it to reproduce right now...
master...abrooksv:gradle-intellij-plugin:testBuildCache

Expected behavior
Test results should be pulled from build cache

Side note
Should the system dir not be an output, but a localState property?

Environment:

• OS: Mac OS
• Gradle IntelliJ Plugin Version 1.8.1
• Gradle Version 7.5

Additional context
Add any other context about the problem here.

[YannCebron]

what if you change non-code test-data? we can't guess where that possibly comes from

[abrooksv]

Do you mean you think it's not possible to make it work with the build cache due to outside testdata directory can affect it? Does that not fall into the realm of the user to add

tasks {
    test {
        inputs.dir(project.file("testdata"))
    }
}

?

---

I could not get the test case to produce the values.at file, but only:

Stubs.storage
Stubs.storage.len
Stubs.storage_i.len
Stubs.ver

but I doubt the specific file matters.

---

I did run this patch:

Index: src/main/kotlin/org/jetbrains/intellij/IntelliJPlugin.kt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/main/kotlin/org/jetbrains/intellij/IntelliJPlugin.kt b/src/main/kotlin/org/jetbrains/intellij/IntelliJPlugin.kt
--- a/src/main/kotlin/org/jetbrains/intellij/IntelliJPlugin.kt	(revision a4b137280d3e01810acb18d50605aed4faa18cce)
+++ b/src/main/kotlin/org/jetbrains/intellij/IntelliJPlugin.kt	(date 1662569740738)
@@ -1152,8 +1152,7 @@
             // the same as previous – setting appClassLoader but outdated. Works for part of 203 builds.
             task.systemProperty("idea.use.core.classloader.for", pluginIds.joinToString(","))
 
-            task.outputs.dir(systemDirectoryProvider)
-                .withPropertyName("System directory")
+            task.localState.register(systemDirectoryProvider)
             task.inputs.dir(configDirectoryProvider)
                 .withPropertyName("Config Directory")
                 .withPathSensitivity(PathSensitivity.RELATIVE)

And it works locally with the above steps against our private plugin.

The change is the system dir won't be stored in the build cache anymore.

I can't see a use case for consuming them in a downstream task, only issue that may arise (which could be a problem in and of itself) is that the system-test is shared across all Test tasks, so if you have integ and unit tests, the IDE caches can be reused (but that may also definitely cause overlapping outputs too...)

It does bother me that I can't seem to reproduce it as an integration test 😞