Retry test mechanism, I am looking for a really good way

[pandoras-toolbox]

What do you recommend for retrying whole tests on failure, especially for E2E tests?

I know the Maven Surefire plugin supports to configure it with "rerunFailingTestsCount" but it seems not to support "skipAfterFailureCount" for JUnit 5. And there is no way to configure on which exception types and exception messages to rerun a failed test. Also there is no way to configure the rerun only for certain tests. And it is impossible to specify a delay time between reruns, which is helpful especially for E2E tests. But it outputs flaky re-run information in test report xml which is good, because Allure then can display the reruns in the test report nicely and can know about flaky tests.

For Gradle it is a bit better with the test-retry-gradle-plugin because it supports arguments "maxFailures" and "failOnPassedAfterRetry", but other than that it has the same disadvantages like the Maven Surefire plugin.

Neither the Maven Surefire nor the test-retry-gradle-plugin maintainers care to improve these plugins.

Then there is the RetryingTest annotation from junit-pioneer project. The advantage is that you can specify to retry only certain tests on failure, specify on which exception type to retry and specify a delay time between retries. But unlike Failsafe for Java it does not include more sophisticated features like specifying a jitter factor or a backoff. And it makes the number of executed tests on the Allure test report wrong because each retry is counted as an additional test. It has no "maxFailures" property for the whole test suite but only for a single test, there is no short-circuit mechanism supported to prevent too many retries if the system is broken. And it does not produce flaky re-run information in the test report xml.

For E2E tests it is very helpful to do entire test re-runs, but I did not find any really good tool for it in the Java/JUnit 5 ecosystem. Everything I seen is half-baked in one way or the other.

Or am I missing something? How would you do retries of tests, having high expectations for what features it needs to include?