chore: test ci for flaky tests

[edalex-yinzi]

Checklist

• the contributor license agreement is signed
• commit message follows commit guidelines
• tests are included
• screenshots are included showing significant UI changes
• documentation is changed or added

Description of change

[edalex-yinzi]

OK looks good.

[edalex-yinzi]

1

[edalex-yinzi]

2

[edalex-yinzi]

3

[edalex-yinzi]

Maybe the issue is the test will be affected by other tests; I will bring another test back.

[edalex-yinzi]

1

[edalex-yinzi]

2

[edalex-yinzi]

It looks like FavouritesPowerSearchIntegration has been fixed.

[edalex-yinzi]

1

[edalex-yinzi]

Interesting, so after I added the screenshot, the portal test became stable.

[edalex-ian]

Interesting, so after I added the screenshot, the portal test became stable.

Makes sense, as these flaky issues seem to all be timing based. So add something in that significantly alters duration (screenshots) and then you change the timing environment.

[edalex-yinzi]

😭

[edalex-yinzi]

Adding screenshots really stabilized the tests a lot... I think I should add explicit waits to fix the tests.

[edalex-ian]

Adding screenshots really stabilized the tests a lot... I think I should add explicit waits to fix the tests.

Although an easy way to fix this kind of issue, it's an approach that is considered a big NO! It's just a temporary fix at best, at worst it can add more unknowns - as it signals a thread switch.

• https://medium.com/cloud-native-daily/why-is-testing-with-thread-sleep-a-bad-idea-and-what-can-you-do-instead-3aad1d26a787
• https://www.browserstack.com/guide/how-to-avoid-flaky-tests

[edalex-yinzi]

Okay at least those timneout issues have been solved. Now we only have java.lang.AssertionError: expected [true] but found [false] .

[edalex-ian]

So this is superseded by #5023 ?

[edalex-yinzi]

So this is superseded by #5023 ?

Yes. Closed.