feat(showcase): speed up tests in CI

[aklkv]

📌 Summary

This change cuts component tests in CI by 50%, today on average it takes around 4 minutes with this change we reduce it to 2 minutes.

The only potential issue I see is this message:

[percy] Error: This organization has exceeded the limits of the Percy BrowserStack plan. Administrators can upgrade here: https://percy.io/organizations/587d9efb/billing

🛠️ Detailed description

📸 Screenshots

Before:

After:

🔗 External links

Jira ticket: HDS-XXX
Figma file: [if it applies]

---

👀 Component checklist

• Percy was checked for any visual regression
• A changelog entry was added via Changesets if needed (see templates here)

💬 Please consider using conventional comments when reviewing this PR.

[vercel]

You must have Developer access to commit code to HashiCorp on Vercel. If you contact an administrator and receive Developer access, commit again to see your changes.

Learn more: https://vercel.com/docs/accounts/team-members-and-roles/access-roles#team-level-roles

[Dhaulagiri]

FWIW the percy message is not unique to this PR. I see it in others as well.

[alex-ju]

Generally, using ember-exam sounds like a good idea for introducing parallel testing.

I've only been able to compare test:ember (not including the VRT testing in Percy) and the differences are marginal (2m12s, compared to 2m20s for main); However, it could make all the difference on test:ember:percy, but we seem to have some challenges around using Percy so not able to test that atm.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
hds-showcase	✅ Ready (Inspect)	Visit Preview	Aug 3, 2024 10:36am
hds-website	✅ Ready (Inspect)	Visit Preview	Aug 3, 2024 10:36am

[aklkv]

Generally, using ember-exam sounds like a good idea for introducing parallel testing.

I've only been able to compare test:ember (not including the VRT testing in Percy) and the differences are marginal (2m12s, compared to 2m20s for main); However, it could make all the difference on test:ember:percy, but we seem to have some challenges around using Percy so not able to test that atm.

I tried adding ember-exam to website tests but it didn't result in much savings so I thought there wouldn't be much sense in adding it at this stage. I am happy to add this instrumentation here as well for consistency.

On the other hand showcase test were much more cooperative and we can see ~2 min saving

[aklkv]

FWIW the percy message is not unique to this PR. I see it in others as well.

oh, good to know I thought ember-exam parallel runs triggered this warning

[Dhaulagiri]

Percy is fixed now so we could re-run the comparisons to see if the improvement in perf you were seeing was related to the introduction of ember-exam or a coincidence because percy tests were no longer running

[aklkv]

@Dhaulagiri I think I got it fixed so the tests aren't flaking anymore but but speed gains are a little less we are sitting at 2m 45s which is still a good reduction

[Dhaulagiri]

@aklkv apologies for the delay on this. lost track of this over the holiday.

the time savings is relatively small but it is faster overall, so this seems worth bringing in to me given the low cost to opt-in to this feature. @alex-ju any objections from your perspective?

[aklkv]

@Dhaulagiri No worries at all. One thing I noticed that Percy is still in Pending state, I wonder if this is something related to my changes

[Dhaulagiri]

🤔

Only 3 of the 8 parallelized build processes reported they were finished.

[Dhaulagiri]

I was doing a a lot of manual starting/stopping of the tests to test this so that may be part of it, but we may also need to try following all the instructions here.

[alex-ju]

the time savings is relatively small but it is faster overall, so this seems worth bringing in to me given the low cost to opt-in to this feature. @alex-ju any objections from your perspective?

no objections, any improvement is progress. I'll try and give this PR a fresh pair of eyes today or on Monday