TRADE-187: Add tests for SAMBAH

[danielfromearth]

Description

Adds a regression test notebook for the SAMBAH service chain.

Supersedes #93, in order to track references files using git lfs.

Jira Issue ID

• TRADE-187 (closed)
• TRADE-410

Also related to:

• TRADE-500 (completed and closed)

Local Test Steps

Ran notebook tests for UAT and PROD environments successfully.

PR Acceptance Checklist

• Acceptance criteria met
• Tests added/updated (if needed) and passing
• [N/A] Documentation updated (if needed)

[danielfromearth]

@owenlittlejohns, @ank1m,

I've revised the Harmony regression test notebook, and I thiiiink it's good. The tests for UAT are passing.

The tests in PROD don't yet produce the same output as the tests in UAT, however, despite using matching granules. I believe that is because of the difference in versions currently deployed in the environments:

PROD: l2ss-2.11.0, batchee-1.1.0, stitchee-1.2.1, concise-0.9.0
UAT: l2ss-2.12.0rc14, batchee-1.2.0, stitchee-1.5.0, concise-0.10.0rc4

The latest versions in UAT have important changes/features—such as stitchee applying a dataset sorting, which enables consistent results—that the PROD versions don't yet have. Once the latest versions are deployed to PROD, I'd expect the tests to pass for PROD too.

[danielfromearth]

Versions in Production have been updated — so now stitchee and batchee are the same in both places!

PROD: l2ss-2.11.0, batchee-1.3.0, stitchee-1.6.1, concise-0.9.0
UAT: l2ss-2.12.0rc14, batchee-1.3.0, stitchee-1.6.1, concise-0.10.0rc5

[chris-durbin]

@ygliuvt - Is there anything that needs to be updated in this PR to ensure the SAMBAH tests run anytime STITCHEE or Batchee are deployed?

[ygliuvt]

@ygliuvt - Is there anything that needs to be updated in this PR to ensure the SAMBAH tests run anytime STITCHEE or Batchee are deployed?

The test needs to be added in config/services_tests_config_uat.json and config/services_tests_config_prod.json for batchee and stitchee services.

[flamingbear]

This mostly looks great with a couple of things to add. Like we mentioned in slack, I do have a couple of commits on my fork that I can push here or you can add them how you like.

As for the tests, I can't see any granules for that time period and you may need to add permissions for harmony_dev_user so that they can run automatically.

[danielfromearth]

@flamingbear , thanks for all of your efforts in reviewing this PR!

This mostly looks great with a couple of things to add. Like we mentioned in slack, I do have a couple of commits on my fork that I can push here or you can add them how you like.

As for the tests, I can't see any granules for that time period and you may need to add permissions for harmony_dev_user so that they can run automatically.

Since you've already made the changes, I'd say go ahead and push them from your fork. We can then assess if there's anything still remaining. I've reached out to some folks about the UAT visibility question, will update here when that is resolved.

[danielfromearth]

oh, I also want to note here that we discovered there is currently a difference between TEMPO's collections in UAT versus PROD — which happened because of recent reprocessing that only updated PROD granules. That needs to be addressed in order for the sambah regression tests to all pass, because there is currently one set of reference files — i.e., both UAT and PROD need to produce the same output. (This will be addressed in ticket TRADE-500)

[flamingbear]

@danielfromearth I pushed the changes to here. It should be ready for you to test. I still don't see any granules during the time period specified in the tests.

[danielfromearth]

I've been told that the permissions got out of sorts during some MMT updates, but will be worked on... standing by.

[flamingbear]

Are we abandoned or forgotten?

[ank1m]

The permissions were sorted out. But we found out that the files in UAT and PROD are not identical, so we were fixing it. They are supposed to by synced this week actually, and then Danny or I will test again. Sorry for delay!

[flamingbear]

@ank1m No problems, I just saw this was sitting for a while. Thanks for the update

[danielfromearth]

The files in UAT have been updated so they match those in PROD. Just ran the test notebook locally for both PROD and UAT, and the tests passed. So, I think this is ready to go 🤞

[flamingbear]

This looks great. I noted just a couple of errant outputs in your notebook that should be cleaned up. I have re-reviewed the new commits. I built the notebook and ran it against UAT and PROD and it looks good.

❯ ./run_notebooks.sh sambah
<snip>
running test with ghcr.io/nasa/regression-tests-sambah:latest
Test suite sambah succeeded
Tests completed (passed)

[danielfromearth]

@flamingbear, does this look good to approve and then merge now?

[flamingbear]

Thanks for cleaning that up. Merge it or if you can't I will.