Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand testing of parse_access_ncfile to include correctness checks #187

Closed
marc-white opened this issue Aug 26, 2024 · 3 comments · Fixed by #232
Closed

Expand testing of parse_access_ncfile to include correctness checks #187

marc-white opened this issue Aug 26, 2024 · 3 comments · Fixed by #232
Assignees
Labels
enhancement New feature or request

Comments

@marc-white
Copy link
Collaborator

Is your feature request related to a problem? Please describe.

The current version of tests/builders/test_builders checks BaseBuilder.parse_access_ncfile against a fixed set of files and expected returns, i.e., regression tests it - however, it doesn't check the available file to see if the data returned are actually correct.

Describe the feature you'd like

Given we have access to the test files (and they're small, so no particular IO penalty), we could expand the test to check the correctness of the parse_access_ncfile return. The idea would be to look at each element of the parse_access_ncfile return, manually extract the related information from the test file, and compare them.

Ideally, this should be done in a slightly different way to how parse_access_ncfile does it, in order to provide a redundant sanity check.

Describe alternatives you've considered

Nil.

Additional context

I created an issue (#186 ) that suggests we should change the return of parse_access_ncfile from a tuple to a dict - doing so would make implementing this feature request easier.

@marc-white marc-white added the enhancement New feature or request label Aug 26, 2024
@charles-turner-1
Copy link
Collaborator

@marc-white are you working on this actively/ have you made any progress on this? Thinking of picking this up - I'd like to expand the test suite before continuing with #153.

@marc-white
Copy link
Collaborator Author

@charles-turner-1 no I'm on other things at the moment, so feel free.

@charles-turner-1 charles-turner-1 self-assigned this Oct 21, 2024
@charles-turner-1
Copy link
Collaborator

I've added a bunch of tests related to this on branch 187-correctness-checks.

As it stands:

  • Updating intake-esm from 0.7 (current pinned version) => to 2.0.7 (most recent) seems to have no impact on results.
  • Added tests are extensions of test_builders/test_parse_access_ncfile.
  • Of the added tests, we get a failure on $ROOT/tests/data/access-om2/output000/ocean/ocean_grid.nc when we use a version of intake-esm that doesn't support searching for coordinate versions (eg. the most recent version, 2024.2.6). All other tests pass. The source of the error is expected & easily understood, but going to be a pain to reliably identify ahead of time so we can tell the test whether to xfail or not.

I don't think these changes are ready for a (draft) PR yet, so I'm leaving this open & pre-draft for now.

Next steps:
We need to extend our testing framework to test against

  1. Different versions of intake-esm (source of the error)
  2. Different versions of intake.

The obvious way to get this working is to use tox to test against the multiple dependency versions, so I'm going to get that configured and once we have that working as expected I'll open a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants