Looking for information on how to run tests for workflows locally

[thawn]

I followed the tutorial on how to add tests for workflows now, before committing the tests, I would like to run them locally on my machine. However, I don't know how.

Is there already documentation for that?

If yes, it would be great to link to that documentation in the tutorial mentioned above

If not, it would be great to add that information.

[hexylena]

Hi @thawn you can indeed run them locally but I would not recommend it. Here is how you can run them against an existing galaxy server, if you like: https://planemo.readthedocs.io/en/latest/running.html#workflow-execution-against-an-external-galaxy

but in general our plan is that the GTN will run the tests for you, so you don't have to configure it.

[thawn]

but in general our plan is that the GTN will run the tests for you, so you don't have to configure it.

My user experience with the GTN and the auto generated tests was suboptimal so far:

1. Initially, it looks like the tests are successful, because it only runs [CI] Pull Request Labeler / triage (pull_request_target) only much later, I get notified that the linting tests failed.
2. The linting tests are run with so much delay, that debugging is extremely painful
3. The auto-generated files (generated following this tutorial are named wrongly

[thawn]

Here is how you can run them against an existing galaxy server, if you like: https://planemo.readthedocs.io/en/latest/running.html#workflow-execution-against-an-external-galaxy

I don't want to run the workflow itself, I want to run the linter and the automatic tests for the workflow.
Essentially, I want to do locally what the Github CI does automatically in this step: [CI] GTN Tutorial Linting / lint (pull_request)
The main reason for wanting to run this locally is that the CI runs the test so infrequently that debugging is very painful and slow.

[hexylena]

Hi @thawn I'm sorry you're having a sub-optimal experience.

Initially, it looks like the tests are successful, because it only runs [CI] Pull Request Labeler / triage (pull_request_target) only much later, I get notified that the linting tests failed.

This is due to GitHub's policies, something we have no control over -- new contributors have to have their tests approved.

The linting tests are run with so much delay, that debugging is extremely painful

The linting (specifically) generally run in <1m30s which all considered is pretty good, considering that we have to clone the GTN (~1GB). The 'accessibility' linting of course takes longer since we need to pull in a lot more which is unfortunate.

The auto-generated files (generated following this tutorial are named wrongly

This can be, yes, we'll look into that. I've seen PRs with all four variants of file extensions contributed by planemo users, I don't know how we got 4 variants of the file extension but we decided to only accept one to make some of our tooling simpler.

I don't want to run the workflow itself, I want to run the linter and the automatic tests for the workflow.

ah that wasn't clear! I think I was mislead by linking to the galaxy workflow testing.

You can of course just look in .github/workflows/ci-main.yml to see what we execute. The main things you'd want to run are

      make check-diffs
      make check-frontmatter 
      make check-contributors
      find topics/ -name tutorial.md | xargs -n 1 python bin/check-broken-boxes.py
      bundle exec ruby bin/lint.rb --format rdjson

but do note that those linters run for every file, as they run very quickly, and we use review-dog's capabiliities to filter the reports by what was touched in a PR.

[thawn]

@hexylena thanks a lot for the thorough explanation! I will try the local testing and get back to you.

Just one small additional comment:

The linting (specifically) generally run in <1m30s which all considered is pretty good, considering that we have to clone the GTN (~1GB). The 'accessibility' linting of course takes longer since we need to pull in a lot more which is unfortunate.

1m30s would be totally fine. However, this is only the actual run time. It takes many hours until the run is even started. For example, in pull request #4493 It took ~ 20 hours between the commit and running the linting. You can see the time difference in the details of each CI step (linked below):

• [CI] Pull Request Labeler / triage (pull_request_target) (executed yesterday)
• [CI] GTN Tutorial Linting / lint (pull_request) (executed today ~11:00)

[hexylena]

It takes many hours until the run is actually started.

Ahh this is again due to GitHub, someone has to see and press the button which indeed happens on the scale of hours :( that's a really unpleasant contribution flow, agreed!

I'll try and think about what workarounds we can find there. Once you've had one PR merged, the tests will run automatically every time you push to the repository.

[thawn]

ah ok that makes sense. Thanks for the explanation :-)