Report a list of violations, instead of just the first violation

[velle]

Normally (at least in my experience) each unit test throws an exception at the first occurrence of a violation. But there are situations where it makes sense to generate a list of violations and put the entire list in the test report.

Is that something that is sometimes done? Are there any design patterns, best practices, plugins, guides on this?

The best I have managed this far is compiling a message string containing violations, and then in the end raising an exception containing this message. See a dummy example here: https://gist.github.com/velle/a5735750f7a5383cf650e93f4dd68207.

[RonnyPfannschmidt]

Subtests and deferred assertions are utilities to archive this

It's not a common flow as within a unittests the common structure is that a failed assertion means the test cannot continue

[velle]

That's a linter, not a testrun

A testrunner is not good to mend into a linter

I believe I know what a linter is; it's for validating my own code (code in the repo) against certain rules, including style guides. Am I wrong?

In my case, the XML is generated by a library; ie a separate unit that calls the library, and the library saves test.xml to disk. So I would not normally consider that linting. Or do you have a different understanding of linting? :)

[RonnyPfannschmidt]

As far as I understand, you want to assert that linting did not produce results for a given content

[RonnyPfannschmidt]

As far as I'm concerned, a linter is a tool that given content points out potential problems with it

Even a spellchecker can be a linter

[velle]

As far as I'm concerned, a linter is a tool that given content points out potential problems with it

As far as I understand, you want to assert that linting did not produce results for a given content

Got it. Yes. A fixture calls my "library under test", and that produces a file text.xml. Subsequently there are several unit tests that assert all sorts of things on this file. I guess all these tests could be considered linting. Until now I have called it file format validation. For some of those tests I prefer that the output is a list of violations, instead of just a "fail".

You said:

That's a linter, not a testrun
A testrunner is not good to mend into a linter

I'm not sure what I am doing wrong and why it's problematic. Not trying to argue; I would just really like to understand :) Your feedback is appreciated!

[RonnyPfannschmidt]

So have your function return a list of violations and make assertions on those

See the tests of the internal pytest junitxml plugin

Plugins like pytest-flake8 may also inspire