Skip to content

Commit

Permalink
feat: provide schema functions tailored for testing
Browse files Browse the repository at this point in the history
- Add the `cast` and `defaults` schema functions.

- Refactor `error` and `error_or` into standalone functions that don't
  depend on the `Namespace` class.  The corresponding `Namespace`
  methods remain, but are now just thin wrappers.

- Interpret iterable schema arguments as pipelines.

- Get rid of the voluptuous dependency.

Fixes #16
  • Loading branch information
kalekundert committed Jun 10, 2022
1 parent db83f4c commit 275c79d
Show file tree
Hide file tree
Showing 22 changed files with 974 additions and 1,028 deletions.
6 changes: 4 additions & 2 deletions docs/api_reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,10 @@ API reference
parametrize_from_file.parametrize
parametrize_from_file.fixture
parametrize_from_file.Namespace
parametrize_from_file.voluptuous.Namespace
parametrize_from_file.voluptuous.empty_ok
parametrize_from_file.cast
parametrize_from_file.defaults
parametrize_from_file.error
parametrize_from_file.error_or
parametrize_from_file.star
parametrize_from_file.add_loader
parametrize_from_file.drop_loader
Expand Down
3 changes: 3 additions & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@
.. _schema: https://github.com/keleshev/schema
.. |PFF| replace:: *Parametrize From File*
.. |error| replace:: :py:func:`error <parametrize_from_file.error>`
.. |error_or| replace:: :py:func:`error_or <parametrize_from_file.error_or>`
.. |NS| replace:: :py:class:`Namespace <parametrize_from_file.Namespace>`
.. |NS_eval| replace:: :py:class:`Namespace.eval <parametrize_from_file.Namespace.eval>`
.. |NS_exec| replace:: :py:class:`Namespace.exec <parametrize_from_file.Namespace.exec>`
Expand All @@ -51,6 +53,7 @@
'python': ('https://docs.python.org/3', None),
'pytest': ('https://docs.pytest.org/en/stable', None),
'tmp_files': ('https://pytest-tmp-files.readthedocs.io/en/latest', None),
'nestedtext': ('https://nestedtext.org/en/stable', None),
'pandas': ('https://pandas.pydata.org/pandas-docs/stable', None),
}
autosummary_generate = True
Expand Down
43 changes: 18 additions & 25 deletions docs/exceptions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@ very common task. The usual way to parametrize tests for such functions is to
write separate tests for the valid and invalid inputs, but this adds
boilerplate and often means duplicating the code that helps setup the test.
Instead, this tutorial will show an elegant way to use the same test function
for all inputs. The key is a `helper function <voluptuous.Namespace.error_or>`
that:
for all inputs. The key is the |error_or| function, which:

- Produces a schema that accepts either an expected value or an expected error.
- When that schema is evaluated, creates a context manager that can be used to
Expand All @@ -24,33 +23,27 @@ To give a concrete example, we'll extend the ``Vector`` class from the
Each test case for this method specifies either an *expected* parameter or an
*error* parameter. The *expected* parameter is just a value, the same as all
the other test parameters we've seen in these tutorials. The *error* parameter
is special, in that it should specify a kind of exception to expect. There are
a few ways to do this (see |NS_error| for details), but the simplest is to give
a string that will evaluate to an exception type:
is special, in that it should specify the exception to expect. There are a few
ways to do this (see |error| for details), but the simplest is to give a string
that will evaluate to an exception type:

.. literalinclude:: exceptions/test_vector.nt
:caption: test_vector.nt
:language: nestedtext

To write the test function, we'll make use of `voluptuous.Namespace.error_or`.
This function builds a schema that will accept either an implicit *error*
parameter or whatever "expected" parameters are explicitly provided to it.
Either way, the test function will receive arguments corresponding to the error
*and* every "expected" parameter. The error argument will be a context manager
that will either check that the expected error was raised (if an error was
specified) or do nothing (otherwise). The "expected" arguments will be either
be passed directly through to the test function (if no error was specified) or
be replaced with `MagicMock <unittest.mock.MagicMock>` objects (otherwise).
The purpose of replacing the "expected" arguments with `MagicMock
<unittest.mock.MagicMock>` objects is to help avoid the intended exception from
getting preempted by some other exception caused by an unspecified expected
value.

.. note::

Functions like `voluptuous.Namespace.error_or` are not currently provided for
any schema libraries except voluptuous_. If you implement something similar
for another schema library, I'd be happy to accept a pull request.
To write the test function, we'll make use of `Namespace.error_or`. This
method returns a schema function that will expect every set of test parameters
to specify either an *error* parameter or whatever "expected" parameters are
listed as arguments (just *expected* in this case). Either way, the test
function will receive arguments corresponding to the error *and* every
"expected" parameter. The error argument will be a context manager that will
either check that the expected error was raised (if an error was specified) or
do nothing (otherwise). The "expected" arguments will be either be passed
directly through to the test function (if no error was specified) or be
replaced with |MagicMock| objects (otherwise). The purpose of replacing the
"expected" arguments with |MagicMock| objects is to help avoid the intended
exception from getting preempted by some other exception caused by an
unspecified expected value.

This sounds complicated, but in practice it's not bad. Hopefully the following
example code will help make everything clear:
Expand All @@ -60,4 +53,4 @@ example code will help make everything clear:

A shortcoming of this example is that it does not show how to check that the
exception has the expected error message. For more information on that,
consult: `voluptuous.Namespace.error_or`
consult the documentation for the |error| function.
16 changes: 8 additions & 8 deletions docs/exceptions/test_vector.py
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
import parametrize_from_file
from pytest import approx
from voluptuous import Schema
from parametrize_from_file.voluptuous import Namespace
from parametrize_from_file import Namespace, cast

with_vec = Namespace('from vector import *')

@parametrize_from_file(
schema=Schema({
'given': with_vec.eval,
**with_vec.error_or({
'expected': with_vec.eval,
}),
}),
schema=[
cast(
given=with_vec.eval,
expected=with_vec.eval,
),
with_vec.error_or('expected'),
],
)
def test_normalize(given, expected, error):
with error:
Expand Down
19 changes: 16 additions & 3 deletions docs/optional_params.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,19 +28,26 @@ The *schema* argument to `parametrize_from_file` can be used to fill in
unspecified parameters with default values. This takes advantage of the fact
that, although every set of parameters needs to have all the same keys, the
schema is applied before this check is made. So it's possible for the schema
to fill in any missing keys. The following example uses voluptuous_, but any
schema library will have support for this feature. First, the parameter file:
to fill in any missing keys. In fact, *Parametrize From File* comes with a
`defaults` function that does exactly this. The following example shows how it
works. First, the parameter file:

.. literalinclude:: optional_params/test_vector_schema.nt
:caption: test_vector.nt
:language: nestedtext

Note that *unit* and *magnitude* are only specified for one test each. The
following schema fills in the defaults:
following schema takes care of evaluating the snippets of python code and
filling in the missing defaults:

.. literalinclude:: optional_params/test_vector_schema.py
:caption: test_vector.py

It's significant that the defaults are specified after the cast functions. If
they were specified before, they would be processed by the cast functions. In
this case, that means they would need to be strings containing python code.
Sometimes that's what you want, but not here.

Note that the test function uses degrees as the default unit, while the
function itself uses radians. This is both a good thing and a bad thing. It's
good that our tests will be robust against changes to the default unit. But
Expand Down Expand Up @@ -68,3 +75,9 @@ parameter in the NestedText_ file, though.

.. literalinclude:: optional_params/test_vector_kwargs.py
:caption: test_vector.nt

It's a little dangerous to set the default *kwargs* value to a mutable object
like an empty dictionary. Any changes made to this dictionary will persist
between tests, possibly leading to confusing results. You can avoid this issue
by setting the default to `None` and replacing it with the desired value within
the test.
82 changes: 0 additions & 82 deletions docs/optional_params.rst.bkup

This file was deleted.

18 changes: 11 additions & 7 deletions docs/optional_params/test_vector_kwargs.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,22 @@
import vector
import parametrize_from_file
from pytest import approx
from voluptuous import Schema, Optional
from parametrize_from_file.voluptuous import Namespace
from parametrize_from_file import Namespace, cast, defaults

with_math = Namespace('from math import *')
with_vec = with_math.fork('from vector import *')

@parametrize_from_file(
schema=Schema({
'angle': with_math.eval,
Optional('kwargs', default={}): with_math.eval,
'expected': with_vec.eval,
}),
schema=[
cast(
angle=with_math.eval,
kwargs=with_math.eval,
expected=with_vec.eval,
),
defaults(
kwargs={},
),
],
)
def test_from_angle(angle, kwargs, expected):
actual = vector.from_angle(angle, **kwargs)
Expand Down
20 changes: 12 additions & 8 deletions docs/optional_params/test_vector_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,23 @@
import vector
import parametrize_from_file
from pytest import approx
from voluptuous import Schema, Optional
from parametrize_from_file.voluptuous import Namespace
from parametrize_from_file import Namespace, cast, defaults

with_math = Namespace('from math import *')
with_vec = with_math.fork('from vector import *')

@parametrize_from_file(
schema=Schema({
'angle': with_math.eval,
Optional('unit', default='deg'): str,
Optional('magnitude', default='1'): with_math.eval,
'expected': with_vec.eval,
}),
schema=[
cast(
angle=with_math.eval,
magnitude=with_math.eval,
expected=with_vec.eval,
),
defaults(
unit='deg',
magnitude=1,
),
],
)
def test_from_angle(angle, unit, magnitude, expected):
actual = vector.from_angle(angle, unit=unit, magnitude=magnitude)
Expand Down
27 changes: 16 additions & 11 deletions docs/python_snippets.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ the actual ``Vector`` object in the test function, as we did in the
``Vector(1, 0) + Vector(0, 1)``.

- We can test invalid inputs, e.g. ``None``, and make sure the proper exception
is raised. See `exceptions` for more information on this topic.
is raised. See the `exceptions` tutorial for more information on how to do
this elegantly.

Below is the parameter file from the `getting_started` tutorial rewritten using
python syntax. Note that the file is also rewritten in the NestedText_ format.
Expand Down Expand Up @@ -94,10 +95,11 @@ variable:
Schema argument: Be careful!
============================
Be careful when using |NS_eval| and especially |NS_exec| with the *schema*
argument to `parametrize_from_file`. This can be convenient, but it's
important to be cognizant of the fact that schema are evaluated during test
collection (i.e. outside of the tests themselves). This has a couple
consequences:
argument to `parametrize_from_file` (e.g. via `cast`). This can be a
convenient way to clearly separate boring type-munging code from interesting
test code, but it's important to be cognizant of the fact that schema are
evaluated during test collection (i.e. outside of the tests themselves). This
has a couple consequences:

- Any errors that occur when evaluating parameters will not be handled very
gracefully. In particular, no tests will run until all errors are fixed (and
Expand All @@ -118,18 +120,21 @@ example, here is a version of ``test_dot()`` that uses a schema:

A few things to note about this example:

- In this case, using a schema isn't really an improvement over processing the
expected value within the test function itself, like we do for the *a* and
*b* parameters. When it is advantageous to use schema is when you have many
test functions with similar parameters. Sharing schema between such
functions often eliminates a lot of boiler-plate code.

- This also is a good example of how the |NS| class can be used to control
which names are available when evaluating expressions. Here we make two
namespaces: one for just built-in names (including all the names from the
math module), and another for our vector package. This distinction allows us
to avoid the possibility of evaluating vector code from the schema.

- This example uses `voluptuous.Namespace` instead of just |NS|. These two
classes are largely equivalent, but `voluptuous.Namespace` includes some
voluptuous_-specific tweaks, e.g. raising the kind of exceptions expected by
voluptuous_ if `python:eval` or `python:exec` fail. Currently there are no
custom |NS| classes for any other schema libraries, but if you write one, I'd
be happy to accept a pull request.
- This example uses `cast`, which is one of the schema functions provided by
*Parametrize From File*. This function is commonly used in conjunction with
`defaults`, |error|, and |error_or|.

- The ``Vector`` expressions used in these examples are actually a bit of a
grey area, because they're simple enough that (i) they're unlikely to break
Expand Down
8 changes: 2 additions & 6 deletions docs/python_snippets/schema/test_vector.py
Original file line number Diff line number Diff line change
@@ -1,18 +1,14 @@
import vector
import parametrize_from_file
from pytest import approx
from voluptuous import Schema
from parametrize_from_file.voluptuous import Namespace
from parametrize_from_file import Namespace, cast

# Define these objects globally, because they will be useful for many tests.
with_math = Namespace('from math import *')
with_vec = Namespace(with_math, 'from vector import *')

@parametrize_from_file(
schema=Schema({
'expected': with_math.eval,
str: str,
}),
schema=cast(expected=with_math.eval),
)
def test_dot(a, b, expected):
a, b = with_vec.eval(a, b)
Expand Down
Loading

0 comments on commit 275c79d

Please sign in to comment.