@xstate/test Add methods for more comprehensive tests

[eliasm307]

Problem

I just want to start by saying the whole xstate project is really cool and it is already very useful, including @xstate/test where some of the big selling points, atleast for me, are that you can

• generate tests based on paths through a test model ie model based testing
• use it on projects that dont use xstate in the production code

The first point is particularly impressive and its a significant point of interest when I'm explaining xstate and @xstate/test to people, however I feel it is currently limited. So currently test generation is in terms of using methods like getSimplePathPlans/getShortestPathPlans (still not sure what the difference is between these) on a test model, which from my understanding generates plans to reach every state in the model. This might be suitable for some use cases but when I have tried to use this on a realistic project I run into some issues such as:

• the test plans only seeming to go "forwards", ie if there are events that result in going back to a previous state then it doesnt generate plans for this, so the plans dont cover realistic usage (ie some actions an actor can take in certain states are not tested) and focus more on "ideal" paths.
• many test paths being generated, however some of the paths being repeated in other paths and so making them redundant. An example is when I used this for a wizard dialog and it generated tests to get to the first step of the wizard, then other tests to get to the second step (which covers the first step), then other steps to get to the third step (which covers the first and second steps) etc.

I understand the limitations of what paths should be tested might be from some optimisation to reduce redundant tests, however this usually means the edge cases or more complex behaviours need to be tested manually, e.g. by specifying the specific sequence of events for the behaviour but then the tests become "manual" and the promises of auto test generation feel less true. This also produces another issue where the tests are partially automated, so its up to a dev to then investigate and understand all the cases the generated tests are covering and create manual tests for the gaps, which is something that then needs to be managed as a project evolves and effectively means 2 sets of tests need to be reviewed with changes.

Suggestions

My suggestions regarding this, are to provide more options for how tests are generated which will allow the test generation to be useful for more use cases and allow combining different test generation strategies to vary the test behaviour. My ideas for test generation methods are:

getMinimumPlansToUseAllEvents

ie generate the minimum number of plans in order to use all events available in the model or at least some given events (with a limit on how many steps it can take in the model or how many times it can re-visit a single state etc).

This could also have additional configuration options like specifying the minimum/maximum number of times it should test specific events and that it should prefer the least visited paths where the model has complex paths that branch and loop back etc. to force it to discover and use different paths to achieve the targets, but also allow the testing time/effort to be limited if required for large projects. This would make sure every "action" an actor can take is tested in every state, but aim to minimise the number of tests that are generated.

I also think this behaviour of traversing the test model with a goal of using all the events would cover more realistic usage.

getMinimumComprehensivePlans

ie generate the minimum number of plans in order to visit every state and use every event/path in the model (with a limit on how many steps it can take in the model). Similar idea to the above but in this case the minimum number of plans are generated with the goal of using/visiting all the events and states in the model, with configuration options to define the maximum parameters in the attempt to do this.

Additonal Comments

For both of the methods above, if the plans were not able to achieve the targets of visiting all the states or using all the events in relevent states then the model should report this either via a warning or a test failure. This will mean a clear list of areas which were not covered by the automated tests is available when creating the manual tests to cover these gaps, and new warnings/failures appear as the system changes and new gaps appear.

I think if @xstate/test can get better at the test generation and utilised the "finite" aspect of state machine states and events a bit more, it will be an even more awesome standalone tool, and I look forward to the development.

Please let me know if anything doesnt make sense.

[davidkpiano]

What do you think about the upcoming changes here? Notably, you can create your own plan generators: #3036

[eliasm307]

@davidkpiano yes that's what I was assuming, but this would mean it's up to the dev to make sure there are no duplicate plans. I was thinking it would be better dx if you can provide multiple path generators then the de-duplication is done internally before producing the output paths. What do you think?

[davidkpiano]

@davidkpiano yes that's what I was assuming, but this would mean it's up to the dev to make sure there are no duplicate plans. I was thinking it would be better dx if you can provide multiple path generators then the de-duplication is done internally before producing the output paths. What do you think?

We have to be careful with implicit behavior like this; for instance, a dev might expect that if M + N plans are passed into .testPlans(...), then M + N plans are tested, rather than fewer than M + N plans.

IMO this should be explicit:

// TENTATIVE API
import { simplifyPlans } from '@xstate/test';

// ...

// Removes duplicate plans
// Also removes shorter plans that are subsets of longer plans
testModel.testPlans(simplifyPlans(plansA.concat(plansB)));

[eliasm307]

Yeah that works too, the main thing for me is that the library provides options to specify this.

Potentially there might be various strategies for how tests should be "simplified" or other criteria for selecting tests so I'm wondering if a better interface should be available to pipe/compose the different strategies or if its up to the dev how they do this, however I feel the more things a dev needs to figure out an do the more the dx suffers.

For this I keep thinking of something like how Rx works with operators. Its quite a nice experience picking and defining the behaviour you want where the library handles the details, but I don't know how much of that is/should be applicable here.

I'm also wondering if the filtering isn't too important and can be implemented as part of the test generators instead, but then this might mean devs needing to make custom test generators most of the time to get certain behaviour.

I think the ideal would be if both the generators and the test filtering strategies were composable and there was one standard library of functions devs could select from. This would hopefully make the need for writing custom test generators/filters rare and hopefully be a good DX

[davidkpiano]

Can you share some (pseudo)-code demonstrating what kind of API you're after?

IME, the more "DX-friendly" you attempt to make an API early on, the higher chance you have of accidentally locking the API into inflexible behavior that makes DX worse for other use-cases.

So IMO the best thing to do is to have a completely flexible API, and let emergent + common use-cases dictate how the API can make things easier. The most important thing right now is that @xstate/test allows this flexibility; improved DX (which is subjective) comes later.

[eliasm307]

Fair points, so what I think is that there are 2 main concerns regarding this, ie:

• creating test plans
• selecting the minimum number of suitable test plans to meet some requirements

Based on this I think there should be 2 families of entities each focussing on one of these concerns and can be composed together for different behaviour. I've made some definitions of what I think would be useful and examples of how they could be used, which you can find in this gist, where:

• First section is definitions,
• Second section is example pre-made implementations that could be provided out of the box
• Third section is example usages.

I've also added comments on my thought process so I hope its clear, but let me know if you have any questions.

The main aspects are:

• A TestStrategy builder class, which represents the definition of the strategy to be used to create and select test plans, this can be defined by adding:
  • Test plan creators - which allow focusing on specific types of paths or just traversing every possible path, the choice here allows the test generation to be optimised to only consider certain types of paths through a model
  • Test plan criteria evaluators - which define the logic for determining if test plans should be discarded/kept, and different combinations of these produce different test generation behaviours to meet requirements
• To use a defined TestStrategy, a specific test model needs to be provided which it will execute the plan creators and criteria evaluators on to create a set of plans that meet the criteria
• Configured TestStrategy objects are quite portable and flexible in that they can be executed on different test models, can be shared, can be extended etc

This configurability allows complex test generation behaviour to be defined in a readable and flexible manner imo.

However, I can imagine some combinations of test generators and criteria that might cause issues but if people can see the steps the generation takes then this should be easy to debug and fix... hopefully.

What do you think?