Autosnaps Spec

[emilgoldsmith]

This is the spec we discussed internally in Aiden for Autosnaps. The first comment below here will be my original design issue created, the next @lingz's reply, and third my reply to that. After that any comments will be original to this issue

[emilgoldsmith]

Motivation

Snapshotting can be a very useful testing tool, but especially gathering fixtures can be time consuming. We would like to automate this.

Proposed solution

We introduce AutoSnaps! (name up for discussion if you don't like it)

AutoSnaps' core functionality would consist of

1. A Higher Order Function (HOF) named withSnapshot (I like the with naming convention for HOFs). Choosing to wrap any function will automatically handle saving fixtures for you (nuances of this to be discussed later). An environment variable called AUTOSNAPS will change the behaviour of this functionality.
2. A simple CLI that can do things like autosnaps test that runs the tests on all your snapshots, and other commands like delete and add

Finer points

File names

I propose that we store files in a __snapshots__ root directory like the current library we do now. And we then split them into two types of files

1. originalFilename.fixture.js. This is where we'll store the fixtures saved in withSnapshot
2. originalFilename.snapshot.js. This is where we'll store the snapshotted output of the functions

Stripping extensions

A point of discussion is whether and/or how we should strip extensions of original filenames. Since we specifically had some pain with snap-shot-it treating untranspiled Typescript and the corresponding JS files like two different sources of snapshots. I think the most straight forward approach is to simply strip everything from the last ., but are there any realistic, probable edge cases I'm missing by doing that which would ruin some filenames? We could of course also just not strip anything, but that leaves us with the problem I mentioned above that we encountered with snap-shot-it.

Having two types of files

I have currently proposed having two types of files, but this can of course also be up for discussion as they could also easily be consolidated into one file per original file that just stores both fixtures and snapshots. I think it might be slightly easier to read the files manually when it's split, since fixtures and snapshots can be very big, and often one can be a lot bigger than the other. This is mostly just my initial instinct though and is very up for discussion

Locating original function

One of the biggest challenges I have come up against while designing this library is how to know where the original function is from inside the called HOF. My current thoughts on it are:

• I think we would need to use something like The Babel Parser to parse where the original function is imported from
  • Pros:
    1. I have already worked with the Babel Parser with some of my other open source projects as I was working on a JS linter plugin.
    2. It'll be pretty performant, as we won't be evaluating any JS/TS, simply doing an O(n) parsing of the file
  • Cons:
    1. Would not work out of the box with import plugins or aliases like we for example use with @aiden/bot.
       • Possible solution: We could add some kind of import configuration like Typescript and other libraries have so aliases could be specified
         • Con: It seems overkill for such a small library to need to have configuration, and it would be great for it to just work out of the box with everything
    2. We would have to support sourcemaps or something similar in order to support running it on bundled code such as after a Webpack bundling

AUTOSNAPS environment variable

I'm thinking that we will be able to set this environment variable to two different names

1. AUTOSNAPS="update [...function_specifiers]"

   • If no function specifier is given it will update all fixtures, and (with a warning) delete any previous fixtures, including any functions that previously had several fixtures attached to it. If function specifiers are given then it will be restricted to only updating matching functions

2. AUTOSNAPS="add [...function_specifiers]"

   • If no function specifier is given it will add an extra fixture to all functions. If specifiers are given it will only add new fixtures to these functions.

Where square brackets means optional argument, and a function_specifier is either a string with no @ included, in which case it is an unambiguous function name. If it includes @ in the string, the file name where the function is located is also specified as in the following example func@filename.ext, or func@parent_directory/filename.ext where after the @ will simply be matched to the end of the path of the file we're currently located in (this will also need us to read sourcemaps if the code is bundled).

AutoSnaps CLI

I think we need a simple binary with a few core features in order to manage AutoSnaps' features. I propose the following features / commands:

1. autosnaps test. This would run all the fixtures and snapshots in a usual way (if no snapshot exists create it, if it conflicts fail, and maybe add an interactive mode that allows you to overwrite snapshots on the fly). I think it would also be important for useability that we somehow added an integration into Mocha / Jest etc. This is for things like code coverage, and not having to run an extra command. I would have to look more into the possibilities of this, but even with that I still think autosnaps test should be an available command
2. autosnaps manage (or autosnaps delete, but I think that sounds more dangerous). Especially if you start getting multiple fixtures on some of your functions, and you don't want to go in and manually mess with the fixture files, I think there should be a simple interface that allows you to specify a function, see the fixtures and delete old ones.
3. autosnaps add. This should give the possibility to manually add a fixture through copy pasting a JSON probably, or maybe you could also specify a .json file as an argument. In case you have your own fixture you want to add and again don't like messing manually with the autosnaps files
4. autosnaps list. This should list all the functions that we have currently saved fixtures for, and possibly also how many fixtures each one has, maybe with the possibility to view them.

I think all of these should each be pretty simple to implement, especially since there are good npm libraries out there that deal with CLI interfaces.

Overall Con

I also realized that this will only (at least in a straight forward manner) work with Node.js. Unless we put more thought into it and maybe somehow setup a server on localhost and communicate from the browser to that server within AutoSnaps will we be able to write on the server from within browser JS.

[emilgoldsmith]

Hey Emil,

Sorry for delay, been super packed round these days :)

It looks very good, I think you should definitely get started. I've sure you have thought of many of these, but here's some working comments:

1. Higher order function (or JS annotation) sounds great, though does the annotation need to stay forever, or can we just attach it once, and record, and then delete it?
2. For testing, isntead of doing a new CLI tool, I would recommend having a JS API so you can just assert its truthiness inside Mocha or whatever test framework you are using
3. Snapshotting is probably you can use an existing library.\
4. The most interesting part is how to configure saving the incoming objects. Do you save everything? First 1, First n? Probably configurable. Also: you might want a filter() function, so that we can blank out secrets and not commit them into github
5. Think about how tests can survive refactoring, perhaps they don't need to and we can just regenerate snapshots, or maybe we can be smart about this
6. Think about when snapshots are off, it would be great for productivity if we could just assert the new results, should be the new snapshot result. Maybe you can just use existing snapshot lib, but an interactive mode for this would be pretty cool if possible.
7. Seems the focus of this is about how to capture input effectively, the output put is mostly a solved problem. If you want a reach goal, you could measure code coverage from different inputs and try maximize it :).

Very looking forward to what comes out, it could save us (and everyone else) a lot of time!

[emilgoldsmith]

No worries, good luck on all the stuff you're working on!

1. I think it should be easy to allow deleting the function, but I think it's interesting to think about if you want that feature? The fact that you can see where you're snapshotting from in your code might be nice, and it can be a kind of "visual human code coverage", seeing that this function is snapshot tested from within the code? I can see people arguing that it's clutter though, so its not like I'm 100% set on not allowing this
2. Yeah we definitely want it to be very seamlessly integrateable with Mocha, Jest etc.
3. Yep I'm planning on using the same dependencies that snap-shot-it uses
4. Yeah for now I was thinking of it just being the first one, but letting that be configurable at some point is definitely worth thinking about
5. Good point to consider, surviving light refactors would definitely be an awesome feature
6. Hmm can you elaborate on that point? I don't think I understand what you mean. Are you talking about interactively updating snapshots when a snapshot test fails? In that case I definitely agree
7. Yeah I agree capturing input is the challenge here. Looking at code coverage could be a super cool goal. And make this a very useful tool indeed. I love that idea.

I'll try and slowly get started on something this week!

[lingz]

Hey @emilgoldsmith I was just reminded of Python Runtime type annotaters, which watch the code run in production and generate static typing afterwards, e.g.: https://github.com/Instagram/MonkeyType

Maybe could take them as inspiration for design and user experience?

[emilgoldsmith]

Thanks, I'll try looking into it!