From f42e1e5000e4b92d8491730bf0efbc80dd139821 Mon Sep 17 00:00:00 2001 From: Mikyo King Date: Tue, 1 Oct 2024 13:04:50 -0600 Subject: [PATCH 1/2] docs: testing philosophy --- docs/internal_docs/TESTING.md | 66 +++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 docs/internal_docs/TESTING.md diff --git a/docs/internal_docs/TESTING.md b/docs/internal_docs/TESTING.md new file mode 100644 index 0000000000..fba22acf77 --- /dev/null +++ b/docs/internal_docs/TESTING.md @@ -0,0 +1,66 @@ +# Testing + +## Why you should care + +Hot take: your software source code is not the asset that you might think it is. It is a liability. More code means more liability. + +“Technical debt” is not like a loan. Loans have known repayment schedules and amounts. Bugs arise from code at unknown times with unknown quantities of effort that must be paid to correct them. This is not like a loan. “Tech debt” is a misnomer - it is actually an un-hedged call option, a kind of contingent liability that must be paid at an unknown time at an unbounded cost. + +In the world of software, the goal is working features, not lines of source code. Working tests that prove the continued existence of those features behave like assets. Code that produces those features is a liability. + +Reduce your risk; balance code with tests. + +## Testing Classifications + +Testing can be classified into 4 types + +- End to End: A helper robot that behaves like a user to click around the app and verify that it functions correctly. Sometimes called "functional testing" or e2e. +- Integration: Verify that several units work together in harmony. +- Unit: Verify that individual, isolated parts work as expected. +- Static: Catch typos and type errors as you write the code. + +We cover these 4 types of tests + +- End to end: Playwright +- Integration: pytest + DB +- Unit: pytest, Jest +- Static: mypy, TypeScript + +## Integration Testing + +> [Write tests, Not too many. Mostly integration](https://kentcdodds.com/blog/write-tests) + +We spin up the `phoenix server` with both `sqlite` and `postgres` for our integration tests. These tests are written in python. + +### Why IT? + +- Testing service boundaries (`server` to `database`, `server` to `smtp` etc.) +- RBAC sanity checks +- Ensure the server can boot in different environments + + +## End to End Testing + +We write end to end tests using [Playwright](https://playwright.dev/), a browser automation library that allows you to write tests that interact with your web application in a browser. + +### Why E2E Testing? + +The most important thing about E2E tests is that they are the closest thing to a real user. They are the most likely to catch bugs that unit tests and integration tests might miss. + +- Browser testing ensures the product is usable as intended (flows, a11y) +- Regression testing (ensure existing functionality works and stays working) +- [future] visual regression testing + +### E2E Testing Best Practices + +- Use stable selectors (human readable text or `data-testid`) See [Making tests resilient to change (data-testid)](https://kentcdodds.com/blog/making-your-ui-tests-resilient-to-change) +- Avoid explicit timers. The tests should try to emulate a real user. You should try to wait for elements to show or for network calls to complete +- Don't hard code `environment` specifics in the tests (strings and IDs). + + +## Resources + + +- [Code is not an asset](https://robinbb.com/blog/code-is-not-an-asset/) +- [Testing trophy and testing classifications](https://kentcdodds.com/blog/the-testing-trophy-and-testing-classifications) +- [Making tests resilient to change (data-testid)](https://kentcdodds.com/blog/making-your-ui-tests-resilient-to-change) From 7cac1590b2f9866a197a7c5040fcc9dc08116cd6 Mon Sep 17 00:00:00 2001 From: Mikyo King Date: Tue, 1 Oct 2024 13:06:33 -0600 Subject: [PATCH 2/2] cleanup --- docs/internal_docs/TESTING.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/internal_docs/TESTING.md b/docs/internal_docs/TESTING.md index fba22acf77..f0196267d5 100644 --- a/docs/internal_docs/TESTING.md +++ b/docs/internal_docs/TESTING.md @@ -60,7 +60,6 @@ The most important thing about E2E tests is that they are the closest thing to a ## Resources - - [Code is not an asset](https://robinbb.com/blog/code-is-not-an-asset/) - [Testing trophy and testing classifications](https://kentcdodds.com/blog/the-testing-trophy-and-testing-classifications) - [Making tests resilient to change (data-testid)](https://kentcdodds.com/blog/making-your-ui-tests-resilient-to-change)