Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update testing dev docs to GitHub Actions #1251

Merged
merged 3 commits into from
Oct 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions dev/testing/en/continuous-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,29 +4,29 @@ title: Continuous Integration - Testing - OJS/OMP

# Continuous Integration

We use [Travis CI](https://travis-ci.com) for Continuous Integration (CI) testing. Integration and unit tests are run against every commit and pull request to help catch regressions before they are released.
We use [GitHub Actions](https://github.com/features/actions) for Continuous Integration (CI) testing. Integration and unit tests are run against every pull request to help catch regressions before they are released.

CI tests are configured to run the tests against multiple PHP versions and databases. The configuration details are stored in the `.travis.yml` file in the application's root directory.
CI tests are configured to run the tests against multiple PHP versions and databases. The configuration details are stored in `.github/workflows/main.yml` for each application.

![OJS test results in Travis.](./travis-overview.png)
![OMP test results in GitHub.](./github-actions-overview.png)

View the tests for [OJS](https://travis-ci.com/pkp/ojs/), [OMP](https://travis-ci.com/pkp/omp/) and [OPS](https://travis-ci.com/pkp/ops/).
View the tests for [OJS](https://github.com/pkp/ojs/actions), [OMP](https://github.com/pkp/omp/actions) and [OPS](https://github.com/pkp/ops/actions).

## Tests for pull requests

Travis will run tests against every pull request to OJS or OMP. The tests are listed in the checks at the bottom of the pull request.
GitHub Actions will run tests against every pull request to OJS, OMP, or OPS. The tests are listed in the checks at the bottom of the pull request.

![A Travis check in the pull request.](./travis-pr.png)
![A list of checks in a pull request.](./github-pr.png)

Click the **Details** link beside the Travis tests to watch the tests as they run and see which tests have passed or failed. All tests must pass before a pull request will be merged.
Click the **Details** link beside a test to watch the test as it runs and see which tests have passed or failed. All tests must pass before a pull request will be merged.

### Prepare pull requests for testing

A single change to an application may involve pull requests to more than one repository. For example, a change that impacts `pkp-lib` and `ojs` will require two pull requests.
A change to an application may involve pull requests to more than one repository. For example, a change that impacts `pkp-lib` and `ojs` will require two pull requests.

Travis must be able to locate the correct repository and branch for the `pkp-lib` submodule in order to run the tests against the correct code. To do this, an extra commit with a special commit message, `Submodule update ##<username>/<branch>##`, is added to the application repository.
GitHub Actions must be able to locate the correct repository and branch for the `pkp-lib` submodule in order to run the tests against the correct code. To do this, an extra commit with a special commit message, `Submodule update ##<username>/<branch>##`, is added to the application repository.

The example below will instruct Travis to clone the `ojs` and `pkp-lib` repositories owned by `NateWr` and check out the `i123_example` branch.
The example below will instruct GitHub Actions to clone the `ojs` and `pkp-lib` repositories owned by `NateWr` and check out the `i123_example` branch.

```
cd path/to/ojs
Expand Down
24 changes: 10 additions & 14 deletions dev/testing/en/debug.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Debug Test Failures - Testing - OJS/OMP

# Debug Test Failures

When a test fails it means that a recent change has broken the application. The log files for the tests will will indicate which tests failed and will list all of the commands that were run before the failure occurred.
When a test fails it means that a recent change has broken the application. The log files for the tests will indicate which tests failed and will list all of the commands that were run before the failure occurred.

In most cases, the server's error log will contain important clues about what went wrong.

Expand All @@ -16,40 +16,36 @@ When the tests are run from the terminal you will find a screenshot of the brows

## Debug CI

The [CI tests](./continuous-integration) can be difficult to debug. Travis provides access to its build logs and screenshots of test failures to help.
The [CI tests](./continuous-integration) can be difficult to debug. GitHub provides access to its build logs and artifacts produced by tests to help. Click **Checks** on a pull request to see the logs from the tests.

<figure class="video_container">
<video controls="true" allowfullscreen="true">
<source src="./travis-log.mp4" type="video/mp4">
<source src="./github-log.mp4" type="video/mp4">
</video>
<figcaption>Short video showing how to access log of build results.</figcaption>
<figcaption>Short video showing how to access the log of build results.</figcaption>
</figure>

This provides a log of the entire build process including the logs from the Cypress and PHP Unit tests. Parts of the log, such as the server's error log, are truncated and can only be reached by accessing the raw log.

<figure class="video_container">
<video controls="true" allowfullscreen="true">
<source src="./travis-log-raw.mp4" type="video/mp4">
<source src="./github-log-raw.mp4" type="video/mp4">
</video>
<figcaption>Short video showing how to access the raw log of build results.</figcaption>
</figure>

> When viewing the raw log, run a search in your browser for "fatal". This is a quick way to find the source of many -- _but not all_ -- test failures.
{:.tip}

### Travis debug mode
### Artifacts

Travis provides a [debug mode](https://docs.travis-ci.com/user/running-build-in-debug-mode/). When enabled it is possible to log into the Travis environment via SSH and explore the tests manually.
When a test fails, GitHub Actions will create individual artifact downloads that contain screenshots generated by Cypress for each failing test, as well as server logs.

### Screenshots
![GitHub Actions log with an artifact download URL.](./download-artifacts.png)

Screenshots can be accessed by using the uuencode tool on the Travis VM. To extract the screenshots related to a test failure, follow these steps.
Download URLs will also be output in the logs for each failing test near the end of a test run.

- When viewing the test output on Travis, click the "Raw Log" button.
- Save the log to your computer, for example `/tmp/log.txt`.
- Run `cat /tmp/log.txt | uudecode | tar xvz` to extract the screenshots locally.

> If you are struggling to understand a test failure in Travis, it may be best to run the tests locally where you have more insight into the state of the application.
> If you are struggling to understand a test failure, it may be best to run the tests locally where you have more insight into the state of the application.
{:.tip}

---
Expand Down
Binary file added dev/testing/en/download-artifacts.png
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@withanage I've noticed that this "Download artifacts" link at the top of a PR when a test fails does not always appear consistently - do you know why that is?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaitlinnewson actually, github should generate them automatically, do you have an example, where you could not see the Download link ?

Copy link
Member Author

@kaitlinnewson kaitlinnewson Oct 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@withanage Nevermind, I think I see the confusion - this link to artifacts at the top only appears on the .../checks page, but then does not appear if you click on a job (but can be accessed in the build log instead). For example:

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 3 additions & 3 deletions dev/testing/en/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ We use [Cypress](https://www.cypress.io/) and [PHPUnit](https://phpunit.de/) to

## Configure your environment

There are many ways to [configure your environment](https://docs.cypress.io/guides/guides/environment-variables.html#Setting) to run the Cypress tests. We recommend creating a `cypress.env.json` file.
There are many ways to [configure your environment](https://docs.cypress.io/guides/guides/environment-variables#Setting) to run the Cypress tests. We recommend creating a `cypress.env.json` file.

```json
{
Expand All @@ -31,7 +31,7 @@ The integration tests will install the software and create test data.

## Run integration tests

The integration tests will install the software and run a number of tests to add users, make submissions, record editorial decisions and publish articles.
The integration tests will install the software and run tests to add users, make submissions, record editorial decisions and publish articles.

Before the integration tests are run, update your `config.inc.php` file and set the `installed` flag to `Off`.

Expand All @@ -41,7 +41,7 @@ Before the integration tests are run, update your `config.inc.php` file and set
installed = Off
```

Run the following comand in the root directory of the application to start a server. The URL should match the `baseUrl` in `cypress.env.json`.
Run the following command in the root directory of the application to start a server. The URL should match the `baseUrl` in `cypress.env.json`.

```
php -S localhost:8000
Expand Down
Binary file added dev/testing/en/github-actions-overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added dev/testing/en/github-log-raw.mp4
Binary file not shown.
Binary file added dev/testing/en/github-log.mp4
Binary file not shown.
Binary file added dev/testing/en/github-pr.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 9 additions & 1 deletion dev/testing/en/plugins-themes.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,19 @@ title: Plugins and Themes - Testing - OJS/OMP

Plugins and themes can make use of parts of the application tests to prepare a test environment and run tests. This allows each plugin to implement [continuous integration](./continuous-integration) testing in its own repository.

The [Static Pages](https://github.com/pkp/staticPages/) and [Quick Submit](https://github.com/pkp/quickSubmit) plugins and the [Manuscript](https://github.com/pkp/defaultManuscript) theme are configured for testing. The `.travis.yml` file in each repository configures the integration tests. The tests that are run can be found in each repository's `cypress/tests` directory .
The `.github/workflows/main.yml` file in each repository configures the integration tests. Additional plugin-dependent commands that are run during tests may be found in `.github/actions/tests.sh`. The tests that are run can be found in each repository's `cypress/tests` directory.

Plugin and theme tests can be run locally by running the following command in the application's root directory.

```
cd path/to/ojs
npx cypress run --config integrationFolder=plugins/generic/staticPages/cypress/tests
```

## Examples

Examples of plugins that have been configured with tests include:

- [OAI JATS](https://github.com/pkp/oaiJats)
- [Quick Submit](https://github.com/pkp/quickSubmit)
- The [Manuscript theme](https://github.com/pkp/defaultManuscript)
Binary file removed dev/testing/en/travis-log-raw.mp4
Binary file not shown.
Binary file removed dev/testing/en/travis-log.mp4
Binary file not shown.
Binary file removed dev/testing/en/travis-overview.png
Binary file not shown.
Binary file removed dev/testing/en/travis-pr.png
Binary file not shown.
2 changes: 1 addition & 1 deletion dev/testing/en/writing-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ It is often appropriate to add to one of the existing tests rather than create a
> If you are writing a test which checks a particular part of the editorial workflow, it should be put in `cypress/tests/data`. If you are writing a test for something else, such as statistics or journal settings, it should be put in `cypress/tests/integration`.
{:.notice}

Read the Cypress documentation to learn how to write your [first test](https://docs.cypress.io/guides/getting-started/writing-your-first-test.html), [organize tests](https://docs.cypress.io/guides/core-concepts/writing-and-organizing-tests.html#Test-Structure), and [interact with buttons and fields](https://docs.cypress.io/guides/core-concepts/interacting-with-elements.html#Actionability).
Read the Cypress documentation to learn how to write your [first test](https://docs.cypress.io/guides/end-to-end-testing/writing-your-first-end-to-end-test), [organize tests](https://docs.cypress.io/guides/core-concepts/writing-and-organizing-tests#Test-Structure), and [interact with buttons and fields](https://docs.cypress.io/guides/core-concepts/interacting-with-elements#Actionability).

## Commands

Expand Down
Loading