Add main configuration file for tests

[mrodm]

What does this PR do?

Adds a new configuration file to define global settings for tests: _dev/test/config.yml

Why is it important?

This configuration file is going to be used to define if tests of specific runners could be run in parallel or they should be run sequentially.

Example of the contents of this configuration file:

system:
  parallel: true
policy:
  parallel: false
  skip:
    reason: flaky tests
    link: https://github.com/elastic/package-spec/issues/1

Checklist

• I have added test packages to test/packages that prove my change is effective.
• I have added an entry in spec/changelog.yml.

Related issues

• Part of Add ability to deploy custom elastic-agents on different OS or runtimes elastic-package#787

[jsoriano]

Nice! Maybe we can also use this common config to skip all the tests of a kind.

[mrodm]

Adding a new field skip in every kind of tests ? Something like this?

static:
  parallel: false
  skip: true

IIUC that would avoid to add the skip field in every test config file?

[jsoriano]

Yes, this would be useful for example to disable all system tests if there is some issue with the service deployment.

[mrodm]

Added skip. 👍

I also renamed parallel to be sequential, not sure if that would be better or not.

Thinking in elastic-package , if sequential is not defined it means is false by default, and therefore tests could be run in parallel.

WDYT ?

[jsoriano]

I think I prefer parallel to sequential, but not a strong opinion, as you prefer.

[mrodm]

Set parallel again with default value false.
That would allow us to enable parallel tests just in the desired packages, not forcing to run all the system tests of all packages in parallel. There could be several packages that could not run those tests in parallel:

• hardcoded exposed ports in services
• running tests in kubernetes
• running tests creating resources in terraform (there could be some limits creating resources for instance)

[jsoriano]

Have you considered having one file per test directory? I guess it depends on what is going to read this config. If it is going to be read by the commands it is fine here, if it is going to be read by the runners, I think it would be better in each test directory. But not a strong opinion.

[mrodm]

These files should be read/parsed before triggering all tests.
Currently, this means to be read in the command definitions. There is where the tests are triggered.

There is an option to move all the logic to get all tests to each runner. If so, then these files could be read in each runner before triggering any test.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: b939ffd

History

• 💚 Build #694 succeeded 8d065e8
• 💚 Build #688 succeeded 1a3f986
• 💚 Build #687 succeeded 2d76f92
• 💚 Build #684 succeeded ad8e946

cc @mrodm