[Buildkite] Add pipeline to test with Elastic serverless daily #1807

mrodm · 2024-04-30T13:53:01Z

Add a new pipeline to test elastic-package using the Elastic tack from a Serverless project (by default observability project). For that:

A new pipeline that is going to be executed daily for Observability project
Currently, this will be tested with all the packages under test/packages/parallel
- No system tests (as in the integrations repository)
- No coverage for pipeline tests, there are errors like:
```
Error: error running package pipeline tests: could not complete test run: error calculating pipeline coverage: error fetching pipeline stats for code coverage calculations: need exactly one ES node in stats response (got 4)
```
- Set an option to trigger this new pipeline using a comment in the Pull Request: test serverless: to be validated once this PR is merged

Example of build running Serverless step:

Move workspace and tmp folder vars to pre-command hook Use path to elastic-package binary in pre-exit hook. PATH variable does not keep the value from the pipeline step.

elasticmachine · 2024-05-02T09:23:58Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package/builds/2999

mrodm · 2024-05-02T10:45:14Z

scripts/test-check-packages.sh

@@ -68,13 +71,15 @@ if [ "${PACKAGE_TEST_TYPE:-other}" == "with-logstash" ]; then
  echo "stack.logstash_enabled: true" >> ~/.elastic-package/profiles/logstash/config.yml
 fi

-# Update the stack
-elastic-package stack update -v
+if [[ "${SERVERLESS}" != "true" ]]; then


When testing with Serverless, Elastic stack is just started once at the beginning of the pipeline step. That's why here it is skipped.

Could we also add a comment here mentioning this?

Or maybe we can extract the common logic to check packages, and follow the same approach as in serverless in all targets? For a future refactor in any case.

Added a comment for now in this script, here and also in the cleanup function.

catalog-info.yaml

mrodm · 2024-05-02T10:48:58Z

.buildkite/scripts/test_packages_with_serverless.sh

+if [[ "${BUILDKITE_PULL_REQUEST}" != "false" ]]; then
+    echo "--- install gh cli"
+    with_github_cli
+
+    add_pr_comment "${BUILDKITE_PULL_REQUEST}" "${BUILDKITE_BUILD_URL}"
+fi


Add a comment in case it is triggered with a Github comment.
Example:
#1807 (comment)

mrodm · 2024-05-02T10:49:26Z

.buildkite/scripts/integration_tests.sh

@@ -78,57 +73,37 @@ if [[ "${TARGET}" == "" ]]; then
    exit 1
 fi

-google_cloud_auth_safe_logs() {


Moved these functions to .buildkite/scripts/tooling.sh

mrodm · 2024-05-02T10:51:15Z

.buildkite/pipeline.serverless.yml

+      cpu: "8"
+      memory: "4G"
+
+  - label: ":go: :linux: Run unit tests"


Skipped in this pipeline windows tests. They should already be run by the PR pipeline or by the push to the main branch.

Umm, and do we need any unit tests here? 🤔 We could run only the integration tests for serverless.

Ok, I'll remove this too 👍
These unit tests are going to be run in each PR , so it's not needed.

Do you mean to remove also the step related to make check-static? @jsoriano

Yep, we could also remove this step if we are not going to use them to block the execution of integration tests.

I would keep check-static step, that will ensure that elastic-package can be compiled without errors.

Yep, we could also remove this step if we are not going to use them to block the execution of integration tests.

The script that starts the Elastic serverless project already performs a make install command https://github.com/elastic/elastic-package/pull/1807/files#diff-6b482aae5cb3ce434c95fe0f9238d6c8ed14a602031d6473ae03373f78ea937fR52

If there is any error there, that script should fail. So there should not be needed to keep check-static step.

I'll remove that step too.

mrodm · 2024-05-02T10:51:54Z

.buildkite/pull-requests.json

+        "build_on_comment": true,
+        "trigger_comment_regex": "^(?:(?:buildkite\\W+)?(?:test)\\W+(?:serverless))$",
+        "always_trigger_comment_regex": "^(?:(?:buildkite\\W+)?(?:test)\\W+(?:serverless))$",


Allow to trigger this pipeline on demand by adding a comment in the Pull Request.

jsoriano · 2024-05-02T14:32:21Z

.buildkite/pipeline.serverless.yml

+      cpu: "8"
+      memory: "4G"
+
+  - label: ":go: :linux: Run unit tests"


Umm, and do we need any unit tests here? 🤔 We could run only the integration tests for serverless.

jsoriano · 2024-05-02T14:33:36Z

.buildkite/pipeline.serverless.yml

+      memory: "4G"
+
+  - wait: ~
+    continue_on_failure: true


Or if we keep the static and unit tests, maybe we can use them to gate if we continue with integration tests? So if static or unit tests fail, we fail fast and don't continue with the integration tests.

Suggested change

continue_on_failure: true

continue_on_failure: false

It is required to set continue_on_failure as true, in order to process the XML files from JUnit in the last step.

Would it be possible to process the junit files for unit tests without executing the step with integration tests?

Not sure if there would be any issue with that.
It could be added a new step to just process the junit files from those two steps.

But we would need to keep the latest step to process the junit files from the integration steps, and that would re-process those junit files.

In any case, as discussed in the other thread, both static and unit tests/steps have been removed from the pipeline.

jsoriano · 2024-05-02T14:36:41Z

.buildkite/scripts/test_packages_with_serverless.sh

+echo "Waiting time to avoid getaddrinfo ENOTFOUND errors..."
+sleep 120
+echo "Done."


Could we do some active wait here?

This is based on the script run in the integrations repository. At least a few months ago, there were some errors even if elastic-package does an active wait to report as initialized the project.

Probably, we could try here in this pipeline to remove this sleep and check if it works. WDYT @jsoriano ?

As you prefer, we can also leave this for a future change.

Ok. so then I'll keep it for now, and we can test to remove it later.

jsoriano · 2024-05-02T14:39:32Z

.buildkite/scripts/tooling.sh

+    # Currently, if STACK_VERSION is not defined, for serverless it will be
+    # used as Elastic stack version (for agents) the default version in elastic-package


Nit. Move this comment closer to elastic-package stack up?

catalog-info.yaml

jsoriano · 2024-05-02T14:43:00Z

scripts/test-check-packages.sh

@@ -68,13 +71,15 @@ if [ "${PACKAGE_TEST_TYPE:-other}" == "with-logstash" ]; then
  echo "stack.logstash_enabled: true" >> ~/.elastic-package/profiles/logstash/config.yml
 fi

-# Update the stack
-elastic-package stack update -v
+if [[ "${SERVERLESS}" != "true" ]]; then


Could we also add a comment here mentioning this?

jsoriano · 2024-05-02T14:46:18Z

scripts/test-check-packages.sh

@@ -68,13 +71,15 @@ if [ "${PACKAGE_TEST_TYPE:-other}" == "with-logstash" ]; then
  echo "stack.logstash_enabled: true" >> ~/.elastic-package/profiles/logstash/config.yml
 fi

-# Update the stack
-elastic-package stack update -v
+if [[ "${SERVERLESS}" != "true" ]]; then


Or maybe we can extract the common logic to check packages, and follow the same approach as in serverless in all targets? For a future refactor in any case.

elasticmachine · 2024-05-02T16:16:02Z

💚 Build Succeeded

Buildkite Build
Commit: a05e4a9

History

💔 Build #3004 failed 28462ec
💚 Build #3003 succeeded 0abd723
💔 Build #3000 failed 7cb77a5
💔 Build #2999 failed 1e7458e
💔 Build #2997 failed fe01122

cc @mrodm

mrodm self-assigned this Apr 30, 2024

mrodm force-pushed the test_serverless_pipeline branch 2 times, most recently from 55a3292 to 0183caa Compare April 30, 2024 13:59

mrodm added 2 commits April 30, 2024 18:27

Add pipeline to test with serverless daily

0df4f27

Move workspace and tmp folder vars to pre-command hook Use path to elastic-package binary in pre-exit hook. PATH variable does not keep the value from the pipeline step.

Fix errors

96cbd80

mrodm force-pushed the test_serverless_pipeline branch from bf0fe3d to 96cbd80 Compare April 30, 2024 16:40

mrodm added 10 commits April 30, 2024 19:07

Update labels

54a8f12

Move traps to pre-exit hook

e2b9e9a

Fix unbound variable

1fa152e

Restore main pipeline

5586dad

Update label for each package

1c4eaa9

Move just clean up files

e2db838

Add comment in pull request

e61efc9

Add gh cli version

799fa18

Add gh cli version as env

fe01122

Add missing env. var

1e7458e

Remove debug changes

6469336

mrodm force-pushed the test_serverless_pipeline branch from 7cb77a5 to 6469336 Compare May 2, 2024 10:28

mrodm commented May 2, 2024

View reviewed changes

catalog-info.yaml Show resolved Hide resolved

mrodm commented May 2, 2024

View reviewed changes

mrodm added 2 commits May 2, 2024 13:00

Add email failure notification just for non pull requests builds

0abd723

Add documentation about this new pipeline

28462ec

mrodm marked this pull request as ready for review May 2, 2024 13:46

mrodm requested a review from a team May 2, 2024 14:42

jsoriano approved these changes May 2, 2024

View reviewed changes

mrodm added 5 commits May 2, 2024 17:10

Move code

4a0f705

Add comment about skipping some elastic-package commands

4dd82be

Remove unit tests step

1d2bde4

Remvoe check-static step

311164f

Rephrased comment

a05e4a9

mrodm changed the title ~~Add pipeline to test with serverless daily~~ [Buildkite] Add pipeline to test with Elastic serverless daily May 2, 2024

mrodm merged commit a9efa3e into elastic:main May 2, 2024
3 checks passed

mrodm deleted the test_serverless_pipeline branch May 2, 2024 16:38

mrodm mentioned this pull request May 2, 2024

[Buildkite] Add gcp agents into serverless step #1813

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Buildkite] Add pipeline to test with Elastic serverless daily #1807

[Buildkite] Add pipeline to test with Elastic serverless daily #1807

mrodm commented Apr 30, 2024 •

edited

Loading

elasticmachine commented May 2, 2024

mrodm May 2, 2024

jsoriano May 2, 2024

jsoriano May 2, 2024

mrodm May 2, 2024

mrodm May 2, 2024

mrodm May 2, 2024

mrodm May 2, 2024

jsoriano May 2, 2024

mrodm May 2, 2024

mrodm May 2, 2024

jsoriano May 2, 2024

mrodm May 2, 2024

mrodm May 2, 2024

mrodm May 2, 2024

jsoriano May 2, 2024

jsoriano May 2, 2024

mrodm May 2, 2024

jsoriano May 2, 2024

mrodm May 2, 2024

jsoriano May 2, 2024

mrodm May 2, 2024

jsoriano May 2, 2024

mrodm May 2, 2024

jsoriano May 2, 2024

mrodm May 2, 2024

jsoriano May 2, 2024

jsoriano May 2, 2024

elasticmachine commented May 2, 2024

		# Currently, if STACK_VERSION is not defined, for serverless it will be
		# used as Elastic stack version (for agents) the default version in elastic-package

[Buildkite] Add pipeline to test with Elastic serverless daily #1807

[Buildkite] Add pipeline to test with Elastic serverless daily #1807

Conversation

mrodm commented Apr 30, 2024 • edited Loading

elasticmachine commented May 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented May 2, 2024

💚 Build Succeeded

History

mrodm commented Apr 30, 2024 •

edited

Loading