Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logstash to serverless provider #1646

Merged
merged 16 commits into from
Feb 8, 2024
Merged

Conversation

bhapas
Copy link
Contributor

@bhapas bhapas commented Jan 26, 2024

Adds logstash service as an optional service to serverless provider.

Few updates as part of this PR:

  • Added elastic-agent client certificate that is used for communication between agent and logstash over ssl in serverless using x509.ExtKeyUsageClientAuth
  • Added UpdateFleetOutput function to be able to update logstash fleet output with agent ssl certificates as these are created after starting local services.
  • logstash requires pem files in pkcs8 format . Hence modifying them using openssl pkcs8 -inform PEM -in /usr/share/logstash/config/certs/key.pem -topk8 -nocrypt -outform PEM -out /usr/share/logstash/config/certs/logstash.pkcs8.key

Note: Testing logstash over ssl with agent in local elastic stack is out of scope of this PR. WIll fix that in a separate PR.
Reference issue - #1658

Output when logstash is enabled in serverless provider

elastic-package stack up -d -v --provider serverless
2024/02/05 12:11:33 DEBUG Enable verbose logging
2024/02/05 12:11:33 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
Boot up the Elastic stack
2024/02/05 12:11:33 DEBUG Using Elastic Cloud URL: https://testConsole/
Using profile /Users/bharat/.elastic-package/profiles/serverless.
2024/02/05 12:11:33  WARN Elastic Serverless provider is in technical preview
2024/02/05 12:11:33 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/cac8d3051f5249d9b3233596e17a2698
2024/02/05 12:11:33  INFO Creating observability project: "elastic-package-test-serverless"
2024/02/05 12:11:33 DEBUG POST https://testConsole/api/v1/serverless/projects/observability
2024/02/05 12:11:35 DEBUG POST https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/_reset-credentials
2024/02/05 12:11:35 DEBUG Waiting for creation plan to be completed
2024/02/05 12:11:35 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:11:36 DEBUG project not initialized, status: initializing
2024/02/05 12:11:41 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:11:41 DEBUG project not initialized, status: initializing
2024/02/05 12:11:46 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:11:46 DEBUG project not initialized, status: initializing
2024/02/05 12:11:51 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:11:51 DEBUG project not initialized, status: initializing
2024/02/05 12:11:56 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:11:57 DEBUG project not initialized, status: initializing
2024/02/05 12:12:02 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:12:02 DEBUG project not initialized, status: initializing
2024/02/05 12:12:07 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:12:07 DEBUG project not initialized, status: initializing
2024/02/05 12:12:12 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:12:13 DEBUG project not initialized, status: initializing
2024/02/05 12:12:18 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:12:18 DEBUG project not initialized, status: initializing
2024/02/05 12:12:23 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:12:23 DEBUG project not initialized, status: initializing
2024/02/05 12:12:28 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:12:31 DEBUG project not initialized, status: initializing
2024/02/05 12:12:36 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:12:36 DEBUG project not initialized, status: initializing
2024/02/05 12:12:41 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:12:41 DEBUG project not initialized, status: initializing
2024/02/05 12:12:46 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:12:46 DEBUG project not initialized, status: initializing
2024/02/05 12:12:51 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:12:52 DEBUG project not initialized, status: initializing
2024/02/05 12:12:57 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:12:57 DEBUG project not initialized, status: initializing
2024/02/05 12:13:02 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:13:02 DEBUG project not initialized, status: initializing
2024/02/05 12:13:07 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:13:07 DEBUG project not initialized, status: initializing
2024/02/05 12:13:12 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f/status
2024/02/05 12:13:13 DEBUG GET https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/status
2024/02/05 12:13:14 DEBUG GET https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/fleet/fleet_server_hosts
2024/02/05 12:13:14 DEBUG POST https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/fleet/outputs
Elasticsearch host: https://elastic-package-test-serverless-b4a398.es.test.elastic.cloud
Kibana host: https://elastic-package-test-serverless-b4a398.test.elastic.cloud
Username: elastic
Password: <password redacted>
2024/02/05 12:13:15 DEBUG GET https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/status
2024/02/05 12:13:15 DEBUG Kibana service not ready: kibana in unhealthy state: unavailable
2024/02/05 12:13:20 DEBUG GET https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/status
2024/02/05 12:13:20 DEBUG Kibana service not ready: kibana in unhealthy state: unavailable
2024/02/05 12:13:25 DEBUG GET https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/status
2024/02/05 12:13:25 DEBUG Kibana service not ready: kibana in unhealthy state: unavailable
2024/02/05 12:13:30 DEBUG GET https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/status
2024/02/05 12:13:31 DEBUG Kibana service not ready: kibana in unhealthy state: unavailable
2024/02/05 12:13:36 DEBUG GET https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/status
2024/02/05 12:13:36 DEBUG Kibana service not ready: kibana in unhealthy state: unavailable
2024/02/05 12:13:41 DEBUG GET https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/status
2024/02/05 12:13:41 DEBUG Kibana service not ready: kibana in unhealthy state: unavailable
2024/02/05 12:13:46 DEBUG GET https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/status
2024/02/05 12:13:46 DEBUG Kibana service not ready: kibana in unhealthy state: unavailable
2024/02/05 12:13:51 DEBUG GET https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/status
2024/02/05 12:13:51 DEBUG Kibana service not ready: kibana in unhealthy state: unavailable
2024/02/05 12:13:56 DEBUG GET https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/status
2024/02/05 12:13:57 DEBUG GET https://b4a398dbd26c47ca92f9754699acb95f.fleet.test.elastic.cloud:443/api/status
2024/02/05 12:13:57 DEBUG GET https://testConsole/api/v1/serverless/projects/observability/b4a398dbd26c47ca92f9754699acb95f
2024/02/05 12:13:57 DEBUG GET https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/status
2024/02/05 12:13:57  INFO Creating agent policy
2024/02/05 12:13:58 DEBUG Found system package - version 1.53.0
2024/02/05 12:13:58 DEBUG POST https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/fleet/agent_policies
2024/02/05 12:14:08 DEBUG POST https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/fleet/package_policies
2024/02/05 12:14:16  INFO Starting local services
2024/02/05 12:14:16 DEBUG running command: /usr/local/bin/docker compose version --short
2024/02/05 12:14:16 DEBUG Determined Docker Compose version: 2.21.0-desktop.1
2024/02/05 12:14:16 DEBUG running command: /usr/local/bin/docker compose -f /Users/bharat/.elastic-package/profiles/serverless/stack/snapshot.yml -p elastic-package-stack-serverless build
[+] Building 0.0s (0/0)                                                                                                                                                                                                             docker:desktop-linux
2024/02/05 12:14:16 DEBUG running command: /usr/local/bin/docker compose -f /Users/bharat/.elastic-package/profiles/serverless/stack/snapshot.yml -p elastic-package-stack-serverless up -d
[+] Running 7/7
 ✔ Network elastic-package-stack-serverless_default                                                                                                                      Created                                                                    0.0s 
 ✔ Container elastic-package-stack-serverless-elastic-agent-1                                                                                                            Healthy                                                                    0.0s 
 ✔ Container elastic-package-stack-serverless-logstash-1                                                                                                                 Healthy                                                                    0.0s 
 ✔ Container elastic-package-stack-serverless-logstash_is_ready-1                                                                                                        Starte...                                                                  0.0s 
 ✔ Container elastic-package-stack-serverless-elastic-agent_is_ready-1                                                                                                   S...                                                                       0.0s 
 ! logstash_is_ready The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested                                                                                 0.0s 
 ! elastic-agent_is_ready The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested                                                                            0.0s 
2024/02/05 12:15:27 DEBUG PUT https://elastic-package-test-serverless-b4a398.test.elastic.cloud/api/fleet/outputs/fleet-logstash-output
Done
System test output
packages/bitdefender - (main) > elastic-package test system -v -d push_configuration
2024/02/05 12:02:30 DEBUG Enable verbose logging
2024/02/05 12:02:30 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
Run system tests for the package
2024/02/05 12:02:30 DEBUG output command: /usr/local/bin/docker ps -a --filter label=com.docker.compose.project=elastic-package-stack-serverless --format {{.ID}}
2024/02/05 12:02:30 DEBUG output command: /usr/local/bin/docker inspect 139257b57753 349b60349f9a 3de147beca82 37bef9f61f33
2024/02/05 12:02:30 DEBUG Connecting with Elasticsearch host from current profile (profile: serverless, host: "https://elastic-package-test-serverless-cac8d3.es.test.elastic.cloud")
2024/02/05 12:02:31 DEBUG output command: /usr/local/bin/docker ps -a --filter label=com.docker.compose.project=elastic-package-stack-serverless --format {{.ID}}
2024/02/05 12:02:31 DEBUG output command: /usr/local/bin/docker inspect 139257b57753 349b60349f9a 3de147beca82 37bef9f61f33
2024/02/05 12:02:31 DEBUG Connecting with Kibana host from current profile (profile: serverless, host: "https://elastic-package-test-serverless-cac8d3.test.elastic.cloud")
2024/02/05 12:02:31 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/status
2024/02/05 12:02:31 DEBUG Running system tests for data stream
2024/02/05 12:02:31 DEBUG running test with configuration 'default'
2024/02/05 12:02:31 DEBUG setting up service...
2024/02/05 12:02:31 DEBUG setting up service using Docker Compose service deployer
2024/02/05 12:02:31 DEBUG running command: /usr/local/bin/docker compose version --short
2024/02/05 12:02:31 DEBUG Determined Docker Compose version: 2.21.0-desktop.1
2024/02/05 12:02:31 DEBUG output command: /usr/local/bin/docker network inspect elastic-package-stack-serverless_default
2024/02/05 12:02:31 DEBUG running command: /usr/local/bin/docker compose -f workspace/integrations/packages/bitdefender/_dev/deploy/docker/docker-compose.yml -p elastic-package-service up --build -d
[+] Running 7/7
 ✔ Network elastic-package-service_default                                                                                                                                            Created                                                       0.0s 
 ✔ Container elastic-package-service-bitdefender-gravityzone-api-mock-1                                                                                                               Started                                                       0.0s 
 ✔ Container elastic-package-service-bitdefender-push-notification-https-1                                                                                                            Started                                                       0.0s 
 ✔ Container elastic-package-service-bitdefender-push-notification-http-1                                                                                                             Started                                                       0.0s 
 ! bitdefender-push-notification-http The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested                                                                0.0s 
 ! bitdefender-push-notification-https The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested                                                               0.0s 
 ! bitdefender-gravityzone-api-mock The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested                                                                  0.0s 
2024/02/05 12:02:32 DEBUG running command: /usr/local/bin/docker compose -f workspace/integrations/packages/bitdefender/_dev/deploy/docker/docker-compose.yml -p elastic-package-service ps -a -q
2024/02/05 12:02:32 DEBUG Wait for healthy containers: 44653decf7841b9c36ae0924699f19c8f3df5f83952cb1df7126facbefbc44bc,fd0fd53a9ebea63e50b1d4ab41e56ec55d3bd951a92f781491ea3993e61dd2e7,917fa29a308d509d090425bfb028597d0fe866247d687f5da53f8bf719d95ff5
2024/02/05 12:02:32 DEBUG output command: /usr/local/bin/docker inspect 44653decf7841b9c36ae0924699f19c8f3df5f83952cb1df7126facbefbc44bc fd0fd53a9ebea63e50b1d4ab41e56ec55d3bd951a92f781491ea3993e61dd2e7 917fa29a308d509d090425bfb028597d0fe866247d687f5da53f8bf719d95ff5
2024/02/05 12:02:32 DEBUG Container status: {"Config":{"Image":"docker.elastic.co/observability/stream:v0.9.1","Labels":{"BRANCH_NAME":"v0.9.1","GIT_SHA":"7ae23560e53c67ec49397caf4d04055c90b87a74","GO_VERSION":"1.19.5","TIMESTAMP":"2023-01-26_01:45","com.docker.compose.config-hash":"0fd8e470003013af336cc3ce4d4ae97327748c732993db3b55bbe5d1c14dc247","com.docker.compose.container-number":"1","com.docker.compose.depends_on":"","com.docker.compose.image":"sha256:3d4ce7c75184e2cc6d237ae7030e0814269919d722489be7ae3bf00b818ccfc6","com.docker.compose.oneoff":"False","com.docker.compose.project":"elastic-package-service","com.docker.compose.project.config_files":"workspace/integrations/packages/bitdefender/_dev/deploy/docker/docker-compose.yml","com.docker.compose.project.working_dir":"workspace/integrations/packages/bitdefender/_dev/deploy/docker","com.docker.compose.service":"bitdefender-gravityzone-api-mock","com.docker.compose.version":"2.21.0"}},"ID":"44653decf7841b9c36ae0924699f19c8f3df5f83952cb1df7126facbefbc44bc","State":{"Status":"running","ExitCode":0,"Health":null}}
2024/02/05 12:02:32 DEBUG Container status: {"Config":{"Image":"docker.elastic.co/observability/stream:v0.9.1","Labels":{"BRANCH_NAME":"v0.9.1","GIT_SHA":"7ae23560e53c67ec49397caf4d04055c90b87a74","GO_VERSION":"1.19.5","TIMESTAMP":"2023-01-26_01:45","com.docker.compose.config-hash":"62a9bba284dc3dbc257ffdcaaf40d5528dceedc4d10f41d547f3e644fb806b99","com.docker.compose.container-number":"1","com.docker.compose.depends_on":"","com.docker.compose.image":"sha256:3d4ce7c75184e2cc6d237ae7030e0814269919d722489be7ae3bf00b818ccfc6","com.docker.compose.oneoff":"False","com.docker.compose.project":"elastic-package-service","com.docker.compose.project.config_files":"workspace/integrations/packages/bitdefender/_dev/deploy/docker/docker-compose.yml","com.docker.compose.project.working_dir":"workspace/integrations/packages/bitdefender/_dev/deploy/docker","com.docker.compose.service":"bitdefender-push-notification-http","com.docker.compose.version":"2.21.0"}},"ID":"fd0fd53a9ebea63e50b1d4ab41e56ec55d3bd951a92f781491ea3993e61dd2e7","State":{"Status":"running","ExitCode":0,"Health":null}}
2024/02/05 12:02:32 DEBUG Container status: {"Config":{"Image":"docker.elastic.co/observability/stream:v0.9.1","Labels":{"BRANCH_NAME":"v0.9.1","GIT_SHA":"7ae23560e53c67ec49397caf4d04055c90b87a74","GO_VERSION":"1.19.5","TIMESTAMP":"2023-01-26_01:45","com.docker.compose.config-hash":"116eb9c37e29ebae90b76388b3472d20553e495c1c305e08236ac5be7b5841a3","com.docker.compose.container-number":"1","com.docker.compose.depends_on":"","com.docker.compose.image":"sha256:3d4ce7c75184e2cc6d237ae7030e0814269919d722489be7ae3bf00b818ccfc6","com.docker.compose.oneoff":"False","com.docker.compose.project":"elastic-package-service","com.docker.compose.project.config_files":"workspace/integrations/packages/bitdefender/_dev/deploy/docker/docker-compose.yml","com.docker.compose.project.working_dir":"workspace/integrations/packages/bitdefender/_dev/deploy/docker","com.docker.compose.service":"bitdefender-push-notification-https","com.docker.compose.version":"2.21.0"}},"ID":"917fa29a308d509d090425bfb028597d0fe866247d687f5da53f8bf719d95ff5","State":{"Status":"running","ExitCode":0,"Health":null}}
2024/02/05 12:02:32 DEBUG run command: /usr/local/bin/docker network connect elastic-package-stack-serverless_default elastic-package-service-bitdefender-gravityzone-api-mock-1
2024/02/05 12:02:32 DEBUG adding service container elastic-package-service-bitdefender-gravityzone-api-mock-1 internal ports to context
2024/02/05 12:02:32 DEBUG running command: /usr/local/bin/docker compose -f workspace/integrations/packages/bitdefender/_dev/deploy/docker/docker-compose.yml -p elastic-package-service config
2024/02/05 12:02:32 DEBUG Installing package...
2024/02/05 12:02:32 DEBUG Build directory: workspace/integrations/build/packages/bitdefender/1.11.0
2024/02/05 12:02:32 DEBUG Clear target directory (path: workspace/integrations/build/packages/bitdefender/1.11.0)
2024/02/05 12:02:32 DEBUG Copy package content (source: workspace/integrations/packages/bitdefender)
2024/02/05 12:02:32 DEBUG Copy license file if needed
2024/02/05 12:02:32  INFO License text found in "workspace/integrations/LICENSE.txt" will be included in package
2024/02/05 12:02:32 DEBUG Encode dashboards
2024/02/05 12:02:32 DEBUG Resolve external fields
2024/02/05 12:02:32 DEBUG Package has external dependencies defined
2024/02/05 12:02:32 DEBUG data_stream/push_configuration/fields/base-fields.yml: source file hasn't been changed
2024/02/05 12:02:32 DEBUG data_stream/push_configuration/fields/ecs.yml: source file has been changed
2024/02/05 12:02:32 DEBUG data_stream/push_configuration/fields/fields.yml: source file hasn't been changed
2024/02/05 12:02:32 DEBUG data_stream/push_notifications/fields/base-fields.yml: source file hasn't been changed
2024/02/05 12:02:32 DEBUG data_stream/push_notifications/fields/ecs.yml: source file has been changed
2024/02/05 12:02:32 DEBUG data_stream/push_notifications/fields/fields.yml: source file hasn't been changed
2024/02/05 12:02:32 DEBUG data_stream/push_statistics/fields/base-fields.yml: source file hasn't been changed
2024/02/05 12:02:32 DEBUG data_stream/push_statistics/fields/ecs.yml: source file has been changed
2024/02/05 12:02:32 DEBUG data_stream/push_statistics/fields/fields.yml: source file hasn't been changed
2024/02/05 12:02:32 DEBUG Package doesn't have to import ECS mappings
2024/02/05 12:02:32 DEBUG Build zipped package
2024/02/05 12:02:32 DEBUG Compress using archiver.Zip (destination: workspace/integrations/build/packages/bitdefender-1.11.0.zip)
2024/02/05 12:02:32 DEBUG Create work directory for archiving: /var/folders/66/bzxk64jn4zb6__z3h34g079r0000gn/T/elastic-package-1675254499/bitdefender-1.11.0
2024/02/05 12:02:32 DEBUG Skip validation of the built .zip package
2024/02/05 12:02:32 DEBUG POST https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/epm/packages
2024/02/05 12:02:36 DEBUG creating test policy...
2024/02/05 12:02:36 DEBUG POST https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agent_policies
2024/02/05 12:02:41 DEBUG adding package data stream to test policy...
2024/02/05 12:02:41 DEBUG POST https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/package_policies
2024/02/05 12:02:45 DEBUG deleting old data in data stream...
2024/02/05 12:02:45 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:02:45 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents
2024/02/05 12:02:45 DEBUG filter agents using criteria: NamePrefix=docker-fleet-agent
2024/02/05 12:02:45 DEBUG found 1 enrolled agent(s)
2024/02/05 12:02:45 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agent_policies/66822b09-cc99-4091-baf8-f549eced4d89
2024/02/05 12:02:45 DEBUG assigning package data stream to agent...
2024/02/05 12:02:45 DEBUG PUT https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646/reassign
2024/02/05 12:02:47 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646
2024/02/05 12:02:47 DEBUG Agent data: {"id":"f48ed401-97bc-4ef3-ac13-6730c6387646","policy_id":"66822b09-cc99-4091-baf8-f549eced4d89","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2024/02/05 12:02:47 DEBUG Wait until the policy (ID: 66822b09-cc99-4091-baf8-f549eced4d89, revision: 2) is assigned to the agent (ID: f48ed401-97bc-4ef3-ac13-6730c6387646)...
2024/02/05 12:02:49 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646
2024/02/05 12:02:49 DEBUG Agent data: {"id":"f48ed401-97bc-4ef3-ac13-6730c6387646","policy_id":"66822b09-cc99-4091-baf8-f549eced4d89","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2024/02/05 12:02:49 DEBUG Wait until the policy (ID: 66822b09-cc99-4091-baf8-f549eced4d89, revision: 2) is assigned to the agent (ID: f48ed401-97bc-4ef3-ac13-6730c6387646)...
2024/02/05 12:02:51 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646
2024/02/05 12:02:51 DEBUG Agent data: {"id":"f48ed401-97bc-4ef3-ac13-6730c6387646","policy_id":"66822b09-cc99-4091-baf8-f549eced4d89","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2024/02/05 12:02:51 DEBUG Wait until the policy (ID: 66822b09-cc99-4091-baf8-f549eced4d89, revision: 2) is assigned to the agent (ID: f48ed401-97bc-4ef3-ac13-6730c6387646)...
2024/02/05 12:02:53 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646
2024/02/05 12:02:53 DEBUG Agent data: {"id":"f48ed401-97bc-4ef3-ac13-6730c6387646","policy_id":"66822b09-cc99-4091-baf8-f549eced4d89","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2024/02/05 12:02:53 DEBUG Wait until the policy (ID: 66822b09-cc99-4091-baf8-f549eced4d89, revision: 2) is assigned to the agent (ID: f48ed401-97bc-4ef3-ac13-6730c6387646)...
2024/02/05 12:02:55 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646
2024/02/05 12:02:55 DEBUG Agent data: {"id":"f48ed401-97bc-4ef3-ac13-6730c6387646","policy_id":"66822b09-cc99-4091-baf8-f549eced4d89","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2024/02/05 12:02:55 DEBUG Wait until the policy (ID: 66822b09-cc99-4091-baf8-f549eced4d89, revision: 2) is assigned to the agent (ID: f48ed401-97bc-4ef3-ac13-6730c6387646)...
2024/02/05 12:02:57 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646
2024/02/05 12:02:57 DEBUG Agent data: {"id":"f48ed401-97bc-4ef3-ac13-6730c6387646","policy_id":"66822b09-cc99-4091-baf8-f549eced4d89","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2024/02/05 12:02:57 DEBUG Wait until the policy (ID: 66822b09-cc99-4091-baf8-f549eced4d89, revision: 2) is assigned to the agent (ID: f48ed401-97bc-4ef3-ac13-6730c6387646)...
2024/02/05 12:02:59 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646
2024/02/05 12:02:59 DEBUG Agent data: {"id":"f48ed401-97bc-4ef3-ac13-6730c6387646","policy_id":"66822b09-cc99-4091-baf8-f549eced4d89","policy_revision":2,"local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2024/02/05 12:02:59 DEBUG Policy revision assigned to the agent (ID: f48ed401-97bc-4ef3-ac13-6730c6387646)...
2024/02/05 12:02:59 DEBUG checking for expected data in data stream...
2024/02/05 12:02:59 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:00 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:01 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:02 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:03 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:04 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:05 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:06 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:07 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:08 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:09 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:10 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:11 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:12 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:13 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:14 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:15 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:16 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:17 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:18 DEBUG found 0 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:19 DEBUG found 1 hits in logs-bitdefender.push_configuration-ep data stream
2024/02/05 12:03:19 DEBUG running command: /usr/local/bin/docker compose version --short
2024/02/05 12:03:19 DEBUG Determined Docker Compose version: 2.21.0-desktop.1
2024/02/05 12:03:19 DEBUG running command: /usr/local/bin/docker compose -f workspace/integrations/packages/bitdefender/_dev/deploy/docker/docker-compose.yml -p elastic-package-service ps -a -q bitdefender-gravityzone-api-mock
2024/02/05 12:03:19 DEBUG output command: /usr/local/bin/docker inspect 44653decf7841b9c36ae0924699f19c8f3df5f83952cb1df7126facbefbc44bc
2024/02/05 12:03:19 DEBUG check whether or not synthetics is enabled (component template logs-bitdefender.push_configuration@package)...
2024/02/05 12:03:19 DEBUG data stream logs-bitdefender.push_configuration-ep has synthetics enabled: false
2024/02/05 12:03:20 DEBUG reassigning original policy back to agent...
2024/02/05 12:03:20 DEBUG PUT https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646/reassign
2024/02/05 12:03:22 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646
2024/02/05 12:03:22 DEBUG Agent data: {"id":"f48ed401-97bc-4ef3-ac13-6730c6387646","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2024/02/05 12:03:22 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 4) is assigned to the agent (ID: f48ed401-97bc-4ef3-ac13-6730c6387646)...
2024/02/05 12:03:24 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646
2024/02/05 12:03:24 DEBUG Agent data: {"id":"f48ed401-97bc-4ef3-ac13-6730c6387646","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2024/02/05 12:03:24 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 4) is assigned to the agent (ID: f48ed401-97bc-4ef3-ac13-6730c6387646)...
2024/02/05 12:03:26 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646
2024/02/05 12:03:26 DEBUG Agent data: {"id":"f48ed401-97bc-4ef3-ac13-6730c6387646","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2024/02/05 12:03:26 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 4) is assigned to the agent (ID: f48ed401-97bc-4ef3-ac13-6730c6387646)...
2024/02/05 12:03:28 DEBUG GET https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agents/f48ed401-97bc-4ef3-ac13-6730c6387646
2024/02/05 12:03:28 DEBUG Agent data: {"id":"f48ed401-97bc-4ef3-ac13-6730c6387646","policy_id":"elastic-agent-managed-ep","policy_revision":4,"local_metadata":{"host":{"name":"docker-fleet-agent"}}}
2024/02/05 12:03:28 DEBUG Policy revision assigned to the agent (ID: f48ed401-97bc-4ef3-ac13-6730c6387646)...
2024/02/05 12:03:28 DEBUG deleting test policy...
2024/02/05 12:03:28 DEBUG POST https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/agent_policies/delete
2024/02/05 12:03:30 DEBUG DELETE https://elastic-package-test-serverless-cac8d3.test.elastic.cloud/api/fleet/epm/packages/bitdefender/1.11.0
2024/02/05 12:03:32 DEBUG tearing down service...
2024/02/05 12:03:32 DEBUG tearing down service using Docker Compose runner
2024/02/05 12:03:33 DEBUG running command: /usr/local/bin/docker compose version --short
2024/02/05 12:03:33 DEBUG Determined Docker Compose version: 2.21.0-desktop.1
2024/02/05 12:03:33 DEBUG running command: /usr/local/bin/docker compose -f workspace/integrations/packages/bitdefender/_dev/deploy/docker/docker-compose.yml -p elastic-package-service logs
2024/02/05 12:03:33  INFO Write container logs to file: workspace/integrations/build/container-logs/bitdefender-gravityzone-api-mock-1707131013191670000.log
2024/02/05 12:03:33 DEBUG running command: /usr/local/bin/docker compose -f workspace/integrations/packages/bitdefender/_dev/deploy/docker/docker-compose.yml -p elastic-package-service down --volumes
[+] Running 4/3
 ✔ Container elastic-package-service-bitdefender-gravityzone-api-mock-1     Removed                                                                                                                                                                 0.2s 
 ✔ Container elastic-package-service-bitdefender-push-notification-http-1   Removed                                                                                                                                                                 0.1s 
 ✔ Container elastic-package-service-bitdefender-push-notification-https-1  Removed                                                                                                                                                                 0.1s 
 ✔ Network elastic-package-service_default                                  Removed                                                                                                                                                                 0.0s 
2024/02/05 12:03:33 DEBUG deleting data in data stream...
2024/02/05 12:03:33 DEBUG Dump Elastic stack data
2024/02/05 12:03:33 DEBUG Dump stack logs (location: /var/folders/66/bzxk64jn4zb6__z3h34g079r0000gn/T/test-system-2768159782)
2024/02/05 12:03:33 DEBUG output command: /usr/local/bin/docker ps -a --filter label=com.docker.compose.project=elastic-package-stack-serverless --format {{.ID}}
2024/02/05 12:03:34 DEBUG output command: /usr/local/bin/docker inspect 139257b57753 349b60349f9a 3de147beca82 37bef9f61f33
2024/02/05 12:03:34 DEBUG Dump stack logs for elastic-agent
2024/02/05 12:03:34 DEBUG running command: /usr/local/bin/docker compose version --short
2024/02/05 12:03:34 DEBUG Determined Docker Compose version: 2.21.0-desktop.1
2024/02/05 12:03:34 DEBUG running command: /usr/local/bin/docker compose -f /Users/bharat/.elastic-package/profiles/serverless/stack/snapshot.yml -p elastic-package-stack-serverless logs elastic-agent
2024/02/05 12:03:34 DEBUG running command: /usr/local/bin/docker compose version --short
2024/02/05 12:03:34 DEBUG Determined Docker Compose version: 2.21.0-desktop.1
2024/02/05 12:03:34 DEBUG run command: /usr/local/bin/docker cp elastic-package-stack-serverless-elastic-agent-1:/usr/share/elastic-agent/state/data/logs/ /var/folders/66/bzxk64jn4zb6__z3h34g079r0000gn/T/test-system-2768159782/logs/elastic-agent-internal
2024/02/05 12:03:34 DEBUG Dump stack logs for logstash
2024/02/05 12:03:34 DEBUG running command: /usr/local/bin/docker compose version --short
2024/02/05 12:03:35 DEBUG Determined Docker Compose version: 2.21.0-desktop.1
2024/02/05 12:03:35 DEBUG running command: /usr/local/bin/docker compose -f /Users/bharat/.elastic-package/profiles/serverless/stack/snapshot.yml -p elastic-package-stack-serverless logs logstash
--- Test results for package: bitdefender - START ---
╭─────────────┬────────────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE     │ DATA STREAM        │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────────┼────────────────────┼───────────┼───────────┼────────┼───────────────┤
│ bitdefender │ push_configuration │ system    │ default   │ PASS   │ 48.655651083s │
╰─────────────┴────────────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: bitdefender - END   ---
Done

Closes #1645

@bhapas bhapas requested a review from mrodm January 26, 2024 13:58
@mrodm mrodm requested a review from a team January 29, 2024 09:45
@bhapas bhapas force-pushed the logstash-serverless branch from cd45ec3 to b87d4af Compare January 29, 2024 16:55
@bhapas bhapas requested a review from mrodm January 29, 2024 17:02
mrodm
mrodm previously approved these changes Jan 30, 2024
@mrodm mrodm self-requested a review January 30, 2024 09:59
@mrodm mrodm self-requested a review January 30, 2024 10:21
@ebeahan ebeahan added the Team:Security-Scalability Security Integrations Scalability Team label Jan 30, 2024
@bhapas bhapas added the enhancement New feature or request label Jan 31, 2024
@mrodm mrodm self-requested a review February 1, 2024 16:07
@bhapas bhapas dismissed mrodm’s stale review February 1, 2024 17:11

It's a mistake. So re referring for a review

@bhapas bhapas force-pushed the logstash-serverless branch from dbaf50e to 36dd626 Compare February 2, 2024 14:49
@bhapas bhapas requested a review from jsoriano February 2, 2024 14:55
@bhapas bhapas force-pushed the logstash-serverless branch from 36dd626 to ec66b59 Compare February 2, 2024 14:55
@bhapas bhapas requested a review from mrodm February 6, 2024 08:50
@bhapas bhapas requested a review from mrodm February 6, 2024 15:51
Copy link
Contributor

@mrodm mrodm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Just a suggestion for an error message.

Thanks @bhapas for adding Logstash into the Serverless provider!

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like requiring client authentication is making this more complex. Do we require it?

@bhapas
Copy link
Contributor Author

bhapas commented Feb 7, 2024

It looks like requiring client authentication is making this more complex. Do we require it?

@jsoriano
Apparently, logstash complains even turning this off without the client certificates

2024-02-07 10:28:30 [2024-02-07T09:28:30,865][INFO ][org.logstash.beats.BeatsHandler][main][27a70fe836509c0c2883112813d4a17e5531207f9926177eab920a7e839f779f] [local: x.x.x.x:5044, remote: x.x.x.x:43744] Handling exception: io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:

And agent fails to publish events to logstash

2024-02-07 10:30:31 {"log.level":"error","@timestamp":"2024-02-07T09:30:31.273Z","message":"failed to publish events: client is not connected","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-fleet-logstash-output","type":"system/metrics"},"log":{"source":"system/metrics-fleet-logstash-output"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":174,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

@bhapas bhapas requested review from jsoriano and mashhurs February 7, 2024 10:57
@bhapas
Copy link
Contributor Author

bhapas commented Feb 7, 2024

/test

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record

This error seems to indicate that the agent is not using SSL when trying to connect to Logstash.

So what seems to be happening here is that we just need to enable SSL in the output. This is implicitly happening when providing a client certificate, but should be also possible without it.

Could you try to enable SSL, without providing a client certificate?

@bhapas bhapas requested a review from jsoriano February 7, 2024 14:20
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bhapas for all the investigation about SSL in the logstash output!

@bhapas bhapas force-pushed the logstash-serverless branch from b33f1bd to 3b5256c Compare February 8, 2024 09:40
@bhapas bhapas requested a review from jsoriano February 8, 2024 10:17
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overal, thanks! Added some small comments.

@bhapas bhapas requested a review from jsoriano February 8, 2024 12:51
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think that we can get rid of the method to update the output. If this is not possible, I am good to go with this.
Thanks!

@jsoriano jsoriano requested a review from mrodm February 8, 2024 13:44
@bhapas bhapas merged commit 31dedaa into elastic:main Feb 8, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Team:Security-Scalability Security Integrations Scalability Team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add logstash to serverless provider
6 participants