Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Initial errors/failures observed when testing AbuseCH integration in Agentless #42815

Open
qcorporation opened this issue Feb 20, 2025 · 5 comments
Assignees
Labels
Agent Filebeat Filebeat :integrations needs_team Indicates that the issue/PR needs a Team:* label
Milestone

Comments

@qcorporation
Copy link

qcorporation commented Feb 20, 2025

Please include configurations and logs if available.
elastic-agent-diagnostics-2025-02-20T20-36-47Z-00.zip

For confirmed bugs, please report:

Version:

9.0.0-beta1

Operating System:

Agentless Deployment

Steps to Reproduce:

  1. Instantiate a cloud security elastic stack instance on staging against v9.0.0
  2. Install a modified ti_abusech integration with agentless enabled using elastic-package to the instance
  3. Create/Install an AbuseCH integration in Agentless
  4. After a few minutes when logs have been ingested, capture an agent diagnostic
  5. note errors within agent logs
{"log.level":"error","@timestamp":"2025-02-20T20:05:47.646Z","message":"fatal error: concurrent map read and map write","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"cel-default","type":"cel"},"log":{"source":"cel-default"},"ecs.version":"1.6.0"}

Running it through the agent analyzer synposis looks like this:

2025-02-20T20:05:47.68Z: Component state changed cel-default (HEALTHY->FAILED): Failed: pid '68' exited with code '2'
2025-02-20T20:05:47.68Z: Unit state changed cel-default (STARTING->FAILED): Failed: pid '68' exited with code '2'
2025-02-20T20:05:47.68Z: Unit state changed cel-default-cel-ti_abusech-06203ce2-d5cc-43a6-83f6-d43252af6890 (STARTING->FAILED): Failed: pid '68' exited with code '2'
2025-02-20T20:06:44.379Z: Component state changed cel-default (HEALTHY->FAILED): Failed: pid '161' exited with code '2'
2025-02-20T20:06:44.379Z: Unit state changed cel-default-cel-ti_abusech-06203ce2-d5cc-43a6-83f6-d43252af6890 (STARTING->FAILED): Failed: pid '161' exited with code '2'
2025-02-20T20:06:44.379Z: Unit state changed cel-default (STARTING->FAILED): Failed: pid '161' exited with code '2'
2025-02-20T20:07:21.605Z: Component state changed cel-default (HEALTHY->FAILED): Failed: pid '226' exited with code '2'
2025-02-20T20:07:21.605Z: Unit state changed cel-default-cel-ti_abusech-06203ce2-d5cc-43a6-83f6-d43252af6890 (STARTING->FAILED): Failed: pid '226' exited with code '2'
2025-02-20T20:07:21.605Z: Unit state changed cel-default (STARTING->FAILED): Failed: pid '226' exited with code '2'
2025-02-20T20:09:12.822Z: Component state changed cel-default (HEALTHY->FAILED): Failed: pid '417' exited with code '2'
2025-02-20T20:09:12.823Z: Unit state changed cel-default-cel-ti_abusech-06203ce2-d5cc-43a6-83f6-d43252af6890 (STARTING->FAILED): Failed: pid '417' exited with code '2'
2025-02-20T20:09:12.823Z: Unit state changed cel-default (STARTING->FAILED): Failed: pid '417' exited with code '2'
2025-02-20T20:09:57.277Z: Component state changed cel-default (HEALTHY->FAILED): Failed: pid '491' exited with code '2'

and

2025-02-20T20:07:43.807Z: Error fetching data for metricset http.json: error making http request: Get "http://unix/inputs": dial unix /agentless/data/tmp/WDSbrgjXxHKnME7Z0CSyI1xJnirO2oRp.sock: connect: connection refused
2025-02-20T20:10:41.879Z: Error fetching data for metricset http.json: error making http request: Get "http://unix/stats": dial unix /agentless/data/tmp/WDSbrgjXxHKnME7Z0CSyI1xJnirO2oRp.sock: connect: connection refused
@qcorporation qcorporation added :integrations Agent Filebeat Filebeat needs_team Indicates that the issue/PR needs a Team:* label labels Feb 20, 2025
@botelastic
Copy link

botelastic bot commented Feb 20, 2025

This issue doesn't have a Team:<team> label.

@efd6
Copy link
Contributor

efd6 commented Feb 20, 2025

It looks like multiple goroutines are holding on to a configuration value.

fatal error: concurrent map read and map write

goroutine 88 [running]:
github.com/elastic/go-ucfg.(*fields).get(...)
	github.com/elastic/go-ucfg@v0.8.8/ucfg.go:249
github.com/elastic/go-ucfg.namedField.GetValue({{0xaaaaf83bf28e?, 0x8?}}, 0x40032340a0, {0xaaaafa20b6b8, 0x4003381560})
	github.com/elastic/go-ucfg@v0.8.8/path.go:189 +0x70
github.com/elastic/go-ucfg.cfgPath.GetValue({{0x400202c360, 0x1, 0x1}, {0xaaaaf8053510, 0x1}}, 0x4003381560, 0x40032340a0)
	github.com/elastic/go-ucfg@v0.8.8/path.go:176 +0x124
github.com/elastic/go-ucfg.reifyGetField(0x4003381560, {0x40032340a0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}}, {0xaaaaf83bf28e, 0x7}, ...)
	github.com/elastic/go-ucfg@v0.8.8/reify.go:315 +0xd0
github.com/elastic/go-ucfg.reifyStruct(0x40032340a0, {0xaaaaf9e62220?, 0x400323e000?, 0x0?}, 0x4003381560)
	github.com/elastic/go-ucfg@v0.8.8/reify.go:292 +0x390
github.com/elastic/go-ucfg.reifyInto(0x40032340a0, {0xaaaaf8ce2da0?, 0x400323e000?, 0xaaaaf9d23600?}, 0x4003381560)
	github.com/elastic/go-ucfg@v0.8.8/reify.go:177 +0x1c8
github.com/elastic/go-ucfg.(*Config).Unpack(0x4003381560, {0xaaaaf8ce2da0, 0x400323e000}, {0x4002445980?, 0x0?, 0x0?})
	github.com/elastic/go-ucfg@v0.8.8/reify.go:155 +0xc0
github.com/elastic/elastic-agent-libs/config.(*C).Unpack(...)
	github.com/elastic/elastic-agent-libs@v0.18.1/config/config.go:159
github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewClients(0x4003381560, {0xaaaaf710115c, 0x8})
	github.com/elastic/beats/v7/libbeat/esleg/eslegclient/connection.go:208 +0x108
github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnectedClient({0xaaaafa1bb4a8, 0x40022d2f50}, 0xaaaaf7162f9e?, {0xaaaaf710115c?, 0x0?})
	github.com/elastic/beats/v7/libbeat/esleg/eslegclient/connection.go:255 +0x3c
github.com/elastic/beats/v7/libbeat/statestore/backend/es.(*store).configure(0x4003316380, {0xaaaafa1bb4a8, 0x40022d2f50}, 0x4003381560)
	github.com/elastic/beats/v7/libbeat/statestore/backend/es/store.go:309 +0x178
github.com/elastic/beats/v7/libbeat/statestore/backend/es.(*store).loop(0x4003316380, {0xaaaafa1bb4a8, 0x40022d2f50}, 0x0?, 0x0?, 0x40000bd200)
	github.com/elastic/beats/v7/libbeat/statestore/backend/es/store.go:337 +0xb0
created by github.com/elastic/beats/v7/libbeat/statestore/backend/es.openStore in goroutine 1
	github.com/elastic/beats/v7/libbeat/statestore/backend/es/store.go:85 +0x2c4

goroutine 1 [chan receive]:
github.com/elastic/beats/v7/filebeat/beater.(*signalWait).Wait(...)
	github.com/elastic/beats/v7/filebeat/beater/signalwait.go:44
github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run(0x4001eab340, 0x4001d8c008)
	github.com/elastic/beats/v7/filebeat/beater/filebeat.go:496 +0x180c
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch(0x4001d8c008, {{0xaaaaf710115c, 0x8}, {0xaaaaf710115c, 0x8}, {0x0, 0x0}, 0x1, 0x1, {{0x0, ...}, ...}, ...}, ...)
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:749 +0x6d0
github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1(0x40024bfbd8, 0x4001dbfc30)
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:200 +0xe0
github.com/elastic/beats/v7/libbeat/cmd/instance.Run({{0xaaaaf710115c, 0x8}, {0xaaaaf710115c, 0x8}, {0x0, 0x0}, 0x1, 0x1, {{0x0, 0x0}, ...}, ...}, ...)
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:201 +0x24
github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1(0x4001a58600?, {0x40018e11e0?, 0x4?, 0xaaaaf70f1cda?})
	github.com/elastic/beats/v7/libbeat/cmd/run.go:36 +0x50
github.com/spf13/cobra.(*Command).execute(0x4002068e08, {0x40018e1040, 0x1a, 0x1a})
	github.com/spf13/cobra@v1.8.1/command.go:989 +0x828
github.com/spf13/cobra.(*Command).ExecuteC(0x4001a3a308)
	github.com/spf13/cobra@v1.8.1/command.go:1117 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.8.1/command.go:1041
main.main()
	github.com/elastic/beats/v7/x-pack/agentbeat/main.go:27 +0x20

goroutine 35 [select]:
go.opencensus.io/stats/view.(*worker).start(0x400105f300)
	go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x88
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x98

goroutine 10 [select]:
github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata(0x4002f43ce0)
	github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata/providers.go:195 +0x3a8
github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1()
	github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go:98 +0x28
sync.(*Once).doSlow(0x4001e040a0?, 0x4001e04098?)
	sync/once.go:74 +0x100
sync.(*Once).Do(...)
	sync/once.go:65
github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init(0x0?)
	github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go:97 +0x40
created by github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.New in goroutine 1
	github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go:87 +0x294

goroutine 31 [chan receive]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*queueReader).run(0x40022d2490, 0x4001e93690)
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/queue_reader.go:51 +0x90
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer in goroutine 1
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:102 +0x1d8

goroutine 67 [select]:
github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata.(*cache).cleanup(0x4002f76300)
	github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata/cache.go:79 +0xb4
created by github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata.newCache in goroutine 1
	github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata/cache.go:44 +0xdc

goroutine 30 [select]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).run(0x40022d2460)
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:161 +0x228
github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.func1()
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:91 +0x54
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer in goroutine 1
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/consumer.go:89 +0x174

goroutine 70 [IO wait]:
internal/poll.runtime_pollWait(0xffff3c2527a8, 0x72)
	runtime/netpoll.go:345 +0xa0
internal/poll.(*pollDesc).wait(0x3?, 0x10?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x4002f55480)
	internal/poll/fd_unix.go:611 +0x250
net.(*netFD).accept(0x4002f55480)
	net/fd_unix.go:172 +0x28
net.(*UnixListener).accept(0x4002f63d01?)
	net/unixsock_posix.go:172 +0x20
net.(*UnixListener).Accept(0x4002f76ae0)
	net/unixsock.go:260 +0x2c
net/http.(*Server).Serve(0x4002576960, {0xaaaafa19c540, 0x4002f76ae0})
	net/http/server.go:3260 +0x2a8
net/http.Serve(...)
	net/http/server.go:2799
github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1({0xaaaafa19c540, 0x4002f76ae0})
	github.com/elastic/beats/v7/libbeat/api/server.go:72 +0x118
created by github.com/elastic/beats/v7/libbeat/api.(*Server).Start in goroutine 1
	github.com/elastic/beats/v7/libbeat/api/server.go:70 +0x118

goroutine 32 [select]:
github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop(0x40022d2780)
	github.com/elastic/beats/v7/libbeat/monitoring/report/log/log.go:166 +0x1b4
github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter.func1()
	github.com/elastic/beats/v7/libbeat/monitoring/report/log/log.go:139 +0x54
created by github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter in goroutine 1
	github.com/elastic/beats/v7/libbeat/monitoring/report/log/log.go:137 +0x2f0

goroutine 82 [chan receive]:
github.com/elastic/elastic-agent-libs/service.HandleSignals.func1()
	github.com/elastic/elastic-agent-libs@v0.18.1/service/service.go:50 +0x4c
created by github.com/elastic/elastic-agent-libs/service.HandleSignals in goroutine 1
	github.com/elastic/elastic-agent-libs@v0.18.1/service/service.go:49 +0x154

goroutine 61 [syscall]:
os/signal.signal_recv()
	runtime/sigqueue.go:152 +0x30
os/signal.loop()
	os/signal/signal_unix.go:23 +0x1c
created by os/signal.Notify.func1.1 in goroutine 1
	os/signal/signal.go:151 +0x28

goroutine 84 [chan receive]:
github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run.func1()
	github.com/elastic/beats/v7/filebeat/beater/filebeat.go:309 +0x30
created by github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run in goroutine 1
	github.com/elastic/beats/v7/filebeat/beater/filebeat.go:308 +0x3c8

goroutine 85 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x4003202180?, 0x4002f5de40)
	github.com/elastic/go-concert@v0.3.0/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f81e0, {0xffff3c056b60, 0x40022d2e60}, 0x40022d2e10, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:160 +0xf8

goroutine 86 [select]:
github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*updateChan).Recv(0x4003312210, {0xaaaafa1bb4a8, 0x40022d2f00})
	github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile/update_writer.go:151 +0x1b0
github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*updateWriter).run(0x400216dae0, {0xaaaafa1bb4a8, 0x40022d2f00})
	github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile/update_writer.go:87 +0x54
github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.newUpdateWriter.func1({0xaaaafa1bb4a8?, 0x40022d2f00?})
	github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile/update_writer.go:65 +0x34
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:160 +0xf8

goroutine 87 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x400320dde8?, 0x4002f5ce48)
	github.com/elastic/go-concert@v0.3.0/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*cleaner).run(0x40020f81f8, {0xffff3c056b60, 0x40022d2e60}, 0x40022d2eb0, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile/clean.go:50 +0xa0
github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile/manager.go:134 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:160 +0xf8

goroutine 89 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x400321de40)
	github.com/elastic/go-concert@v0.3.0/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f8220, {0xffff3c056b60, 0x40022d2e60}, 0x40022d2fa0, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:160 +0xf8

goroutine 90 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x400321ce40)
	github.com/elastic/go-concert@v0.3.0/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f8238, {0xffff3c056b60, 0x40022d2e60}, 0x40022d2ff0, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:160 +0xf8

goroutine 91 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x4002f61e40)
	github.com/elastic/go-concert@v0.3.0/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f8250, {0xffff3c056b60, 0x40022d2e60}, 0x40022d3090, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:160 +0xf8

goroutine 92 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x40000ece40)
	github.com/elastic/go-concert@v0.3.0/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f8268, {0xffff3c056b60, 0x40022d2e60}, 0x40022d30e0, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:160 +0xf8

goroutine 93 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x4002f60e40)
	github.com/elastic/go-concert@v0.3.0/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f8280, {0xffff3c056b60, 0x40022d2e60}, 0x40022d3130, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:160 +0xf8

goroutine 94 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x4002f5ee40)
	github.com/elastic/go-concert@v0.3.0/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f8298, {0xffff3c056b60, 0x40022d2e60}, 0x40022d3180, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:160 +0xf8

goroutine 95 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x40000ede40)
	github.com/elastic/go-concert@v0.3.0/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f82b0, {0xffff3c056b60, 0x40022d2e60}, 0x40022d31d0, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:160 +0xf8

goroutine 96 [select]:
github.com/elastic/go-concert/timed.Periodic({0xffff3c056b88, 0x40022d2e60}, 0x0?, 0x4002f5ae40)
	github.com/elastic/go-concert@v0.3.0/timed/timed.go:82 +0xc8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*cleaner).run(0x40020f82e8, {0xffff3c056b60, 0x40022d2e60}, 0x40022d3220, 0x45d964b800)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/clean.go:48 +0x9c
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*InputManager).Init.func1({0xaaaafa1bb4a8, 0x40022d2e60})
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/manager.go:140 +0xf0
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1()
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:164 +0x94
created by github.com/elastic/go-concert/unison.(*TaskGroup).Go in goroutine 1
	github.com/elastic/go-concert@v0.3.0/unison/taskgroup.go:160 +0xf8

goroutine 97 [select]:
github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Run(0x4001afc360)
	github.com/elastic/beats/v7/filebeat/registrar/registrar.go:164 +0x1cc
github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start.func1()
	github.com/elastic/beats/v7/filebeat/registrar/registrar.go:122 +0x54
created by github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Start in goroutine 1
	github.com/elastic/beats/v7/filebeat/registrar/registrar.go:120 +0xb4

goroutine 98 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x4002031b00, {0xaaaafa1bb4a8, 0x40022d3450})
	google.golang.org/grpc@v1.67.3/internal/grpcsync/callback_serializer.go:88 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
	google.golang.org/grpc@v1.67.3/internal/grpcsync/callback_serializer.go:52 +0x118

goroutine 99 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x4002031b30, {0xaaaafa1bb4a8, 0x40022d34a0})
	google.golang.org/grpc@v1.67.3/internal/grpcsync/callback_serializer.go:88 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
	google.golang.org/grpc@v1.67.3/internal/grpcsync/callback_serializer.go:52 +0x118

goroutine 100 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x4002031b70, {0xaaaafa1bb4a8, 0x40022d34f0})
	google.golang.org/grpc@v1.67.3/internal/grpcsync/callback_serializer.go:88 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
	google.golang.org/grpc@v1.67.3/internal/grpcsync/callback_serializer.go:52 +0x118

goroutine 63 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	google.golang.org/grpc@v1.67.3/stream.go:393 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 102
	google.golang.org/grpc@v1.67.3/stream.go:392 +0xb2c

goroutine 102 [semacquire]:
sync.runtime_Semacquire(0x4000183180?)
	runtime/sema.go:62 +0x2c
sync.(*WaitGroup).Wait(0x40019b89a0)
	sync/waitgroup.go:116 +0x74
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip(0x4001e0a700)
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:501 +0x1c4
github.com/elastic/elastic-agent-client/v7/pkg/client.durationOf(0x4003321f78)
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:406 +0x38
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startCheckin.func1()
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:435 +0xb4
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startCheckin in goroutine 1
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:427 +0x70

goroutine 103 [semacquire]:
sync.runtime_Semacquire(0x4000183880?)
	runtime/sema.go:62 +0x2c
sync.(*WaitGroup).Wait(0x40019b8b40)
	sync/waitgroup.go:116 +0x74
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip(0x4001e0a700, 0x4001afc8a0)
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:841 +0x1ec
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startActions.func1.1()
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:781 +0x24
github.com/elastic/elastic-agent-client/v7/pkg/client.durationOf(0x4003322f70)
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:406 +0x38
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startActions.func1()
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:780 +0xc8
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).startActions in goroutine 1
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:772 +0x94

goroutine 104 [select]:
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).watchErrChan(0x4002edd0e0, {0xaaaafa1bb4a8, 0x40022d39f0})
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:461 +0x98
created by github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).Start in goroutine 1
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:287 +0x12c

goroutine 105 [runnable]:
runtime.CallersFrames(...)
	runtime/symtab.go:77
go.uber.org/zap/internal/stacktrace.Capture(0x5, 0x0)
	go.uber.org/zap@v1.27.0/internal/stacktrace/stack.go:107 +0x158
go.uber.org/zap.(*Logger).check(0x4003330d00, 0x0, {0x40033366c0, 0x118})
	go.uber.org/zap@v1.27.0/logger.go:379 +0x540
go.uber.org/zap.(*Logger).Check(...)
	go.uber.org/zap@v1.27.0/logger.go:222
go.uber.org/zap.(*SugaredLogger).log(0x40020f8730, 0x0, {0xaaaaf723322b?, 0x24?}, {0x4003378cb0?, 0x2?, 0x2?}, {0x0, 0x0, 0x0})
	go.uber.org/zap@v1.27.0/sugar.go:354 +0xb8
go.uber.org/zap.(*SugaredLogger).Infof(...)
	go.uber.org/zap@v1.27.0/sugar.go:204
github.com/elastic/elastic-agent-libs/logp.(*Logger).Infof(...)
	github.com/elastic/elastic-agent-libs@v0.18.1/logp/logger.go:156
github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES({_, _}, {{0xaaaaf710115c, 0x8}, {0xaaaaf710115c, 0x8}, {0x4001fdde00, 0xb}, 0x1, {0x4001fff700, ...}, ...}, ...)
	github.com/elastic/beats/v7/libbeat/outputs/elasticsearch/elasticsearch.go:57 +0x204
github.com/elastic/beats/v7/libbeat/outputs.Load({_, _}, {{0xaaaaf710115c, 0x8}, {0xaaaaf710115c, 0x8}, {0x4001fdde00, 0xb}, 0x1, {0x4001fff700, ...}, ...}, ...)
	github.com/elastic/beats/v7/libbeat/outputs/output_reg.go:109 +0xd4
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createOutput(0x4001d8c008, {0xaaaafa201280, 0x4003330800}, {{0x40032bc410, 0xd}, 0x4003381560})
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:1505 +0x198
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).Reload.func1({0xaaaafa201280?, 0x4003330800?})
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:179 +0x68
github.com/elastic/beats/v7/libbeat/publisher/pipeline.loadOutput({0x400223ae00?, 0x40022f0100?, 0x4001e93690?, 0x40018e1ba0?}, 0x40033794a8?)
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:139 +0x12c
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).Reload(0x4001e8e480, 0x4003386100, 0x40020520a0)
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:177 +0xbc
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater.(*Beat).makeOutputReloader.func2(0x4003386100)
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:1392 +0xd0
github.com/elastic/beats/v7/libbeat/common/reload.ReloadableFunc.Reload(0x40032ba510?, 0x40020399ff?)
	github.com/elastic/beats/v7/libbeat/common/reload/reload.go:226 +0x28
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).reloadOutput(0x4002edd0e0, 0x40032a8540)
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:784 +0x270
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).reload(0x4002edd0e0, 0x4003289e40)
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:646 +0x7d0
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen(0x4002edd0e0)
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:547 +0x6c4
created by github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).Start in goroutine 1
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:295 +0x1f0

goroutine 106 [chan receive]:
github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run.(*signalWait).AddChan.waitChannel.func17()
	github.com/elastic/beats/v7/filebeat/beater/signalwait.go:74 +0x24
github.com/elastic/beats/v7/filebeat/beater.(*signalWait).Add.func1()
	github.com/elastic/beats/v7/filebeat/beater/signalwait.go:51 +0x2c
created by github.com/elastic/beats/v7/filebeat/beater.(*signalWait).Add in goroutine 1
	github.com/elastic/beats/v7/filebeat/beater/signalwait.go:50 +0x7c

goroutine 64 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readHeaderClient(0x4002b9f310, {0x40031f5210, 0x5, 0x5})
	google.golang.org/grpc@v1.67.3/internal/transport/transport.go:199 +0x78
google.golang.org/grpc/internal/transport.(*recvBufferReader).ReadHeader(0x4002b9f310, {0x40031f5210?, 0x40033386c8?, 0x4003284360?})
	google.golang.org/grpc@v1.67.3/internal/transport/transport.go:145 +0x3c
google.golang.org/grpc/internal/transport.(*transportReader).ReadHeader(0x4001a79ba0, {0x40031f5210?, 0x40020b1c00?, 0x100004001bbd305?})
	google.golang.org/grpc@v1.67.3/internal/transport/transport.go:614 +0x2c
google.golang.org/grpc/internal/transport.(*Stream).ReadHeader(0x4003284360, {0x40031f5210, 0x5, 0x5})
	google.golang.org/grpc@v1.67.3/internal/transport/transport.go:557 +0xa4
google.golang.org/grpc.(*parser).recvMsg(0x40031f5200, 0x6400000)
	google.golang.org/grpc@v1.67.3/rpc_util.go:659 +0x44
google.golang.org/grpc.recvAndDecompress(0x40031f5200, 0x4003284360, {0x0, 0x0}, 0x6400000, 0x0, {0x0, 0x0}, 0x0)
	google.golang.org/grpc@v1.67.3/rpc_util.go:823 +0x60
google.golang.org/grpc.recv(0x4003359b88?, {0xffff3436f648, 0xaaaaffdbf340}, 0xaaaaf06a8160?, {0x0?, 0x0?}, {0xaaaaf9bc3200, 0x4001db5490}, 0x4003359c28?, 0xaaaaf088a4bc?, ...)
	google.golang.org/grpc@v1.67.3/rpc_util.go:916 +0x6c
google.golang.org/grpc.(*csAttempt).recvMsg(0x4002ee8820, {0xaaaaf9bc3200, 0x4001db5490}, 0x58?)
	google.golang.org/grpc@v1.67.3/stream.go:1129 +0x250
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x10?)
	google.golang.org/grpc@v1.67.3/stream.go:969 +0x28
google.golang.org/grpc.(*clientStream).withRetry(0x4002f7a480, 0x4003359e40, 0x4003359e30)
	google.golang.org/grpc@v1.67.3/stream.go:773 +0x4e8
google.golang.org/grpc.(*clientStream).RecvMsg(0x4002f7a480, {0xaaaaf9bc3200?, 0x4001db5490?})
	google.golang.org/grpc@v1.67.3/stream.go:968 +0xf4
github.com/elastic/elastic-agent-client/v7/pkg/proto.(*elasticAgentCheckinV2Client).Recv(0x4001e93e40)
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/proto/elastic-agent-client_grpc.pb.go:109 +0x58
github.com/elastic/elastic-agent-client/v7/pkg/client/chunk.RecvExpected({0xffff3c056c58, 0x4001e93e40})
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/chunk/expected.go:108 +0x48
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip.func1()
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:484 +0xd0
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip in goroutine 102
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:480 +0x138

goroutine 109 [IO wait]:
internal/poll.runtime_pollWait(0xffff3c2526b0, 0x72)
	runtime/netpoll.go:345 +0xa0
internal/poll.(*pollDesc).wait(0x4003212380?, 0x400329ca80?, 0x0)
	internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x4003212380, {0x400329ca80, 0x2a80, 0x2a80})
	internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x4003212380, {0x400329ca80?, 0xffff3c199ed8?, 0x4001ea5428?})
	net/fd_posix.go:55 +0x28
net.(*conn).Read(0x40020f8588, {0x400329ca80?, 0x4003287938?, 0xaaaaf014981c?})
	net/net.go:185 +0x34
crypto/tls.(*atLeastReader).Read(0x4001ea5428, {0x400329ca80?, 0x0?, 0x4001ea5428?})
	crypto/tls/conn.go:806 +0x40
bytes.(*Buffer).ReadFrom(0x4001afb430, {0xaaaafa10a000, 0x4001ea5428})
	bytes/buffer.go:211 +0x90
crypto/tls.(*Conn).readFromUntil(0x4001afb188, {0xaaaafa107520, 0x40020f8588}, 0x4003287a10?)
	crypto/tls/conn.go:828 +0xd0
crypto/tls.(*Conn).readRecordOrCCS(0x4001afb188, 0x0)
	crypto/tls/conn.go:626 +0x35c
crypto/tls.(*Conn).readRecord(...)
	crypto/tls/conn.go:588
crypto/tls.(*Conn).Read(0x4001afb188, {0x4003362000, 0x8000, 0xaaaaf014981c?})
	crypto/tls/conn.go:1370 +0x168
bufio.(*Reader).Read(0x4003350ba0, {0x4001d689e0, 0x9, 0xaaaaf01a0490?})
	bufio/bufio.go:241 +0x1b4
io.ReadAtLeast({0xaaaafa1061d8, 0x4003350ba0}, {0x4001d689e0, 0x9, 0x9}, 0x9)
	io/io.go:335 +0xa0
io.ReadFull(...)
	io/io.go:354
golang.org/x/net/http2.readFrameHeader({0x4001d689e0, 0x9, 0x5fe38fa?}, {0xaaaafa1061d8?, 0x4003350ba0?})
	golang.org/x/net@v0.33.0/http2/frame.go:237 +0x58
golang.org/x/net/http2.(*Framer).ReadFrame(0x4001d689a0)
	golang.org/x/net@v0.33.0/http2/frame.go:501 +0x78
google.golang.org/grpc/internal/transport.(*http2Client).reader(0x40033386c8, 0x4003350c00)
	google.golang.org/grpc@v1.67.3/internal/transport/http2_client.go:1645 +0x154
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 101
	google.golang.org/grpc@v1.67.3/internal/transport/http2_client.go:412 +0x1698

goroutine 62 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x400331b6c0, 0x1)
	google.golang.org/grpc@v1.67.3/internal/transport/controlbuf.go:412 +0x140
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x400219dc00)
	google.golang.org/grpc@v1.67.3/internal/transport/controlbuf.go:575 +0x7c
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
	google.golang.org/grpc@v1.67.3/internal/transport/http2_client.go:470 +0xb8
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 101
	google.golang.org/grpc@v1.67.3/internal/transport/http2_client.go:468 +0x1b18

goroutine 65 [select]:
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinWriter(0x4001e0a700, {0xaaaafa1df048, 0x4001e93e40}, 0x40024ed080)
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:518 +0xd4
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip.func2()
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:496 +0x64
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).checkinRoundTrip in goroutine 102
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:494 +0x1bc

goroutine 114 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	google.golang.org/grpc@v1.67.3/stream.go:393 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 103
	google.golang.org/grpc@v1.67.3/stream.go:392 +0xb2c

goroutine 115 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0x40032845a0)
	google.golang.org/grpc@v1.67.3/internal/transport/transport.go:378 +0x6c
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
	google.golang.org/grpc@v1.67.3/internal/transport/transport.go:393
google.golang.org/grpc.(*csAttempt).recvMsg(0x400333c680, {0xaaaaf9bc2fc0, 0x4003292200}, 0xaaaaf014981c?)
	google.golang.org/grpc@v1.67.3/stream.go:1115 +0xc0
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x10?)
	google.golang.org/grpc@v1.67.3/stream.go:969 +0x28
google.golang.org/grpc.(*clientStream).withRetry(0x4001a83200, 0x4003238ea0, 0x4003238e90)
	google.golang.org/grpc@v1.67.3/stream.go:789 +0x188
google.golang.org/grpc.(*clientStream).RecvMsg(0x4001a83200, {0xaaaaf9bc2fc0?, 0x4003292200?})
	google.golang.org/grpc@v1.67.3/stream.go:968 +0xf4
github.com/elastic/elastic-agent-client/v7/pkg/proto.(*elasticAgentActionsClient).Recv(0x40020403f0)
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/proto/elastic-agent-client_grpc.pb.go:140 +0x58
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionsReader(0x4001e0a700, {0xaaaafa1deff0, 0x40020403f0}, 0x4001afc8a0)
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:848 +0x38
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip.func1()
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:829 +0x88
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip in goroutine 103
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:826 +0x150

goroutine 116 [select]:
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionsWriter(0x4001e0a700, {0xaaaafa1deff0, 0x40020403f0}, 0x4001afc8a0, 0x40024ed200)
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:885 +0x140
github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip.func2()
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:837 +0x94
created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).actionRoundTrip in goroutine 103
	github.com/elastic/elastic-agent-client/v7@v7.15.0/pkg/client/client_v2.go:834 +0x1e4

@cmacknz
Copy link
Member

cmacknz commented Feb 20, 2025

I suspect this is from the ES state store seeing it and the top of the stack trace and also what it is doing in that stack trace:

func openStore(ctx context.Context, log *logp.Logger, name string, notifier *Notifier) (*store, error) {
ctx, cn := context.WithCancel(ctx)
s := &store{
ctx: ctx,
cn: cn,
log: log.With("name", name).With("backend", "elasticsearch"),
name: name,
index: renderIndexName(name),
notifier: notifier,
chReady: make(chan struct{}),
}
chCfg := make(chan *conf.C)
unsubFn := s.notifier.Subscribe(func(c *conf.C) {
select {
case chCfg <- c:
case <-ctx.Done():
}
})
go s.loop(ctx, cn, unsubFn, chCfg)

CC @orestisfl

The agent policy here has multiple CEL inputs running the same Filebeat process, so perhaps related to that.

@orestisfl
Copy link
Contributor

Hmm, can't see anything in the stack trace that is supposed to be writing to the configuration value for it to cause a concurrent map read and map write. However, if not all go routines are captured at the exact moment when the exception happens, the most likely candidate I see is goroutine 105:

goroutine 105 [runnable]:
runtime.CallersFrames(...)
	runtime/symtab.go:77
go.uber.org/zap/internal/stacktrace.Capture(0x5, 0x0)
	go.uber.org/zap@v1.27.0/internal/stacktrace/stack.go:107 +0x158
go.uber.org/zap.(*Logger).check(0x4003330d00, 0x0, {0x40033366c0, 0x118})
	go.uber.org/zap@v1.27.0/logger.go:379 +0x540
go.uber.org/zap.(*Logger).Check(...)
	go.uber.org/zap@v1.27.0/logger.go:222
go.uber.org/zap.(*SugaredLogger).log(0x40020f8730, 0x0, {0xaaaaf723322b?, 0x24?}, {0x4003378cb0?, 0x2?, 0x2?}, {0x0, 0x0, 0x0})
	go.uber.org/zap@v1.27.0/sugar.go:354 +0xb8
go.uber.org/zap.(*SugaredLogger).Infof(...)
	go.uber.org/zap@v1.27.0/sugar.go:204
github.com/elastic/elastic-agent-libs/logp.(*Logger).Infof(...)
	github.com/elastic/elastic-agent-libs@v0.18.1/logp/logger.go:156
github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES({_, _}, {{0xaaaaf710115c, 0x8}, {0xaaaaf710115c, 0x8}, {0x4001fdde00, 0xb}, 0x1, {0x4001fff700, ...}, ...}, ...)
	github.com/elastic/beats/v7/libbeat/outputs/elasticsearch/elasticsearch.go:57 +0x204
github.com/elastic/beats/v7/libbeat/outputs.Load({_, _}, {{0xaaaaf710115c, 0x8}, {0xaaaaf710115c, 0x8}, {0x4001fdde00, 0xb}, 0x1, {0x4001fff700, ...}, ...}, ...)
	github.com/elastic/beats/v7/libbeat/outputs/output_reg.go:109 +0xd4
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createOutput(0x4001d8c008, {0xaaaafa201280, 0x4003330800}, {{0x40032bc410, 0xd}, 0x4003381560})
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:1505 +0x198
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).Reload.func1({0xaaaafa201280?, 0x4003330800?})
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:179 +0x68
github.com/elastic/beats/v7/libbeat/publisher/pipeline.loadOutput({0x400223ae00?, 0x40022f0100?, 0x4001e93690?, 0x40018e1ba0?}, 0x40033794a8?)
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/module.go:139 +0x12c
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*outputController).Reload(0x4001e8e480, 0x4003386100, 0x40020520a0)
	github.com/elastic/beats/v7/libbeat/publisher/pipeline/controller.go:177 +0xbc
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater.(*Beat).makeOutputReloader.func2(0x4003386100)
	github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:1392 +0xd0
github.com/elastic/beats/v7/libbeat/common/reload.ReloadableFunc.Reload(0x40032ba510?, 0x40020399ff?)
	github.com/elastic/beats/v7/libbeat/common/reload/reload.go:226 +0x28
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).reloadOutput(0x4002edd0e0, 0x40032a8540)
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:784 +0x270
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).reload(0x4002edd0e0, 0x4003289e40)
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:646 +0x7d0
github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen(0x4002edd0e0)
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:547 +0x6c4
created by github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).Start in goroutine 1
	github.com/elastic/beats/v7/x-pack/libbeat/management/managerV2.go:295 +0x1f0

This line here is just 4 lines above the current line:

overriddenFields, presetConfig, err := ApplyPreset(preset, cfg)

and ApplyPreset() calls Merge():
err = userConfig.Merge(presetConfig)

@olegsu olegsu added this to the 8.18 milestone Feb 26, 2025
@blakerouse
Copy link
Contributor

blakerouse commented Feb 28, 2025

I think the issue is here:

outCfg := conf.Namespace{}
if err := r.Config.Unpack(&outCfg); err != nil || outCfg.Name() != "elasticsearch" {
logp.Err("Failed to unpack the output config: %v", err)
return nil
}
stateStore.notifier.Notify(outCfg.Config())
return nil

At first you would think this would ensure that you get a unique set of memory for the outCfg := conf.Namespace{}. But being that r.Config is already a *config.C the code path will hit the following:

https://github.com/elastic/go-ucfg/blob/5e68a5e3f8ac97168b6d7a5fb714cda391442606/reify.go#L166-L168

Which will just merge at the key/value level and keep the same memory addresses of the values from r.Config. That means changes to r.Config inside of another goroutine will then affect the same objects that is passed into the Notifier.

I would recommend ensuring a full copy is taken here:

stateStore.notifier.Notify(outCfg.Config())

That will ensure that it will not result in concurrent access.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Agent Filebeat Filebeat :integrations needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Development

No branches or pull requests

6 participants