Fix concurrent read write error for entity store

[sky333999]

Description of the issue

Fixes #1479. Entity store's operations to read and write the logFile and logGroups maps can occur from diff go routines corresponding to files monitored by the agent and can lead to a panic if accessed concurrently.

Description of changes

Use a mutex lock to prevent concurrent access to the log maps.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Wrote up a script to generate a high load of 1000 logs being constantly written to and monitored by the agent. Prior to fix, I see the agent crash every few mins:

$ journalctl -u amazon-cloudwatch-agent --since "2 minutes ago" -f | grep "concurrent"
Jan 03 19:19:17 ip-172-31-27-109.ec2.internal start-amazon-cloudwatch-agent[1084199]:         status code: 404, request id: fatal error: concurrent map read and map write
Jan 03 19:23:39 ip-172-31-27-109.ec2.internal start-amazon-cloudwatch-agent[1098369]:         status code: 404, request id: fatal error: concurrent map read and map write
Jan 03 19:25:45 ip-172-31-27-109.ec2.internal start-amazon-cloudwatch-agent[1125246]:         status code: 404, request id: fatal error: concurrent map read and map write
Jan 03 19:29:37 ip-172-31-27-109.ec2.internal start-amazon-cloudwatch-agent[1138406]:         status code: 404, request id: fatal error: concurrent map read and map write
Jan 03 19:32:39 ip-172-31-27-109.ec2.internal start-amazon-cloudwatch-agent[1162539]:         status code: 404, request id: fatal error: concurrent map read and map write
Jan 03 19:35:05 ip-172-31-27-109.ec2.internal start-amazon-cloudwatch-agent[1181379]:         status code: 404, request id: fatal error: concurrent map read and map write
Jan 03 19:38:47 ip-172-31-27-109.ec2.internal start-amazon-cloudwatch-agent[1195915]:         status code: 404, request id: fatal error: concurrent map read and map write
Jan 03 19:40:41 ip-172-31-27-109.ec2.internal start-amazon-cloudwatch-agent[1218979]:         status code: 404, request id: fatal error: concurrent map read and map write
Jan 03 19:42:51 ip-172-31-27-109.ec2.internal start-amazon-cloudwatch-agent[1230962]:         status code: 404, request id: fatal error: concurrent map read and map write

Post this fix, I no longer see the agent crash.

Requirements

Before commit the code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint

[jefchien]

With this fix, can we remove extension/entitystore from the data race ignore list https://github.com/aws/amazon-cloudwatch-agent/blob/main/Makefile#L206?

[sky333999]

There's more besides this thatll require a bigger restructuring to address :(