Arkime container unhealthy

[devilman85]

I installed the new version of Malcolm but only have the arkime unhelathy container. My instance of Malcolm is remote on an elasticsearch cluster.

[devilman85]

error in post malcom start:

arkime-1 | Creating history ilm policy 'arkime_molochhistory' with: deleteTime 91d
arkime-1 | Creating sessions ilm policy 'arkime_molochsessions' with: forceTime: 3d deleteTime: 90d segments: 1 replicas: 1
arkime-1 | You will need to run db.pl upgrade with --ilm to update the templates the first time you turn ilm on.
arkime-1 | This is a fresh Arkime install
arkime-1 | Looks like Arkime wasn't installed, must do init at /opt/arkime/db/db.pl line 6150.
arkime-1 | ilm created
arkime-1 | {"_shards":{"total":574,"successful":574,"failed":0}}2025-01-13 08:39:36,743 INFO exited: initialize (exit status 0; expected)
arkime-1 | Launch viewer...
arkime-1 | WARNING - Using authMode=header since not set, add to config file to silence this warning.
arkime-1 | ERROR - Couldn't retrieve database version, is OpenSearch/Elasticsearch running? Have you run ./db.pl host:port init? ResponseError: Response Error
arkime-1 | at onBody (/opt/arkime/node_modules/@elastic/elasticsearch/lib/Transport.js:311:23)
arkime-1 | at IncomingMessage.onEnd (/opt/arkime/node_modules/@elastic/elasticsearch/lib/Transport.js:240:11)
arkime-1 | at IncomingMessage.emit (node:events:531:35)
arkime-1 | at endReadableNT (node:internal/streams/readable:1696:12)
arkime-1 | at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
arkime-1 | meta: {
arkime-1 | body: {},
arkime-1 | statusCode: 404,
arkime-1 | headers: {
arkime-1 | 'x-elastic-product': 'Elasticsearch',
arkime-1 | 'content-type': 'application/json',
arkime-1 | 'content-length': '2'
arkime-1 | },
arkime-1 | meta: {
arkime-1 | context: null,
arkime-1 | request: [Object],
arkime-1 | name: 'elasticsearch-js',
arkime-1 | connection: [Object],
arkime-1 | attempts: 0,
arkime-1 | aborted: false
arkime-1 | }
arkime-1 | }
arkime-1 | }

Having activated the ilm maybe I created problems with the arkime installation?

I removed the directory with 24.11 and did the git clone of 24.12

[mmguero]

 arkime-1 | ERROR - Couldn't retrieve database version, is OpenSearch/Elasticsearch running? Have you run ./db.pl host:port init? ResponseError: Response Error

That seems to be the issue. Do you see anything in the output of ./scripts/logs -s opensearch that might indicate why it can't connect to the opensearch instance? What happens in the output of ./scripts/logs -s arkime after the error message you shared?

[mmguero]

Okay, so looking at this it looks like you're running with a remote elasticsearch instance instead of Malcolm's built-in opensearch instance. It appears to me that Arkime is unable to talk to your ElasticSearch instance.

Double check the settings in your ./config/opensearch.env file (grep -v '^#' config/opensearch.env) and make sure that the OPENSEARCH_URL seems to be correct. You could also run this command, which will look something like this:

$ docker compose exec arkime opensearch_status.sh
elasticsearch-remote is up and healthy at https://x.x.x.x:9200

[devilman85]

the elastic node is healthy. probably something happened to the indexes and they no longer make malcolm communicate with elasticsearch. i would also not want that by not deleting the old arkime_sessions index from 24.11 and doing the 24.12 startup has compromised the communication that used to work and thus also the startup of arkime

[mmguero]

I guess you could shut down Malcolm and delete all of the arkime indexes in your elasticsearch cluster. Upon restarting Malcolm Arkime should try to go through and reinitialize things.

The root of the problem is:

ERROR - Couldn't retrieve database version, is OpenSearch/Elasticsearch running? Have you run ./db.pl host:port init? ResponseError: Response Error

but that should be happening automatically on Malcolm startup, and I don't know how to debug "ResponseError: Response Error"

[mmguero]

You could also double check the credentials stored in the .opensearch.primary.curlrc file in Malcolm.

[devilman85]

the credential is correct. I will restore the cluster in the next few days