Setup metrics and alerting

[mattiekat]

No description provided.

[mattiekat]

An update on the state of this issue. The middlewear was created for extracting metrics from ethers which also gives us a useful framework for more in the future. Metrics are broken in the currently deployed agent version due to a duplicate label name in many of them between the const labels and dynamic ones. This has been fixed in main with #518.

The dashboard for k8s metrics is more or less done (can always add stuff if needed later but should be a really good start). The dashboard for agent metrics is partially done and is mostly waiting on the new metrics to be working.

[mattiekat]

After brainstorming session the list of metrics we came up with are:

Priority:

• Block height
• Count of messages in outboxes (Overall and per agent) by destination chain
• Count of messages processed by inbox by source chain
• Current outbox state (active/failed)
• Relayer balances
• Latest signed index by validator

Other:

• Block rate by chain
• blocks each agent is behind the latest known
• relayer inflight message forwards
• relayer message forward time taken
• gelato operation metric to track successes/failures and time (bundle of a few RPC calls)
• ∆ between inboxes and outbox message processed
• ∆ between outbox index and latest signed index by validator