Multiple Managed Alertmanager and alerts centralisation

[m4vr0x]

Hi,

We currently have 8 GKE clusters using GMP (version=0.7.0) in Managed mode.
We would like to centralise alerting as described below:

• All rule-evaluator send their alerts to one Managed Alertmanager in cluster-1
• Managed Alertmanager in cluster-1 is using a webhook receiver to forward all alerts from all cluster to a custom alert router connected to different channels (slack, pager,...).

With our current config we are able to send alerts generated directly from Managed Alertmanager in cluster-1 but none of the alerts coming from the 7 other clusters.

Base on my understanding, even if our alertmanagers are all Managed, we should use this setup to configure additional alertmanagers into the gmp collectors: Managed rule evaluation and alerting - Self-deployed Alertmanager to ensure all collectors are sending their alerts to cluster-1

Here is the config on all clusters but cluster-1:

• GMP Operator config

apiVersion: monitoring.googleapis.com/v1
kind: OperatorConfig
metadata:
  name: config
  namespace: gmp-public
rules:
  alerting:
    alertmanagers:
      - name: alertmanager-gateway
        namespace: gmp-public
        port: "443"
        scheme: https

• The service endpoint configuration pointing to Managed alertmanager in cluster-1:

apiVersion: v1
kind: Service
metadata:
  name: alertmanager-gateway
  namespace: gmp-public
spec:
  type: ExternalName
  externalName: external.dns.to.managed.alertmanager.on.cluster-1.com

To complete, we already use this design with "community" Prometheus/AlertManager without issues and we are also able receive (and forward) in Managed Alertmanager in cluster-1 alerts from these ones with following config:

• Prometheus operator config

---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: community-prometheus
  [...]
spec:
  additionalAlertManagerConfigs:
    name: external-alertmanager-config
    key: additional-alertmanager-configs.yaml
[...]

• additional-alertmanager-configs.yaml from secret:

alertmanager:
  external:
    config: >
      - static_configs:
        - targets:
          - alertmanager.prometheus.svc.cluster.local
        tls_config:
          insecure_skip_verify: true
        scheme: https

• The service endpoint configuration pointing to alertmanager in cluster-1:

apiVersion: v1
kind: Service
metadata:
  name: alertmanager
  namespace: prometheus
spec:
  type: ExternalName
  externalName: external.dns.to.alertmanager.on.cluster-1.com

And we also tried to replicate this config, as documented in Managed rule evaluation and alerting - Managed Alertmanager in the gmp operator without success:

apiVersion: monitoring.googleapis.com/v1
kind: OperatorConfig
metadata:
  name: config
  namespace: gmp-public
managedAlertmanager:
  configSecret:
    name: alertmanager-external-config
    key: managed-config.yaml

• managed-config.yaml from secret:

alerting:
  alertmanagers:
    - static_configs:
      - targets:
        - alertmanager-gateway.gmp-public.svc.cluster.local
      tls_config:
        insecure_skip_verify: true
      scheme: https

Finally, we can see using the ALERTS metrics that all the alerts are successfully fired from all clusters in my metrics scope and stored in Monarch:

ALERTS{alertname="AlwaysFiring", alertstate="firing", cluster="cluster-1", location="region-zone-1", namespace="gmp-public", project_id="project-1"}
1
ALERTS{alertname="AlwaysFiring", alertstate="firing", cluster="cluster-2", location="region-zone-1", namespace="gmp-public", project_id="project-2"}
1
ALERTS{alertname="AlwaysFiring", alertstate="firing", cluster="cluster-3", location="region-zone-1", namespace="gmp-public", project_id="project-3"}
1
[...]
ALERTS{alertname="AlwaysFiring", alertstate="firing", cluster="cluster-8", location="region-zone-1", namespace="gmp-public", project_id="project-8"}
1

Now questions:

• Which setup is relevant in our use case between Self-deployed Alertmanager or Managed Alertmanager?
• Is there anyone using a similar setup with success?
• Am I missing something? :-)

[lyanco]

I believe in this case you'll have to use the self-deployed alertmanager, as you found, as the Managed Alertmanagers don't support out of cluster alerts. I can't really tell you what's failing here, but we only designed and tested the managed alertmanager to handle in-cluster alerts. If you deploy your own AM it should be able to receive from the managed rule evaluators.

We don't have anyone else trying to centralize on one managed alertmanager... the pattern we have is consistent with what most people do in the wild with Prometheus Operator.

That being said, we understand the use case of centralizing your alerts and alert management in one place. To solve this problem we are emphasizing the new way to create PromQL alerts in Cloud Monitoring, soon to be GA, which allows you to upload rule_files and alertmanager config and have it all execute centrally, in the cloud, as a FULLY fully managed service. We're not getting rid of the current managed rule eval/managed AM path, but we will be deemphasizing it.

Hope this helps!

[m4vr0x]

Hi @lyanco,
Thank you for your feedback and your time.

I get that this is still pre GA and the emphasize is on using alerting in Cloud Monitoring but we just follow the official documentation (Self-deployed Alertmanager) and trying to make it work. :-)

Actually we ran more tests with our existing self-deployed alertmanager and the problem seems to come from the Managed Prometheus Operator:

Test 1 :

• Self-deployed prometheus on cluster-2 is configured to send alerts to Managed Alertmanager on cluster-1. It works

Test 2 :

• Managed prometheus on cluster-2 is configured to send alerts to Managed Alertmanager on cluster-1. It does not work

Is there to way to increase logs verbosity from the collector container in the rule-evaluator pod or gmp-operator pod to actually see where/when fired alert are sent to the different alertmanagers?

[pintohutch]

Hi @m4vr0x,

Thanks for raising.

IIUC you're interested in forwarding alerts from your rule-evaluators from each cluster to a central alertmanager instance in cluster-1, correct?

Can you share or compare the alertmanager_config for the (working) self-deployed prometheus and the managed rule-evaluator? You can get the latter via:

kubectl get cm -ngmp-system rule-evaluator -oyaml

You may also take a look at potential error logs there:

kubectl logs -f deploy/rule-evaluator -ngmp-system -c evaluator

[m4vr0x]

Hi @pintohutch,

IIUC you're interested in forwarding alerts from your rule-evaluators from each cluster to a central alertmanager instance in cluster-1, correct?

Correct

• Here's the alertmanager_config for the (working) self-deployed prometheus:

- static_configs:
  - targets:
    - alertmanager-gateway.gmp-public.svc.cluster.local
  tls_config:
    insecure_skip_verify: true
  scheme: https

• And the managed rule-evaluator:

❯ kubectl get cm -ngmp-system rule-evaluator -oyaml
apiVersion: v1
data:
  config.yaml: |
    global: {}
    alerting:
        alertmanagers:
            - follow_redirects: true
              enable_http2: true
              scheme: http
              timeout: 10s
              api_version: v2
              static_configs:
                - targets:
                    - alertmanager.gmp-system:9093
            - tls_config:
                insecure_skip_verify: true
              follow_redirects: true
              enable_http2: true
              scheme: https
              path_prefix: /
              timeout: 10s
              api_version: v2
              relabel_configs:
                - source_labels: [__meta_kubernetes_endpoints_name]
                  regex: alertmanager-gateway
                  action: keep
                - source_labels: [__address__]
                  regex: (.+):\d+
                  target_label: __address__
                  replacement: $1:443
                  action: replace
              kubernetes_sd_configs:
                - role: endpoints
                  kubeconfig_file: ""
                  follow_redirects: true
                  enable_http2: true
                  namespaces:
                    own_namespace: false
                    names:
                        - gmp-public
    rule_files:
        - /etc/rules/*.yaml
kind: ConfigMap
metadata:
  name: rule-evaluator
  namespace: gmp-system
 [...]

I'm not sure to understand but static_configs.targets seems to be missing for the "external" alertmanager defined in the operator config?
I guess it should be:

              static_configs:
                - targets:
                    - alertmanager-gateway.gmp-public.svc.cluster.local

Because the service alertmanager-gateway in gmp-public is pointing to the central alertmanager instance in cluster-1 (as configured in my first post).

Also not sure why alertmanager-gateway is used in relabel_configs and gmp-public in kubernetes_sd_configs?

[pintohutch]

Hey @m4vr0x,

Great questions.

I'm not sure to understand but static_configs.targets seems to be missing for the "external" alertmanager defined in the operator config?

We "hardcode" the static config for the managed alertmanager that comes with managed collection without using K8s service discovery. That's why that is there.

I believe the reason this isn't working is because the rule-evaluator is looking for Kubernetes Endpoint objects to discover your alertmanager gateway. Since it's an externalName service without selectors, those endpoints aren't created.

In order to support externalName services, we would need to add a feature - either to support static configs, as prometheus-operator does. or maybe find some other way to do this.

However, I wonder if you're able to manually create an Endpoint object and attach it to your Service, as discussed here, if that could work?

[m4vr0x]

Hi @pintohutch,
Thanks again for your time and feedbacks.

I believe the reason this isn't working is because the rule-evaluator is looking for Kubernetes Endpoint objects to discover your alertmanager gateway. Since it's an externalName service without selectors, those endpoints aren't created.
[...]
However, I wonder if you're able to manually create an Endpoint object and attach it to your Service, as discussed here, if that could work?

We looked into EndpointSlices and even if it works, using IP address to connect to k8s services is not really a suitable option for us.

In order to support externalName services, we would need to add a feature - either to support static configs, as prometheus-operator does. or maybe find some other way to do this.

Is there any chance this could be added in the roadmap as an enhancement?

[pintohutch]

Opened #595 to track. Thanks for reporting!