Aerospike Monitoring Stack

This is now in beta. If you're an enterprise customer feel free to reach out to support with any questions. We appreciate feedback from community members on the issues.

Installing Aerospike Prometheus Exporter

Aerospike Prometheus Exporter can be compiled and built from the source aerospike/aerospike-prometheus-exporter GitHub repository.

Use build/docker-compose.yaml file to quickly build aerospike-prometheus-exporter binary for linux and osx platforms.

git clone https://github.com/aerospike/aerospike-monitoring.git
cd aerospike-monitoring/

docker-compose -f build/docker-compose.yaml up

Aerospike Prometheus Exporter requires a configuration file to run. Here's a sample configuration file.

curl https://raw.githubusercontent.com/aerospike/aerospike-prometheus-exporter/v1.0.0/aeroprom.conf.dev -o ./build/ape.toml

Aerospike Prometheus Exporter Configuration

As a minimum required configuration, edit ./build/ape.toml to add db_host and db_port to point to an Aerospike server IP and port.
```
[Aerospike]

db_host="localhost"
db_port=3000
```

Update Aerospike security and TLS configurations (if applicable),

# certificate file
cert_file=""

# key file
key_file=""

# node TLS name for authentication
node_tls_name=""

# root certificate file
root_ca=""

# authentication mode: internal (for server), external (LDAP, etc.)
auth_mode=""

# database user
user=""

# database password
password=""

Update exporter's bind address and port (default: 0.0.0.0:9145), and add tags.
```
bind=":9145"

tags=['agent', 'aerospike']
```

Installation

Add the aerospike-prometheus-exporter binary to /usr/local/bin and add ape.toml file to /etc/aerospike-prometheus-exporter/ape.toml.

cp ./build/aerospike-prometheus-exporter-linux /usr/local/bin/aerospike-prometheus-exporter
mkdir /etc/aerospike-prometheus-exporter
cp ./build/ape.toml /etc/aerospike-prometheus-exporter/ape.toml

Create a systemd unit file for aerospike-prometheus-exporter.service,

# cat /usr/lib/systemd/system/aerospike-prometheus-exporter.service

[Unit]
Description=Aerospike Prometheus Exporter Service
Documentation=https://github.com/aerospike/aerospike-prometheus-exporter
Wants=network.target
After=network.target

[Service]
ExecStart=/usr/local/bin/aerospike-prometheus-exporter --config /etc/aerospike-prometheus-exporter/ape.toml

[Install]
WantedBy=multi-user.target

systemctl daemon-reload

Run aerospike-prometheus-exporter,

systemctl start aerospike-prometheus-exporter.service

Prometheus Configuration

This section covers the configuration of Prometheus server to scrape metrics from Aerospike Prometheus Exporter, send alerts to an Alertmanager instance and add an Aerospike alert rules configuration file.

Install Prometheus

https://prometheus.io/docs/prometheus/latest/installation/

Configuration

Create a Prometheus configuration file /etc/prometheus/prometheus.yml,

Add scrape_configs with targets pointing to each instance of aerospike-prometheus-exporter.

scrape_configs:
  - job_name: 'aerospike'
    static_configs:
      - targets: ['172.20.0.2:9145', '172.20.0.3:9145']

To enable alerting, add alertmanager instance's IP:Port target.

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - "alertmanager:9093"

Add aerospike rules configuration file to path /etc/prometheus/aerospike_rules.yaml and configure rule_files to point to the same.

rule_files:
  - "/etc/prometheus/aerospike_rules.yaml"

# /etc/prometheus/prometheus.yml

# Global Configuration
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

# Alertmanager Configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - "alertmanager:9093"

# Add Aerospike Rules YAML File
rule_files:
- "/etc/prometheus/aerospike_rules.yaml"

# Scrape Configuration. Must point to Aerospike Prometheus Exporter
scrape_configs:
# The job name 'aerospike' is added as a label `job=aerospike` to any timeseries scraped from this config.
- job_name: 'aerospike'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
  static_configs:
    - targets: ['172.20.0.2:9145', '172.20.0.3:9145']

Create a data directory for Prometheus to store the metrics data,
```
mkdir /var/lib/prometheus
```

Start Prometheus Server

Start Prometheus server.

/usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml \
                          --storage.tsdb.path /var/lib/prometheus/

OR,

(Optional) Create a systemd unit file for prometheus.service,

# cat /usr/lib/systemd/system/prometheus.service

[Unit]
Description=Prometheus Service
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network.target
After=network.target

[Service]
ExecStart=/usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/

[Install]
WantedBy=multi-user.target

systemctl daemon-reload

systemctl start prometheus.service

Grafana Configuration

This section covers the configuration of Grafana server to add Prometheus datasource, Alermanager datasource and Aerospike dashboards.

Install Grafana

https://grafana.com/docs/grafana/latest/installation/

Configuration

Add provisioning config directory path in /etc/grafana/grafana.ini
```
[paths]
provisioning = /etc/grafana/provisioning/
```

Create two directories dashboards and datasources under provisioning path.

mkdir -p /etc/grafana/provisioning/datasources
mkdir -p /etc/grafana/provisioning/dashboards

Create a file all.yaml in /etc/grafana/provisioning/dashboards/ directory to add a dashboard provider that will load dashboards into Grafana from the local filesystem. Here, the path to dashboard files is configured as /var/lib/grafana/dashboards.

# /etc/grafana/provisioning/dashboards/all.yaml

apiVersion: 1

providers:
# <string> an unique provider name
- name: 'default'
    # <int> org id. will default to orgId 1 if not specified
    # orgId: 1

    # <string, required> name of the dashboard folder. Required
    folder: 'Aerospike'

    # <string, required> provider type. Required
    type: file

    # <bool> disable dashboard deletion
    disableDeletion: false

    # <bool> enable dashboard editing
    editable: true

    # <int> how often Grafana will scan for changed dashboards
    updateIntervalSeconds: 10

    options:
    # <string, required> path to dashboard files on disk. Required
      path: /var/lib/grafana/dashboards

Create a file all.yaml in /etc/grafana/provisioning/datasources/ directory to define datasources.

# /etc/grafana/provisioning/datasources/all.yaml

apiVersion: 1

datasources:
- name: "Aerospike Prometheus Alertmanager"
  type: camptocamp-prometheus-alertmanager-datasource
  access: proxy
  url: http://alertmanager:9093
  editable: true
  isDefault: false

- name: "Aerospike Prometheus"
  type: prometheus
  access: proxy
  url: http://prometheus:9090
  editable: true
  isDefault: false

Install camptocamp-prometheus-alertmanager-datasource plugin for Alertmanager datasource.
```
grafana-cli plugins install camptocamp-prometheus-alertmanager-datasource
```
Add dashboard files from this repository to /var/lib/grafana/dashboards/ directory.

Start Grafana Server

Start Grafana server.
```
systemctl start grafana-server
```

Alertmanager Configuration

Alertmanager supports various configuration depending on where the notifications need to be sent.

An example configuration for generating notifications to a slack channel is shown below.

# This is an example alertmanager.yaml which sends alert notifications to a slack channel.

global:
  slack_api_url: "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
route:
  group_by: ['alertname', 'cluster', 'service']
  receiver: slack_general

receivers:
- name: slack_general
  slack_configs:
  - channel: '#channel'
    text: "summary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}"

For more details, check Alertmanager Documentation.