Proxy

.gitignore

@@ -18,3 +18,5 @@ media/
 .terraform/
 .nox/
 __pycache__
+./central-prometheus.yml
+./on-site-prometheus.yml

README.md

@@ -1,7 +1,14 @@
 # bittensor-prometheus-proxy
 
-Proxy that allows for pushing prometheus metrics signed with bittensor wallets
+Proxy that allows for 
 
+1. pushing prometheus metrics signed with bittensor wallets. Operating in this manner does not require a db or redis.
+2. verifying incoming signed metrics. Operating in this manner does not require a wallet. Verification is two-fold:
+   1. the full payload is signed, both the signature and the hotkey are included in the headers - that is verified 
+   2. the metrics data blob is unpacked and each metric is checked for the "hotkey" label - it has to be the same as
+      the value in the header
+
+![Diagram](./docs/diagram.svg)
 - - -
 
 # Base requirements
@@ -15,11 +22,20 @@ Proxy that allows for pushing prometheus metrics signed with bittensor wallets
 
 ```sh
 ./setup-dev.sh
-docker compose up -d
+docker compose up -d  # this will also start node_Exporter and two prometheus instances
 cd app/src
 pdm run manage.py wait_for_database --timeout 10
 pdm run manage.py migrate
-pdm run manage.py runserver
+pdm run manage.py runserver 0.0.0.0:8000
+```
+
+this setup requires a working bittensor wallet (for the on-site prometheus to read the hotkey and so that the proxy
+can sign requests). Requests will be sent from on-site prometheus to proxy then to the same proxy (different view 
+though) and to the central prometheus. Starting celery and celery beat is not, however, required for local development,
+because instead of having a periodic task populate the validator list, one can add records to it manually using
+
+```bash
+python manage.py debug_add_validator <hotkey>
 ```
 
 # Setup production environment (git deployment)
@@ -158,39 +174,6 @@ with some_calculation_time.labels('blabla').time():
 
 
 
-# Cloud deployment
-
-## AWS
-
-<details>
-Initiate the infrastructure with Terraform:
-TODO
-
-To push a new version of the application to AWS, just push to a branch named `deploy-$(ENVIRONMENT_NAME)`.
-Typical values for `$(ENVIRONMENT_NAME)` are `prod` and `staging`.
-For this to work, GitHub actions needs to be provided with credentials for an account that has the following policies enabled:
-
-- AutoScalingFullAccess
-- AmazonEC2ContainerRegistryFullAccess
-- AmazonS3FullAccess
-
-See `.github/workflows/cd.yml` to find out the secret names.
-
-For more details see [README_AWS.md](README_AWS.md)
-</details>
-
-## Vultr
-
-<details>
-Initiate the infrastructure with Terraform and cloud-init:
-
-- see Terraform template in `<project>/devops/vultr_tf/core/`
-- see scripts for interacting with Vultr API in `<project>/devops/vultr_scripts/`
-  - note these scripts need `vultr-cli` installed
-
-For more details see [README_vultr.md](README_vultr.md).
-</details>
-
 # Backups
 
 <details>

app/bittensor_prometheus/Dockerfile

@@ -0,0 +1,36 @@
+# Use Python base image from DockerHub
+FROM python:3.11
+
+RUN pip install bittensor==8.2.0
+
+WORKDIR /app
+
+RUN apt-get update && \
+    apt-get install -y wget curl && \
+    wget https://github.com/prometheus/prometheus/releases/download/v2.55.0/prometheus-2.55.0.linux-amd64.tar.gz && \
+    tar xvzf prometheus-*.tar.gz && \
+    mkdir /etc/prometheus && \
+    mv prometheus-2.55.0.linux-amd64/prometheus /bin/ && \
+    mv prometheus-2.55.0.linux-amd64/promtool /bin/ && \
+    mv prometheus-2.55.0.linux-amd64/prometheus.yml /etc/prometheus/ && \
+    rm -rf prometheus-*.tar.gz prometheus-2.55.0.linux-amd64 && \
+    chown -R nobody:nogroup /etc/prometheus && \
+    chown -R nobody:nogroup /etc/prometheus
+
+RUN chown nobody: /etc/prometheus
+
+COPY read_wallet_and_substitute_config.py /app/
+RUN chown -R nobody: /app/
+
+COPY entrypoint.sh /
+RUN chown nobody: /entrypoint.sh
+
+RUN mkdir /nonexistent
+RUN chown nobody: /nonexistent
+
+RUN mkdir /wallets
+RUN chown nobody: /wallets
+
+#USER nobody
+
+ENTRYPOINT ["/entrypoint.sh"]

app/bittensor_prometheus/README.md

@@ -0,0 +1,32 @@
+Here lie the tools to build an image that runs prometheus but before starting reads the hotkey of a configured (
+via env vars) bittensor wallet and allows for including that hotkey in prometheus' config.
+
+
+To run it, you need to provide a template of the prometheus config (only the hotkey part is meant to substituted when
+materializing this template), mount your wallet and specify it using env vars. for example:
+
+config:
+
+```yaml
+global:
+  scrape_interval: 5s
+  evaluation_interval: 5s
+
+scrape_configs:
+  - job_name: "node"
+    scrape_interval: 5s
+    static_configs:
+      - targets: ['host.docker.internal:9100']
+        labels:
+          hotkey: '{hotkey}'  # the 'template engine' in use is python's str.format()
+
+```
+
+running:
+
+docker run \
+  -v config.yml:/etc/prometheus/prometheus.yml.template \
+  -v /home/user/.bittensor/wallets/:/wallets/ \
+  -e BITTENSOR_WALLET_NAME=validator \
+  -e BITTENSOR_WALLET_HOTKEY_NAME=default \
+  backenddevelopersltd/bittensor_prometheus

app/bittensor_prometheus/entrypoint.sh

@@ -0,0 +1,3 @@
+#!/bin/sh -e
+python /app/read_wallet_and_substitute_config.py
+exec /bin/prometheus --config.file=/etc/prometheus/prometheus.yml "$@"

app/bittensor_prometheus/read_wallet_and_substitute_config.py

@@ -0,0 +1,35 @@
+import os
+import pathlib
+
+import bittensor
+
+
+BITTENSOR_WALLET_NAME = os.environ.get("BITTENSOR_WALLET_NAME")
+BITTENSOR_WALLET_HOTKEY_NAME = os.environ.get("BITTENSOR_WALLET_HOTKEY_NAME")
+
+if not BITTENSOR_WALLET_NAME or not BITTENSOR_WALLET_HOTKEY_NAME:
+    raise RuntimeError("You must set BITTENSOR_WALLET_NAME and BITTENSOR_WALLET_HOTKEY_NAME env vars")
+
+
+def get_wallet() -> bittensor.wallet:
+    wallet = bittensor.wallet(
+        name=BITTENSOR_WALLET_NAME,
+        hotkey=BITTENSOR_WALLET_HOTKEY_NAME,
+        path="/wallets",
+    )
+    wallet.hotkey_file.get_keypair()  # this raises errors if the keys are inaccessible
+    return wallet
+
+
+def read_and_substitute_config(hotkey: str):
+    tmpl = pathlib.Path("/etc/prometheus/prometheus.yml.template").read_text()
+    pathlib.Path("/etc/prometheus/prometheus.yml").write_text(tmpl.format(hotkey=hotkey))
+
+
+def main():
+    wallet = get_wallet()
+    read_and_substitute_config(wallet.hotkey.ss58_address)
+
+
+if __name__ == "__main__":
+    main()