Skip to content

Commit 18647d5

Browse files
bwplotkafabxc
bwplotka
authored and
fabxc
committed
Added Kube manifests for GCS-enabled setup + store. Added small tutorial. (thanos-io#61)
* Added Kube manifests for GCS-enabled setup + store. Signed-off-by: Bartek Plotka <bwplotka@gmail.com> * Addressed issues. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
1 parent 30b6221 commit 18647d5

File tree

12 files changed

+235
-60
lines changed

12 files changed

+235
-60
lines changed

‎Dockerfile

-1
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,4 @@ LABEL maintainer="The Thanos Authors"
33

44
COPY thanos /bin/thanos
55

6-
USER nobody
76
ENTRYPOINT [ "/bin/thanos" ]

‎kube/README.md

+41-19
Original file line numberDiff line numberDiff line change
@@ -17,29 +17,51 @@ To use cluster from your terminal do:
1717
`source ./kube/envs.sh`
1818

1919
From now on you can use `kubectl` as well as `minikube` command, including `minikube stop` to stop the whole cluster.
20-
21-
## Start Thanos service for Thanos gossip peers
2220

23-
This allows query to discover thanos services.
21+
## Example setup.
2422

25-
```bash
26-
echo "Starting Thanos service for gathering all thanos gossip peers."
27-
kubectl apply -f manifests/thanos
23+
This directory covers are required k8s manifest to start example setup that will include:
24+
- Thanos headless service for discovery purposes.
25+
- Prometheus + Thanos sidecar.
26+
- Thanos query node
2827

29-
```
30-
31-
## Start Prometheus with Thanos sidecar
28+
This setup will have GCS upload disabled, but will show how we can proxy requests from Prometheus.
29+
30+
This example can be easily extended to show the HA Prometheus use case. (TODO)
31+
32+
To run example setup:
33+
1. `bash kube/apply-example.sh`
34+
35+
You will be now able to reach Prometheus on http://prometheus.default.svc.cluster.local:9090/graph
36+
And Thanos Query UI on http://thanos-query.default.svc.cluster.local:19099/graph
37+
38+
Thanos Query UI should show exactly the same data as Prometheus.
39+
40+
To tear down example setup:
41+
1. `bash kube/delete-example.sh`
42+
43+
## Long term storage setup
44+
45+
This example is running setup that is supposed to upload blocks to GCS for long term storage. This setup includes:
46+
- Thanos headless service for discovery purposes.
47+
- Prometheus + Thanos sidecar with GCS shipper configured
48+
- Thanos query node
49+
- Thanos store node.
50+
51+
To run example setup:
52+
1. Create GCS bucket in your GCP project. Either name it "thanos-test" or put its name into
53+
* manifest/prometheus-gcs/deployment.yaml inside `"--gcs.bucket` flag.
54+
* manifest/thanos-query/deployment.yaml inside `"--gcs.bucket` flag.
55+
2. Create service account that have permission to this bucket
56+
3. Download JSON credentials for service account and run: `kubectl create secret generic gcs-credentials --from-file=<your-json-file>`
57+
4. Run `bash kube/apply-lts.sh`
3258

33-
```bash
34-
echo "Starting Prometheus pod with sidecar."
35-
kubectl apply -f kube/manifests/prometheus
36-
```
59+
You will be now able to reach Prometheus on http://prometheus-gcs.default.svc.cluster.local:9090/graph
60+
And Thanos Query UI on http://thanos-query.default.svc.cluster.local:19099/graph
3761

38-
## Start query node targeting Prometheus sidecar
62+
Thanos Query UI should show exactly the same data as Prometheus, but also older data if it's running longer that 12h.
3963

40-
```bash
41-
echo "Starting Thanos query pod targeting sidecar."
42-
kubectl apply -f kube/manifests/thanos-query
43-
```
64+
After 3h sidecar should upload first block to GCS. You can make that quicker by changing prometheus `storage.tsdb.{min,max}-block-duration` to smaller value (e.g 20m)
4465

45-
You can invoke `bash kube/apply-example.sh` that will do all these steps.
66+
To tear down example setup:
67+
1. `bash kube/delete-lts.sh`

‎kube/apply-lts.sh

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/usr/bin/env bash
2+
3+
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
4+
5+
source ${DIR}/envs.sh
6+
7+
cd ${DIR}
8+
9+
echo "Starting Thanos service for gathering all thanos gossip peers."
10+
kubectl apply -f manifests/thanos
11+
12+
echo "Starting Prometheus pod with sidecar."
13+
kubectl apply -f manifests/prometheus-gcs
14+
15+
echo "Starting Thanos query pod targeting sidecar."
16+
kubectl apply -f manifests/thanos-query
17+
18+
echo "Starting Thanos query pod targeting sidecar."
19+
kubectl apply -f manifests/thanos-store

‎kube/delete-lts.sh

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/usr/bin/env bash
2+
3+
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
4+
5+
source ${DIR}/envs.sh
6+
7+
cd ${DIR}
8+
kubectl delete -f manifests/thanos
9+
kubectl delete -f manifests/prometheus-gcs
10+
kubectl delete -f manifests/thanos-query
11+
kubectl delete -f manifests/thanos-store

‎kube/manifests/local/deployment.yaml

-31
This file was deleted.

‎kube/manifests/local/config.yaml ‎kube/manifests/prometheus-gcs/config.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
apiVersion: v1
22
kind: ConfigMap
33
metadata:
4-
name: prometheus
4+
name: prometheus-gcs
55
data:
66
prometheus.yaml: |
77
global:

‎kube/manifests/prometheus-gcs/deployment.yaml

+76
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
apiVersion: extensions/v1beta1
2+
kind: Deployment
3+
metadata:
4+
name: prometheus-gcs
5+
labels:
6+
app: prometheus-gcs
7+
thanos-peer: "true"
8+
spec:
9+
replicas: 1
10+
template:
11+
metadata:
12+
labels:
13+
app: prometheus-gcs
14+
thanos-peer: "true"
15+
spec:
16+
containers:
17+
- image: prom/prometheus:v2.0.0
18+
args: [
19+
"--config.file=/etc/prometheus/config/prometheus.yaml",
20+
"--storage.tsdb.path=/data",
21+
"--storage.tsdb.min-block-duration=2h",
22+
"--storage.tsdb.max-block-duration=2h",
23+
"--storage.tsdb.retention=12h"
24+
]
25+
name: prometheus
26+
resources:
27+
requests:
28+
cpu: 100m
29+
memory: 200Mi
30+
volumeMounts:
31+
- name: config-volume
32+
mountPath: /etc/prometheus/config
33+
- name: tsdb-data
34+
mountPath: /data
35+
# To use your own thanos image you need to do `eval $(minikube docker-env) && make docker` and place thanos/thanos:latest here.
36+
# TODO(bplotka): With vm-driver=none, even that is uses same docker, images are not recognized. Investigate.
37+
- image: bplotka/thanos:latest
38+
imagePullPolicy: Always
39+
env:
40+
- name: GOOGLE_APPLICATION_CREDENTIALS
41+
value: /creds/gcs-credentials.json
42+
args: [
43+
"sidecar",
44+
"--log.level=debug",
45+
"--debug.name=sidecar",
46+
"--api-address=0.0.0.0:19090", # This address will be properly deduced and propagated via cluster.PeerState.
47+
"--metrics-address=0.0.0.0:19190",
48+
"--prometheus.url=http://localhost:9090",
49+
"--tsdb.path=/data",
50+
"--cluster.address=0.0.0.0:19390",
51+
"--cluster.peers=thanos.default.svc.cluster.local:19390",
52+
# This is required to be added in GCS prior startup of this.
53+
"--gcs.bucket=thanos-test"
54+
]
55+
name: thanos
56+
resources:
57+
requests:
58+
cpu: 100m
59+
memory: 50Mi
60+
volumeMounts:
61+
- name: tsdb-data
62+
mountPath: /data
63+
- name: gcs-credentials
64+
mountPath: /creds/
65+
volumes:
66+
- name: config-volume
67+
configMap:
68+
name: prometheus-gcs
69+
- name: tsdb-data
70+
emptyDir: {}
71+
- name: gcs-credentials
72+
secret:
73+
defaultMode: 420
74+
# gcs-credentials secret with single file gcs-credentials.json is required.
75+
secretName: gcs-credentials
76+
terminationGracePeriodSeconds: 300

‎kube/manifests/prometheus-gcs/service.yaml

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
apiVersion: v1
2+
kind: Service
3+
metadata:
4+
labels:
5+
app: prometheus-gcs
6+
name: prometheus-gcs
7+
spec:
8+
externalTrafficPolicy: Cluster
9+
ports:
10+
- port: 9090
11+
protocol: TCP
12+
targetPort: 9090
13+
name: http-prometheus
14+
- port: 19190
15+
protocol: TCP
16+
targetPort: 19190
17+
name: http-sidecar-metrics
18+
selector:
19+
app: prometheus-gcs
20+
sessionAffinity: None
21+
type: NodePort
22+
status:
23+
loadBalancer: {}

‎kube/manifests/prometheus/deployment.yaml

+3-1
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ spec:
1818
args: [
1919
"--config.file=/etc/prometheus/config/prometheus.yaml",
2020
"--storage.tsdb.path=/data",
21+
"--storage.tsdb.min-block-duration=2h",
22+
"--storage.tsdb.max-block-duration=2h",
2123
]
2224
name: prometheus
2325
resources:
@@ -43,7 +45,7 @@ spec:
4345
"--tsdb.path=/data",
4446
"--cluster.address=0.0.0.0:19390",
4547
"--cluster.peers=thanos.default.svc.cluster.local:19390",
46-
# "--gcs.bucket=<bucket>" Add bucket and service account to enable shipping blocks to GCS.
48+
# This an example of running sidecar without shipping data to GCS.
4749
]
4850
name: thanos
4951
resources:

‎kube/manifests/thanos-store/deployment.yaml

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
apiVersion: extensions/v1beta1
2+
kind: Deployment
3+
metadata:
4+
name: thanos-store
5+
labels:
6+
app: thanos-store
7+
thanos-peer: "true"
8+
spec:
9+
replicas: 1
10+
template:
11+
metadata:
12+
labels:
13+
app: thanos-store
14+
thanos-peer: "true"
15+
spec:
16+
containers:
17+
# To use your own thanos image you need to do `eval $(minikube docker-env) && make docker` and place thanos/thanos:latest here.
18+
# TODO(bplotka): With vm-driver=none, even that is uses same docker, images are not recognized. Investigate.
19+
- image: bplotka/thanos:latest
20+
imagePullPolicy: Always
21+
env:
22+
- name: GOOGLE_APPLICATION_CREDENTIALS
23+
value: /creds/gcs-credentials.json
24+
args: [
25+
"store",
26+
"--log.level=debug",
27+
"--debug.name=store",
28+
"--api-address=0.0.0.0:19090", # This address will be properly deduced and propagated via cluster.PeerState.
29+
"--metrics-address=0.0.0.0:19190",
30+
"--tsdb.path=/data",
31+
"--cluster.address=0.0.0.0:19390",
32+
"--cluster.peers=thanos.default.svc.cluster.local:19390",
33+
"--gcs.bucket=thanos-test"
34+
]
35+
name: thanos
36+
resources:
37+
requests:
38+
cpu: 200m
39+
memory: 200Mi
40+
volumeMounts:
41+
- mountPath: /creds/
42+
name: gcs-credentials
43+
readOnly: true
44+
- name: tsdb-data
45+
mountPath: /data
46+
volumes:
47+
- name: gcs-credentials
48+
secret:
49+
defaultMode: 420
50+
# gcs-credentials secret with single file gcs-credentials.json is required.
51+
secretName: gcs-credentials
52+
- name: tsdb-data
53+
emptyDir: {}
54+
terminationGracePeriodSeconds: 300

‎kube/manifests/local/service.yaml ‎kube/manifests/thanos-store/service.yaml

+6-7
Original file line numberDiff line numberDiff line change
@@ -2,18 +2,17 @@ apiVersion: v1
22
kind: Service
33
metadata:
44
labels:
5-
app: prometheus
6-
name: prometheus
5+
app: thanos-store
6+
name: thanos-store
77
spec:
8-
clusterIP: 10.0.0.88
98
externalTrafficPolicy: Cluster
109
ports:
11-
- nodePort: 30600
12-
port: 9090
10+
- port: 19190
1311
protocol: TCP
14-
targetPort: 9090
12+
targetPort: 19190
13+
name: http-store-metrics
1514
selector:
16-
app: prometheus
15+
app: thanos-query
1716
sessionAffinity: None
1817
type: NodePort
1918
status:

‎pkg/cluster/stores.go

+1
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ func (s *StoreSet) Update(ctx context.Context) {
5252
level.Warn(s.logger).Log("msg", "dialing connection failed; skipping", "store", addr, "err", err)
5353
continue
5454
}
55+
level.Debug(s.logger).Log("msg", "successfully made grpc connection", "store", addr)
5556
store := &storeInfo{conn: conn}
5657
s.stores[addr] = store
5758
}

0 commit comments

Comments
 (0)