Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Added Simple Scalable Deployment guide for AWS #14327

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
d8dd1c7
added SSD guide for AWS
Jayclifford345 Oct 1, 2024
fccc7dc
Merge branch 'main' into aws-helm
Jayclifford345 Oct 1, 2024
3305c96
fixed title
Jayclifford345 Oct 1, 2024
ff2b1db
fixed trailing slashes
Jayclifford345 Oct 1, 2024
933ee34
updated microservice ref
Jayclifford345 Oct 1, 2024
edf4605
updated index
Jayclifford345 Oct 1, 2024
7886f0d
Update docs/sources/setup/install/helm/install-microservices/_index.md
Jayclifford345 Oct 1, 2024
c7cd341
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 1, 2024
bcbc2af
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 1, 2024
a94257c
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 1, 2024
119676b
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 1, 2024
af08bcc
Merge branch 'main' into aws-helm
Jayclifford345 Oct 1, 2024
108a3f0
Update docs/sources/setup/install/helm/install-scalable/aws.md
Jayclifford345 Oct 2, 2024
f77d2d7
added recommended changes
Jayclifford345 Oct 3, 2024
6b72e1e
Merge branch 'aws-helm' of https://github.com/grafana/loki into aws-helm
Jayclifford345 Oct 3, 2024
7103f1c
fixed formatting
Jayclifford345 Oct 3, 2024
8225c4c
fixed namespace
Jayclifford345 Oct 3, 2024
271ef6d
added suggestion around seperating chunk and ruler bucket
Jayclifford345 Oct 7, 2024
d9767d2
Added suggestions by trevor
Jayclifford345 Oct 8, 2024
ae277ce
Merge branch 'main' into aws-helm
Jayclifford345 Oct 8, 2024
f553894
Update docs/sources/setup/install/helm/install-microservices/_index.md
Jayclifford345 Oct 8, 2024
dd57de1
added Poyzans recommendations
Jayclifford345 Oct 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/sources/setup/install/helm/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,17 @@ This guide references the Loki Helm chart version 3.0 or greater and contains th

If you are installing Grafana Enterprise Logs, follow the [GEL Helm installation](https://grafana.com/docs/enterprise-logs/<ENTERPRISE_LOGS_VERSION>/setup/helm/).

## Recommended Installation

The recommended installation method for initial deployments is to use the [Loki Simple Scalable Helm chart]({{< relref "./install-scalable" >}}). This chart provides a simple scalable deployment mode for Loki, separating execution paths into read, write, and backend targets. For small to medium-sized deployments, this chart is a good starting point.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great specific opinionated starting point. I'm not wondering what you think I should do.


### Cloud Deployment Guides

The following guides provide step-by-step instructions for deploying Loki on cloud providers:

- [Deploy Loki Simple Scalable Helm chart on AWS]({{< relref "./install-scalable/aws" >}})


## Reference

[Values reference]({{< relref "./reference" >}})
156 changes: 79 additions & 77 deletions docs/sources/setup/install/helm/install-microservices/_index.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
---
title: Install the microservice Helm chart
title: Loki Microservice Helm chart
menuTitle: Install microservice Loki
description: Installing Loki in microservice (distributed) mode using the Helm chart.
weight: 300
keywords:
---

# Install the microservice Helm chart
# Loki Microservice Helm chart

This Helm Chart deploys Grafana Loki on Kubernetes.

Expand All @@ -21,16 +21,17 @@ The default Helm chart deploys the following components:
- **QueryFrontend component** (2 replicas, maxUnavailable: 1): Manages frontend queries. Up to 1 replica can be unavailable during updates.
- **QueryScheduler component** (2 replicas): Schedules queries.

{{< admonition type="note" >}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't necessarily require a change, but I'm curious why this is the right spot to make this point.

I understand it's important and why it's here, but an alternative could be to guide me down the path of the opinionated right way of doing things, and simply provide a summary block somewhere else saying "we chose to embed the following best practices in this approach"

It is not recommended to run scalable mode with `filesystem` storage. For the purpose of this guide, we will use MinIO as the object storage to provide a complete example.
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved
{{< /admonition >}}

**Prerequisites**
## Prerequisites

- Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/).
- A running Kubernetes cluster.
- (Optional) A Memcached deployment for better query performance. For information on configuring Memcached, refer to the [caching section](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/caching/).


**To deploy Loki in microservice mode (with MinIO):**
## Deploying the Helm chart for development and testing


1. Add [Grafana's chart repository](https://github.com/grafana/helm-charts) to Helm:
Expand All @@ -49,31 +50,30 @@ It is not recommended to run scalable mode with `filesystem` storage. For the pu

```yaml
loki:
schemaConfig:
configs:
- from: 2024-04-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
ingester:
chunk_encoding: snappy
tracing:
enabled: true
querier:
# Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
max_concurrent: 4

#gateway:
# ingress:
# enabled: true
# hosts:
# - host: FIXME
# paths:
# - path: /
# pathType: Prefix
schemaConfig:
configs:
- from: 2024-04-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
ingester:
chunk_encoding: snappy
querier:
# Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
max_concurrent: 4
pattern_ingester:
enabled: true
limits_config:
allow_structured_metadata: true
volume_enabled: true
retention_period: 672h
compactor:
retention_enabled: true
delete_request_store: s3


deploymentMode: Distributed

Expand Down Expand Up @@ -101,20 +101,15 @@ It is not recommended to run scalable mode with `filesystem` storage. For the pu
bloomGateway:
replicas: 0

# This exposes the Loki gateway so it can be written to and queried externaly
gateway:
service:
type: LoadBalancer


# Enable minio for storage
minio:
enabled: true

Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved
# Zero out replica counts of other deployment modes
backend:
replicas: 0
read:
replicas: 0
write:
replicas: 0

singleBinary:
replicas: 0
```

4. Install or upgrade the Loki deployment.
Expand All @@ -138,7 +133,7 @@ It is not recommended to run scalable mode with `filesystem` storage. For the pu
loki-canary-8thrx 1/1 Running 0 167m
loki-canary-h965l 1/1 Running 0 167m
loki-canary-th8kb 1/1 Running 0 167m
loki-chunks-cache-0 0/2 Pending 0 167m
loki-chunks-cache-0 2/2 Running 0 167m
loki-compactor-0 1/1 Running 0 167m
loki-compactor-1 1/1 Running 0 167m
loki-distributor-7c9bb8f4dd-bcwc5 1/1 Running 0 167m
Expand All @@ -165,7 +160,7 @@ It is not recommended to run scalable mode with `filesystem` storage. For the pu

## Object Storage Configuration

After testing Loki with MinIO, it is recommended to configure Loki with an object storage provider. The following examples shows how to configure Loki with different object storage providers:
After testing Loki with MinIO, we recommended to configure Loki with an object storage provider. The following examples shows how to configure Loki with different object storage providers:
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved

{{< admonition type="caution" >}}
When deploying Loki using S3 Storage **DO NOT** use the default bucket names; `chunk`, `ruler` and `admin`. Choose a unique name for each bucket. For more information see the following [security update](https://grafana.com/blog/2024/06/27/grafana-security-update-grafana-loki-and-unintended-data-write-attempts-to-amazon-s3-buckets/). This caution does not apply when you are using MinIO. When using MinIO we recommend using the default bucket names.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

outside the scope of this PR so ignore me but we probably shouldn't ever have this in the docs. The helm chart should ideally block you from foot-gunning in this particular way

Expand All @@ -176,48 +171,55 @@ When deploying Loki using S3 Storage **DO NOT** use the default bucket names; `
```s3
# Example configuration for Loki with S3 storage

loki:
schemaConfig:
configs:
- from: 2024-04-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
ingester:
chunk_encoding: snappy
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved
tracing:
loki:
schemaConfig:
configs:
- from: 2024-04-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
storage_config:
aws:
region: eu-west-2
Jayclifford345 marked this conversation as resolved.
Show resolved Hide resolved
bucketnames: loki-aws-bucket # Define your AWS bucket here
s3forcepathstyle: false
pattern_ingester:
enabled: true
querier:
limits_config:
allow_structured_metadata: true
volume_enabled: true
retention_period: 672h # 28 days retention
querier:
max_concurrent: 4

storage:
storage:
type: s3
bucketNames:
chunks: "<INSERT BUCKET NAME>"
ruler: "<INSERT BUCKET NAME>"
admin: "<INSERT BUCKET NAME>"
s3:
# s3 URL can be used to specify the endpoint, access key, secret key, and bucket name
s3: s3://access_key:secret_access_key@custom_endpoint/bucket_name
# AWS endpoint URL
endpoint: <your-endpoint>
# AWS region where the S3 bucket is located
region: <your-region>
# AWS secret access key
secretAccessKey: <your-secret-access-key>
# AWS access key ID
accessKeyId: <your-access-key-id>
# AWS signature version (e.g., v2 or v4)
signatureVersion: <your-signature-version>
# Forces the path style for S3 (true/false)
s3ForcePathStyle: false
# Allows insecure (HTTP) connections (true/false)
insecure: false
# HTTP configuration settings
http_config: {}
s3:
# s3 URL can be used to specify the endpoint, access key, secret key, and bucket name this works well for S3 compatible storages or are hosting Loki on-premises and want to use S3 as the storage backend. Either use the s3 URL or the individual fields below (AWS endpoint, region, secret).
s3: s3://access_key:secret_access_key@custom_endpoint/bucket_name
# AWS endpoint URL
endpoint: <your-endpoint>
# AWS region where the S3 bucket is located
region: <your-region>
# AWS secret access key
secretAccessKey: <your-secret-access-key>
# AWS access key ID
accessKeyId: <your-access-key-id>
# AWS signature version (e.g., v2 or v4)
signatureVersion: <your-signature-version>
# Forces the path style for S3 (true/false)
s3ForcePathStyle: false
# Allows insecure (HTTP) connections (true/false)
insecure: false
# HTTP configuration settings
http_config: {}

deploymentMode: Distributed

Expand Down
Loading
Loading