Skip to content

Commit

Permalink
feat(cluster-autoscaler): internal helm chart for cluster-autoscaler …
Browse files Browse the repository at this point in the history
…and example (#34)

* feat(cluster-autoscaler): internal helm chart for cluster-autoscaler

* feat(cluster-autoscaler): updated cluster autoscaler example
  • Loading branch information
Young-ook authored Jan 15, 2021
1 parent e0c417d commit 8b9b73a
Show file tree
Hide file tree
Showing 13 changed files with 544 additions and 25 deletions.
30 changes: 27 additions & 3 deletions examples/autoscaling/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,10 @@ The [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application
This example requires a running Kubernetes cluster and kubectl. [Metrics server](https://github.com/kubernetes-sigs/metrics-server) monitoring needs to be deployed in the cluster to provide metrics through the [Metrics API](https://github.com/kubernetes/metrics). Horizontal Pod Autoscaler uses this API to collect metrics. To learn how to deploy the metrics-server, see the metrics-server documentation.

### PHP application
First, we will start a deployment running the image and expose it as a service.
Run the following command:
First, we will start a deployment running the image and expose it as a service. Run the following command:
```
$ kubectl apply -f https://k8s.io/examples/application/php-apache.yaml
```

Here is the details of `php-apache.yaml` file to deploy web application. This manifest creates a simple PHP-based web server and Kubernetes service. It will listen for http requests on port 80.
```
apiVersion: apps/v1
Expand Down Expand Up @@ -105,6 +103,32 @@ php-apache-79544xxxxx-rj9p6 1/1 Running 0 6m27s
php-apache-79544xxxxx-ts5d2 1/1 Running 0 56s
```

## Cluster Autoscaler (CA)
### Before you begin
If you have tested the Horizontal Pod Autoscaler (HPA), you need to reset the configuration of php-apache application. You need to remove and redeploy the php-apache application or you need to adjust the desired capacity of the EC2 autoscaling group to 1. This is important because you want to see that the cluster autoscaling processing is working properly to automatically increase instance capacity when there is no space to launch the reserved container.

### Verify
To check the latest update of cluster autoscaler in the EKS clsuter, please refer to [this](https://github.com/Young-ook/terraform-aws-eks/blob/main/modules/cluster-autoscaler) for more details.

### PHP application
First, we will start a deployment running the nginx container. Run the following command:
```
$ kubectl apply -f https://k8s.io/examples/application/php-apache.yaml
```
### Scale out the application
```
$ kubectl scale --replicas=3 deployment/php-apache
$ kubectl get po -o wide -w
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
php-apache-79544xxxxx-9lbhz 1/1 Running 0 41h 172.31.36.219 ip-172-31-38-165.ap-northeast-2.compute.internal <none> <none>
php-apache-79544xxxxx-r64b2 1/1 Running 0 71s 172.31.32.31 ip-172-31-38-165.ap-northeast-2.compute.internal <none> <none>
php-apache-79544xxxxx-ws6qw 0/1 Pending 0 71s <none> <none> <none> <none>
php-apache-79544xxxxx-ws6qw 0/1 Pending 0 2m13s <none> <none> <none> <none>
php-apache-79544xxxxx-ws6qw 0/1 Pending 0 2m45s <none> ip-172-31-54-84.ap-northeast-2.compute.internal <none> <none>
php-apache-79544xxxxx-ws6qw 0/1 ContainerCreating 0 2m45s <none> ip-172-31-54-84.ap-northeast-2.compute.internal <none> <none>
php-apache-79544xxxxx-ws6qw 1/1 Running 0 2m55s 172.31.60.13 ip-172-31-54-84.ap-northeast-2.compute.internal <none> <none>
```

## Clean up
Run terraform:
```
Expand Down
2 changes: 1 addition & 1 deletion examples/autoscaling/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ module "alb-ingress" {

module "cluster-autoscaler" {
source = "Young-ook/eks/aws//modules/cluster-autoscaler"
enabled = false
enabled = true
cluster_name = module.eks.cluster.name
oidc = module.eks.oidc
tags = { env = "test" }
Expand Down
32 changes: 23 additions & 9 deletions modules/cluster-autoscaler/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@
[Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) is a tool that automatically adjusts the size of a Kubernetes Cluster so that all pods have a place to run and there are no unneeded nodes when one of the following conditions is true:
* there are pods that failed to run in the cluster due to insufficient resources.
* there are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.
On AWS, Cluster Autoscaler utilizes Amazon EC2 Auto Scaling Groups to manage node groups. Cluster Autoscaler typically runs as a `Deployment` in your cluster. For more details, please check out [Cluster Autoscaler on AWS](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md)
On AWS, Cluster Autoscaler utilizes Amazon EC2 Auto Scaling Groups to manage node groups. Cluster Autoscaler typically runs as a `Deployment` in your cluster. For more details, please check out [Cluster Autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html)

## Examples
- [Quickstart Example](https://github.com/Young-ook/terraform-aws-eks/blob/main/modules/cluster-autoscaler/README.md#quickstart)
- [Cluster Autoscaler Exmaple](https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html)
- [Amazon EKS Autoscaling](https://github.com/Young-ook/terraform-aws-eks/blob/main/examples/autoscaling/)
- [EKS Cluster Autoscaler Setup](https://aws.amazon.com/premiumsupport/knowledge-center/eks-cluster-autoscaler-setup/)

## Quickstart
### Setup
Expand All @@ -22,12 +22,12 @@ provider "helm" {
host = module.eks.helmconfig.host
token = module.eks.helmconfig.token
cluster_ca_certificate = base64decode(module.eks.helmconfig.ca)
load_config_file = false
}
}
module "cluster-autoscaler" {
source = "Young-ook/eks/aws//modules/cluster-autoscaler"
enabled = true
cluster_name = module.eks.cluster.name
oidc = module.eks.oidc
tags = { env = "test" }
Expand All @@ -40,20 +40,34 @@ terraform apply
```

### Verify
#### Check container status
All steps are finished, check that there are pods that are `Ready` in `kube-system` namespace:
Ensure the `eks-as-aws-cluster-autoscaler-chart` pod is generated and running:

```
$ kubectl -n kube-system get po
NAME READY STATUS RESTARTS AGE
aws-node-g4mh5 1/1 Running 0 10m
coredns-7dd7f84d9-bb9mq 1/1 Running 0 10m
coredns-7dd7f84d9-xhpd4 1/1 Running 0 10m
eks-alb-aws-alb-ingress-controller-6f68cb8df5-zjgxn 1/1 Running 0 10m
eks-as-aws-cluster-autoscaler-chart-59d48879c4-mwjzk 1/1 Running 0 10m
cluster-autoscaler-xxxxxxxxx-mwjzk 1/1 Running 0 10m
kube-proxy-q79tk 1/1 Running 0 10m
```
If the pod is not healthy, please try to check the log:
```
$ kubectl -n kube-system logs eks-as-aws-cluster-autoscaler-chart-59d48879c4-mwjzk
$ kubectl -n kube-system logs cluster-autoscaler-xxxxxxxxx-mwjzk
```

#### Check configmap status
If users make sure that the EC2 autoscaling group(ASG) has the tags that cluster autoscaler is looking for, the users can see the latest update of cluster autoscaler. It may look like below and the users can check whether cluster autoscaler is able to recognize the ASG.
```
$ kubectl -n kube-system get cm cluster-autoscaler-status -o yaml
apiVersion: v1
data:
status: |+
Cluster-autoscaler status at 2021-01-08 04:04:55.644106199 +0000 UTC:
NodeGroups:
Name: eks-xxxxyyyy-c03a-xxxx-1111-2dc09d308552
Health: Healthy (ready=2 unready=0 notStarted=0 longNotStarted=0 registered=2 longUnregistered=0 cloudProviderTarget=2 (minSize=1, maxSize=3))
LastProbeTime: 2021-01-08 04:04:55.643676127 +0000 UTC m=+6684.061091143
LastTransitionTime: 2021-01-08 02:14:17.530198588 +0000 UTC m=+45.947613652
```
If cluster autoscaler is able to recognize the ASG, the users should see the ASG name under NodeGroups section. If you don't even see NodeGroups section, it means that cluster autoscaler is still not able to autodiscover your ASG.
23 changes: 23 additions & 0 deletions modules/cluster-autoscaler/charts/cluster-autoscaler/.helmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
23 changes: 23 additions & 0 deletions modules/cluster-autoscaler/charts/cluster-autoscaler/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
apiVersion: v2
name: cluster-autoscaler
description: A Helm chart for Kubernetes

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
appVersion: "v1.17.3"
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "cluster-autoscaler.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "cluster-autoscaler.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "cluster-autoscaler.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "cluster-autoscaler.labels" -}}
helm.sh/chart: {{ include "cluster-autoscaler.chart" . }}
{{ include "cluster-autoscaler.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "cluster-autoscaler.selectorLabels" -}}
app.kubernetes.io/name: {{ include "cluster-autoscaler.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

{{/*
Create the name of the service account to use
*/}}
{{- define "cluster-autoscaler.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "cluster-autoscaler.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
{{- if .Values.serviceAccount.create -}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "cluster-autoscaler.fullname" . }}
labels:
{{- include "cluster-autoscaler.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "cluster-autoscaler.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "cluster-autoscaler.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "cluster-autoscaler.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: cluster-autoscaler
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
command:
- ./cluster-autoscaler
- --cloud-provider={{ .Values.cloudProvider }}
- --namespace={{ .Release.Namespace }}
- --skip-nodes-with-local-storage=false
- --expander=least-waste
{{- if .Values.autoscalingGroups }}
{{- range .Values.autoscalingGroups }}
- --nodes={{ .minSize }}:{{ .maxSize }}:{{ .name }}
{{- end }}
{{- end }}
{{- if eq .Values.cloudProvider "aws" }}
{{- if .Values.autoDiscovery.clusterName }}
- --node-group-auto-discovery=asg:tag={{ tpl (join "," .Values.autoDiscovery.tags) . }}
{{- end }}
{{- range $key, $value := .Values.extraArgs }}
- --{{ $key }}={{ $value }}
{{- end }}
{{- end }}
env:
{{- if and (eq .Values.cloudProvider "aws") (ne .Values.awsRegion "") }}
- name: AWS_REGION
value: "{{ .Values.awsRegion }}"
{{- if .Values.awsAccessKeyID }}
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: AwsAccessKeyId
name: {{ template "cluster-autoscaler.fullname" . }}
{{- end }}
{{- if .Values.awsSecretAccessKey }}
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: AwsSecretAccessKey
name: {{ template "cluster-autoscaler.fullname" . }}
{{- end }}
{{- end }}
{{- if .Values.extraVolumeMounts }}
volumeMounts:
{{ toYaml .Values.extraVolumeMounts | nindent 12 }}
{{- end }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- if .Values.extraVolumes }}
volumes:
{{- toYaml .Values.extraVolumes | nindent 10 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
{{- if .Values.ingress.enabled -}}
{{- $fullName := include "cluster-autoscaler.fullname" . -}}
{{- $svcPort := .Values.service.port -}}
{{- if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
name: {{ $fullName }}
labels:
{{- include "cluster-autoscaler.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
- hosts:
{{- range .hosts }}
- {{ . | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ . }}
backend:
serviceName: {{ $fullName }}
servicePort: {{ $svcPort }}
{{- end }}
{{- end }}
{{- end }}
Loading

0 comments on commit 8b9b73a

Please sign in to comment.