k8s-example

Setup Kubernetes cluster from scratch

1. Spinning Up Kubernetes Cluster

Prerequisites

• An SSH key pair on your local machine¹
• Servers running CentOS 7 with at least 2GB RAM and 2 vCPUs each and you should be able to SSH into each server as the root user with your SSH key pair²
• Ansible installed on your local machine^3,4
• Extarnal load balancer is provisioned
• Domain name is provisioned (i.e. astakhoff.ru)
• Common names are assigned to the public IP (external load balancer):
  • k8s.astakhoff.ru
  • grafana.k8s.astakhoff.ru
  • prometheus.k8s.astakhoff.ru
  • store.k8s.astakhoff.ru

Setting Up the Workspace Directory and Ansible

• Setup a ./ansible/hosts.ini file containing inventory information such as the IP addresses of your servers and the groups that each server belongs to.
• Define common names in ./ansible/vars/cnames.yml
• (Optional) Define Ingress NodePorts in ./ansible/vars/main.yml⁵

Installing Kubernetes cluster

• Execute the playbook:

  ansible-galaxy install kwoodson.yedit
  ansible-playbook -i hosts.ini main.yml

Checking Kubernetes cluster

kubectl get no -o wide

kubectl get po -A -o wide

Checking Ingress controller

kubectl get po -n ingress-nginx -o wide

kubectl get svc -n ingress-nginx

curl http://64.227.132.241:30080

curl http://k8s.astakhoff.ru

2. Deploying Services to Kubernetes Cluster

Install cert-manager⁶

• Execute the playbook:

  ansible-playbook -i hosts.ini cert-manager.yml

• Verify the installation⁷

  kubectl get pods --namespace cert-manager

  

Deploy the sample app to the cluster

• Deploy "Online Boutique" demo application⁸:

  kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/main/release/kubernetes-manifests.yaml

• Wait for the Pods to be ready:

  watch kubectl get pods -o wide

  
• Deploy ingress resource for frontend service:

  ansible-playbook -i hosts.ini frontend-ingress-resource.yml

• Access the web frontend using public IP of the worker node and ingress controller NodePort port:

  curl -kI -H 'Host: store.k8s.astakhoff.ru' https://64.227.136.238:30443

  
• Access the web frontend in a browser using external LoadBalancer's public IP:

Deploy loadgenerator service using Helm via CI/CD

• Update existing loadgenerator instance in order to control deployment via helm:

  kubectl -n default label deployment loadgenerator "app.kubernetes.io/managed-by=Helm"
  kubectl -n default annotate deployment loadgenerator "meta.helm.sh/release-name=loadgenerator" "meta.helm.sh/release-namespace=default"

• Prepare kube config as kube config secret for CI/CD tool:

  cat $HOME/.kube/config | base64

  

• Push a commit into master branch of loadgenerator's source code or prepared Helm chart:

  • https://github.com/viastakhov/k8s-example/tree/main/src/loadgenerator
  • https://github.com/viastakhov/k8s-example/tree/main/helm

• Verify GitHub Actions workflow runs:

Note: GitHub Actions workflow configuration here: https://github.com/viastakhov/k8s-example/blob/main/.github/workflows/ci-cd.yaml

3. Monitoring Setup

Install OpenEBS local PV device storage engine⁹

• Execute the playbook:

  ansible-playbook -i hosts.ini openebs.yml

• Verify installation:
  • Verify pods:

    kubectl get pods -n openebs

    
  • Verify StorageClasses:

    kubectl get sc

    

Install Prometheus stack¹⁰

• (Optional) Setup prometheus stack in ./ansible/vars/prom-stack.yml

• Execute the playbook:

  ansible-playbook -i hosts.ini prometheus.yml

• Verify installation:

  • Verify Prometheus related pods are installed under monitoring namespace:

    kubectl get pod -n monitoring

    
  • Verify Prometheus related PVCs are created under monitoring namespace:

    kubectl get pvc -n monitoring

    
  • Verify Prometheus related services created under monitoring namespace:

    kubectl get svc -n monitoring

    

• Import Grafana dashboards from /dashboard folder

• There are following metrics being used:

  • Pod Resource Usage by Namespace:

    Metriс	Purpose
    CPU Usage	Detect CPU bottlenecks, setup CPU resource requests/limits and VerticalPodAutoscaler/HorizontalPodAutoscaler
    Memory Usage	Detect high memory presure and leakage, setup memory resource requests/limits and VerticalPodAutoscaler/HorizontalPodAutoscaler

    

  • Nodes Resourse Usage:

    Metriс	Purpose
    CPU Usage	Control memory usage on the node, add additional CPU cores on demand
    Load Average	Control system load on the node, add additional CPU cores on demand
    Memory Usage	Control memory utilization on the node, provide additional CPU cores on demand
    Disk I/O	Detect disk I/O bottleneck
    Disk Space Usage	Control disk space utilization
    Network Received	Inspect inbound network traffic, monitors for reception of data that exceeds the bandwidth of the network interface
    Network Transmitted	Inspect outbound network traffic on the interface

    

  • Persistent Volume Usage:

    Metriс	Purpose
    Volume Space Usage	Control volume space utilization by PVC
    Volume Inode Usage	Control the number of inodes available on volume by PVC

    

Note: Grafana URL: https://grafana.k8s.astakhoff.ru (login/password: guest/guest)

4. Logging Setup

Install Loki stack

• Execute the playbook:

  ansible-playbook -i hosts.ini loki.yml

• Wait for the Pods to be ready:

  kubectl get po -n monitoring -o wide --selector release=loki

  

Pod logs inspection

Open "Loki Logs" dashboard in Grafana in order to review pod logs:

Note: Grafana URL: https://grafana.k8s.astakhoff.ru (login/password: guest/guest)

5. Pod Autoscaling

Install metrics-server¹¹

• Execute the playbook:

  ansible-playbook -i hosts.ini metrics-server.yml

• Check whether the metrics-server is available and running

  kubectl get apiservices | grep metrics.k8s.io

  

Setup pod autoscalling

• Create HorizontalPodAutoscaler resource for frontend service:

  kubectl apply -f https://raw.githubusercontent.com/viastakhov/k8s-example/main/manifests/frontend-hpa.yaml

• Verify created resource:

  kubectl get hpa

  

• Increase workload on frontend service:

  kubectl set env deployment/loadgenerator USERS=500

• After a fiew minutes there are additional frontend pods created:

  kubectl get pod --selector app=frontend
  

  

• Decrease workload on frontend service:

  kubectl set env deployment/loadgenerator USERS=1

• Watch for the terminating of several frontend pods:

  kubectl get pod --selector app=frontend -w
  

  

  kubectl get pod --selector app=frontend
  

  

6. Infrastructure Development Plan

IDP

---