Skip to content

Cluster Setup

Jump to bottom
benoit74 edited this page Nov 1, 2023 · 9 revisions

Instructions on how-to/how-was setup the k8s Kosmos Cluster at Scaleway, using webUI

  • Create Cluster without pool (those are scaleway)
  • Enable Ingress controller (nginx) via UI or API
  • Download kubeconfig and store as ~/.kube/scw-admin_kiwix-prod.config
  • Add Pool (foreign)
  • Add a node. It will be your first node (it's special but not CP).
  • set CNAME scw.k8s to 58cd205e-743d-4cf2-bf23-db3c7297f675.nodes.k8s.fr-par.scw.cloud.
  • set CNAME api.scw.k8s to 58cd205e-743d-4cf2-bf23-db3c7297f675.api.k8s.fr-par.scw.cloud.
  • why?
    • scw.k8s has one entry per node so requests are are redirected to any of the nodes
    • api.scw.k8s is control plane. should not change but is referenced in config files, so easier maintenance with clear name.
    • k8s CNAME to scw.k8s. That's where we point all domains.
    • allows us to setup a parallel xyz.k8s node or even a different CP (api.xyz.k8s) and test it independently then instantly switch everything by pointing k8s to xyz.k8s
  • Add api.scw.k8s to ControlPlane SAN
  • you'll need some environment variables:
# get those following https://developers.scaleway.com/en/quickstart/
export SCW_ACCESS_KEY=xxx
export SCW_SECRET_KEY=yyy
export KIWIX_PROD_CLUSTER=c7704f7d-c6e9-49b3-9aaf-a04aa4d5a727
export KIWIX_PROD_FOREIGN_POOL=4a574aa5-737e-4993-961a-1a8d629ee4ea
curl -X PATCH -d '{"apiserver_cert_sans": ["api.scw.k8s.kiwix.org"]}' -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$CLUSTER_ID

Our Control Plane is now at https://api.scw.k8s.kiwix.org:6443

  • Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.2/cert-manager.yaml
# verify its working
kubectl get pods --namespace cert-manager

  • Follow rest of HTTP Steup steps (ClusterIssuer)

  • Create superuser

./create-superuser reg

Upgrading Kubernetes

  • Find out version to upgrade to. ⚠️ downgrade impossible.
http https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER/available-versions "X-Auth-Token: $SCW_SECRET_KEY"
  • Upgrade the cluster and node-pools using UI or API
curl -X POST -d '{"version": "1.23.4", "upgrade_pools": true}' -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER/upgrade
  • Verify version and wait until status is ready (after updating)
http https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER "X-Auth-Token: $SCW_SECRET_KEY"

  • Upgrade nodes (scaleway-managed pools may not require this)

  • disable schedule on node: kubectl cordon $NODE_NAME

  • [inside node – ssh] ./upgrade-node.sh $NEW_VERSION

  • re-enable schedule: kubectl uncordon $NODE_NAME

  • Upgrade kube-state-metrics image

kube-state-metrics version has to be aligned with k8s version ; there as some upward and downward compatibility, and it is expected that a recent kube-state-metrics will more nicely cooperate with an old k8s cluster than the other way around.

Find the appropriate version for current k8s API version: https://github.com/kubernetes/kube-state-metrics#compatibility-matrix

Update the version in https://github.com/kiwix/k8s/blob/main/grafana/grafana.values.yaml and upgrade the release with helm upgrade (see detailed instructions in https://github.com/kiwix/k8s/blob/main/grafana/README.md#installation--upgrade)

Clone this wiki locally