binder.intel4coro.de-deploy

This repo is for maintaining the K8S configuration of the site binder.intel4coro.de.

Configure file	Descriptions
binder.yaml	Main config file
commands.md	Some commonly used K8S commands.
binder_yaml.py	A Python script to insert network policy configs on top of binder.yaml.
requirements.txt	Dependencies of script binder_yaml.py.
resource-quota.yaml	Configuration of resource limitations.

For the complete deployment guide, see platform deployment.

Common operations

Deploy BinderHub

• Create file secret.yaml with DockerHub authentication:

  registry:
    username: <DockerHub user>
    password: <Token>

• Run the following command to apply updates:

Note: The script binder_yaml.py inserts configs of the network policy and whitelist of GitHub repos, and generates a new file _binder.yaml which is the actual config file applied to Kubernetes.

python3 binder_yaml.py && \
microk8s.helm upgrade --cleanup-on-fail \
  --install binder \
  jupyterhub/binderhub --version=1.0.0-0.dev.git.3170.h84b1db9 \
  --namespace=binder \
  --create-namespace \
  -f ./secret.yaml \
  -f ./_binder.yaml

Update Binderhub

To upgrade the binderhub version:

1. Check the latest version at https://hub.jupyter.org/helm-chart/#development-releases-binderhub.
2. Run helm repo update
3. Replace the version 1.0.0-0.dev.git.3506.hba24eb2a with a newer one.

python3 binder_yaml.py && \
helm upgrade binder --cleanup-on-fail \
  jupyterhub/binderhub --version=1.0.0-0.dev.git.3506.hba24eb2a \
  --namespace=binder \
  -f ./secret.yaml \
  -f ./_binder.yaml

Kubernetes Dashboard

https://dashboard.intel4coro.de/#/pod?namespace=binder

Generate new dashboard login tokens:

kubectl create token default --duration=2000h

Delete a batch of pods

Delete pods with the keyword "jupyter-" (user pods have the prefix "jupyter-"):

microk8s.kubectl get pods -n binder --no-headers=true | awk '/jupyter-/{print $1}'| xargs microk8s.kubectl delete pod -n binder

Show resource limits

kubectl get resourcequota -n binder

Update resource limits

kubectl apply -f ./resource-quota.yaml -n binder

Reserve Compute Resources for System Daemons

It is important to reserve compute resources for the OS process. If we do not properly configure the pod resources, it will lead to resource starvation issues on the node. Modify the kubelet config file /var/snap/microk8s/current/args/kubelet, add new lines like:

• --kube-reserved=cpu=1,memory=512Mi,ephemeral-storage=10Gi
• --system-reserved=cpu=1,memory=512Mi,ephemeral-storage=10Gi

Update Repos whitelist

If config.GitHubRepoProvider.whitelist_enabled is enabled, only repos from those GitHub users are allowed. Edit the list config.GitHubRepoProvider.whitelist in binder.yaml.

Network policy of user's pod

The user's pod cannot directly connect to public IPs; this is to prevent users from downloading arbitrary content from the Internet or mining cryptocurrency.

We added the IP ranges of pip.org, github.com, and some specific services to the egress rules. Users can still clone GitHub repositories and install Python packages with pip install. Such config is generated by the script binder_yaml.py, it will only be updated when updating deployment. Therefore, if the services' DNS records changed, the old network policy might expired.

To enable connections to other external web services, add new egress rules under the field jupyterhub.singleuser.networkPolicy.egress.

Add Node to cluster

Source: https://microk8s.io/docs/clustering

Note: A cluster with nodes not inside the same LAN is not so stable.

microk8s add-node

get the output like:

From the node you wish to join to this cluster, run the following:
microk8s join 192.168.1.230:25000/92b2db237428470dc4fcfc4ebbd9dc81/2c0cb3284b05

Use the '--worker' flag to join a node as a worker not running the control plane, eg:
microk8s join 192.168.1.230:25000/92b2db237428470dc4fcfc4ebbd9dc81/2c0cb3284b05 --worker

If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 192.168.1.230:25000/92b2db237428470dc4fcfc4ebbd9dc81/2c0cb3284b05
microk8s join 10.23.209.1:25000/92b2db237428470dc4fcfc4ebbd9dc81/2c0cb3284b05
microk8s join 172.17.0.1:25000/92b2db237428470dc4fcfc4ebbd9dc81/2c0cb3284b05

Choose the current network interface and run it on the worker node.

Check node status with the command or on the Dashboard.

microk8s kubectl get no

After the node is added, better to redeploy the BinderHub services.

Note: The steps above assume the firewall is properly configured. See details of the ports at: https://microk8s.io/docs/services-and-ports

Troubleshooting

Setting up a cluster with nodes not under the same local network might have issues where the clusterIPs are not accessible between nodes. I haven't figured out the exact reason; my guess is the coreDNS is not properly configured. My solution:

1. On Master node, allow all connections from the worker node.

   sudo ufw allow from <worker-node-ip>
   sudo ufw allow to <worker-node-ip>
   

2. Install the same version of Microk8s on both nodes (tested with v1.31.1)

3. Join the cluster before enabling any Kubernetes Addons (especially MetalLB).

4. Enable addons and deploy BinderHub.

Remove Node from cluster

On Worker Node run:

microk8s leave

On Master Node get the node name:

microk8s kubectl get no

Remove the worker node:

microk8s remove-node u24worker

Manage built images

Microk8s uses a separate containerd instance. Therefore, you cannot see the k8s images in the Docker engine.

List all images

microk8s ctr images ls

Delete images with 'teaching' in the name

microk8s ctr images rm $(microk8s ctr images ls name~='teaching' | awk {'print $1'})