Home

Welcome to the k8s-ha-nfs wiki!

Here, I'd like to document my journey to container land using only freely available tools.

My aim is building a dedicated Kubernetes cluster that provides highly available NFS storage. The provisioned NFS exports shall be consumed by containerized applications in separate K8s clusters.

Physical hosting platform

• Hardware

1. Three refurbished DELL T1600 workstations with 24GB RAM and two HDD (250GB for OS and 2TB DATA) each, used as KVM hypervisor hosts: kvm-node11, kvm-node12, kvm-node13.
2. One refurbished DELL E5420 laptop with 16GB RAM and 250GB SSD for OS, used for K8s management with Rancher 2.x: kvm-mgmt.
3. One refurbished HP 8000 workstation with 8GB RAM and 500GB HDD, used as infrastructure server, gateway and firewall with OPNsense: kvm-node10.

• Software

1. Operating system: openSUSE Leap 15.0 installed as Transactional Server
2. Virtualization layer: KVM + qemu as supplied by openSUSE 15.0

• Network

1. One 8-port Gbit-switch for intra cluster traffic.
2. One Wifi access point and router, providing Intranet and Internet connections.

KVM hypervisor

• 12 QEMU virtual guests forming 4 Kubernetes clusters:

1. cluster01: 3 nodes running Ubuntu 16.04 (with 2vCPU and 4GB RAM each): k8s-node11, k8s-node21, k8s-node31.
2. cluster02: 3 nodes running RancherOS 1.4.0 (with 2vCPU and 4GB RAM each): k8s-node12, k8s-node22, k8s-node32.
3. cluster03: 3 nodes running RancherOS 1.4.3 (with 2vCPU and 4GB RAM each): k8s-node13, k8s-node23, k8s-node33.
4. cluster04: 3 nodes running Ubuntu 18.04 (with 2vCPU and 4GB RAM each): k8s-node14, k8s-node24, k8s-node34.

Kubernetes distributions

I'm using the k8s distributions supplied either via the Rancher 2.x management GUI or via Rancher's RKE tool.

Initially I started with a standalone installation of Rancher v2.0.6 on the management node rancher20. This has been upgraded recently to Rancher v2.1.6.

1. cluster01: Initial configuration: kubernetes v1.10.5 on top of docker 17.03.2. Current configuration: kubernetes v.1.10.12
2. cluster02: Initial configuration: kubernetes v1.10.5 on top of docker 17.03.2. Current configuration: kubernetes v.1.11.6
3. cluster03: Initial configuration: kubernetes v1.11.6 on top of docker 17.03.2. Current configuration: kubernetes v.1.11.6
4. cluster04: tbd.

NFS servers

cluster01: SDS with CEPH and Rook.io

SDS with Rook.io and CEPH

cluster02: Longhorn and various NFS providers

Longhorn and various NFS providers

NFS clients

cluster03: Proof-of-Concept with User Applications

Proof of Concept with User Applications

cluster04: Infrastructure Applications