Skip to content

Devantler's Homelab - including deployment artifacts for running the Homelab in CI/CD and in Talos Omni.

Notifications You must be signed in to change notification settings

devantler/homelab

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Welcome to Devantler's Homelab πŸš€

Screenshot 2024-09-03 at 00 51 44

This repo contains the deployment artifacts for Devantler's Homelab. The Homelab is a Kubernetes cluster that is highly automated with the use of Flux GitOps, CI/CD with Automated Testing, and much more. Feel free to look around. You might find some inspiration πŸ™ŒπŸ»

Show/hide folder structure
.
β”œβ”€β”€ .github
β”‚Β Β  └── workflows
β”œβ”€β”€ .vscode
β”œβ”€β”€ docs
β”‚Β Β  └── images
β”œβ”€β”€ k8s
β”‚Β Β  β”œβ”€β”€ clusters
β”‚Β Β  β”‚Β Β  β”œβ”€β”€ homelab-local
β”‚Β Β  β”‚Β Β  β”‚Β Β  β”œβ”€β”€ apps
β”‚Β Β  β”‚Β Β  β”‚Β Β  β”œβ”€β”€ flux-system
β”‚Β Β  β”‚Β Β  β”‚Β Β  β”œβ”€β”€ infrastructure
β”‚Β Β  β”‚Β Β  β”‚Β Β  β”‚Β Β  └── controllers
β”‚Β Β  β”‚Β Β  β”‚Β Β  └── variables
β”‚Β Β  β”‚Β Β  └── homelab-prod
β”‚Β Β  β”‚Β Β      β”œβ”€β”€ apps
β”‚Β Β  β”‚Β Β      β”œβ”€β”€ flux-system
β”‚Β Β  β”‚Β Β      β”œβ”€β”€ infrastructure
β”‚Β Β  β”‚Β Β      β”‚Β Β  β”œβ”€β”€ controllers
β”‚Β Β  β”‚Β Β      β”‚Β Β  └── gha-runner-scale-sets
β”‚Β Β  β”‚Β Β      └── variables
β”‚Β Β  β”œβ”€β”€ components
β”‚Β Β  β”‚Β Β  β”œβ”€β”€ flux-kustomization-post-build-variables-label
β”‚Β Β  β”‚Β Β  β”œβ”€β”€ flux-kustomization-sops-label
β”‚Β Β  β”‚Β Β  β”œβ”€β”€ helm-release-crds-label
β”‚Β Β  β”‚Β Β  β”œβ”€β”€ helm-release-remediation-label
β”‚Β Β  β”‚Β Β  └── network-policy-default-deny
β”‚Β Β  β”œβ”€β”€ distributions
β”‚Β Β  β”‚Β Β  β”œβ”€β”€ k3s
β”‚Β Β  β”‚Β Β  β”‚Β Β  β”œβ”€β”€ apps
β”‚Β Β  β”‚Β Β  β”‚Β Β  β”œβ”€β”€ infrastructure
β”‚Β Β  β”‚Β Β  β”‚Β Β  β”‚Β Β  └── controllers
β”‚Β Β  β”‚Β Β  β”‚Β Β  └── variables
β”‚Β Β  β”‚Β Β  └── talos
β”‚Β Β  β”‚Β Β      β”œβ”€β”€ apps
β”‚Β Β  β”‚Β Β      β”œβ”€β”€ infrastructure
β”‚Β Β  β”‚Β Β      β”‚Β Β  └── controllers
β”‚Β Β  β”‚Β Β      β”‚Β Β      β”œβ”€β”€ cilium
β”‚Β Β  β”‚Β Β      β”‚Β Β      β”œβ”€β”€ kubelet-serving-cert-approver
β”‚Β Β  β”‚Β Β      β”‚Β Β      └── longhorn
β”‚Β Β  β”‚Β Β      └── variables
β”‚Β Β  └── shared
β”‚Β Β      β”œβ”€β”€ apps
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ fleetdm
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ headlamp
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ homepage
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ open-webui
β”‚Β Β      β”‚Β Β  └── plantuml
β”‚Β Β      β”œβ”€β”€ infrastructure
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ cloudflared
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ controllers
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ capi-operator
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ cert-manager
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ gha-runner-scale-set-controller
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ goldilocks
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ k8sgpt-operator
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ kyverno
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ metrics-server
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ reloader
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ testkube
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”‚Β Β  └── crds
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ traefik
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── trivy-operator
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ dex
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ harbor
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ helm-charts-oci-proxy
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ kube-prometheus-stack
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ middlewares
β”‚Β Β      β”‚Β Β  β”‚Β Β  β”œβ”€β”€ basic-auth
β”‚Β Β      β”‚Β Β  β”‚Β Β  └── forward-auth
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ oauth2-proxy
β”‚Β Β      β”‚Β Β  β”œβ”€β”€ ollama
β”‚Β Β      β”‚Β Β  └── selfsigned-cluster-issuer
β”‚Β Β      β”œβ”€β”€ tenants
β”‚Β Β      └── variables
└── talos
    β”œβ”€β”€ hetzner
    └── patches
        β”œβ”€β”€ cluster
        └── nodes

79 directories

Prerequisites

For development:

For production:

  • A Talos Cluster

Note

You can use other distributions as well, but the configuration is optimized for Talos, and thus it is not guaranteed to work with other distributions.

Usage

To run this cluster locally, simply run the following command:

ksail up homelab-local

Note

To run this cluster on your metal, would require that you have access to my SOPS keys. This is ofcourse not possible, so you would need to create your own keys and replace the existing ones, if you want to run my cluster configuration on your own metal.

  • The keys that KSail uses are stored in ~/.ksail/age where one Age key is store for each cluster, and named according to the cluster name. For example ~/.ksail/age/homelab-local.
  • To update SOPS to work with Ksail, you need to update the .sops.yaml file in the root of the repository, and replace the age keys with your own keys.
  • To update the manifests to work with KSail, you need to replace all .sops.yaml files with new ones, that are encrypted with your own keys.

For the production cluster, you would need to do the same, but in addition to storing the keys in ~/.ksail/age, you would also need to store the keys in GitHub Secrets, such that the CI/CD pipeline can provision the keys to the cluster.

Stack

The cluster uses Flux GitOps to reconcile the state of the cluster with single source of truth stored in this repository and published as an OCI image. For development, the cluster is spun up by KSail and for production, the cluster is provisioned by Talos Omni.

The cluster configuration is stored in the k8s/* directories where the structure is as follows:

  • clusters/: Contains the the cluster specific configuration for each environment.
  • components/: Contains the reusable components that are used across the cluster.
  • distributions/: Contains the distribution specific configuration.
  • shared/: Contains the shared configuration for all clusters.
    • apps/: Contains the application specific manifests.
      • FleetDM - To provide a device management for my devices. (currently not in use, as it does not support ARM64)
      • Headlamp - To provide a lightweight and extensible Kubernetes UI.
      • Homepage - To provide a dashborad for the cluster.
      • Open WebUI - To provide a web interface and a REST API for interacting with LLM's.
      • PlantUML - To provide a web interface and a REST API for generating PlantUML diagrams.
      • Traefik - To provide an ingress controller for the cluster.
    • custom-resources/: Contains the custom resources that are used across the cluster.
    • infrastructure/: Contains the infrastructure specific manifests.
    • tenants: Contains Flux kustomizations to bootstrap and onboard tenants. (currently not in use)
    • variables/: Contains global variables, that are the same for all clusters.

Kustomize and Flux Kustomization Flow

To support hooking into the kustomize flow for adding or modifying resources for a specific cluster, a specific distribution, or shared across all clusters, the following structure is used:

Structure

This means that for every root level kustomization that is applied to the cluster, there should be a corresponding folder in either clusters, distributions, or shared that contains the resources that should be applied to the cluster at that scope. For example, for a root level kustomization in k8s/clusters/<cluster-name>/flux-system/infrastructure.yaml, there should be a corresponding folder in:

  • k8s/clusters/<cluster-name>/infrastructure/
  • k8s/distributions/<distribution-name>/infrastructure/
  • k8s/shared/infrastructure/

Production Environment

Nodes

  • 1x Hetzner CAX21 node (QEMU ARM64 4CPU 8Gb RAM 160Gb SSD) for both control plane and worker node
  • 2x Hetzner CAX41 node (QEMU ARM64 16CPU 32Gb RAM 320Gb SSD) for both control plane and worker nodes
  • 1x Apple Hypervisor ARM64 VM (Running on Mac Mini M2 Pro with access to 32GB RAM and 20 cores (overprovisioned 2/1) as a worker node

Hardware

Software

  • Unifi - For configuring a DMZ zone for my own nodes to run in, along with other security features.
  • UTM - For running Kubernetes on Mac Mini via Apple Hypervisor.
  • Talos Omni - For provisioning the production cluster, and managing nodes, updates, and the Talos configuration.
  • Cloudflare - For etcd backups, DNS, and tunneling all traffic so my network stays private.
  • Flux GitOps - For managing the kubernetes applications and infrastructure declaratively.
  • SOPS and Age - For encrypting secrets at rest, allowing me to store them in this repository with confidence.
  • KSail - For developing the cluster locally, and for running the cluster in CI to ensure all changes are properly tested before being applied to the production cluster.
  • K8sGPT - To analyze the cluster for improvements, vulnerabilities or bugs. It integrates with Trivy and Kuverno to also provide security and policy suggestions.

Monthly Cost

Item No. Per unit Total
Hetzner CAX21 3 7,49€ $24,9
Hetzner CAX41 1 29,99€ $33,23
Talos Omni 1 $10 $10
Cloudflare Domains 2 $0,87 $1,74
$69,87

Star History

Star History Chart

About

Devantler's Homelab - including deployment artifacts for running the Homelab in CI/CD and in Talos Omni.

Topics

Resources

Security policy

Stars

Watchers

Forks

Sponsor this project

 

Packages