Building a Production-Grade k3s Homelab

Most homelab guides stop at 'get Kubernetes running.' This one doesn't. I wanted something that behaved like a real production cluster: GitOps-driven deploys, proper observability, TLS everywhere, and no open ports on my router. I also wanted to understand every layer — from the hypervisor to the running pods. Here's how I built it.

Why build this

A big part of my career has been in tech support and solutions engineering. Understanding every moving part — and being able to reproduce production situations on demand — is how I grow. A homelab makes that possible. It's a place to experiment with new technology, validate ideas before they reach production, and stay sharp on infrastructure patterns that otherwise only show up in incident postmortems.

The hardware

Everything runs on a single Proxmox node I bought used. Proxmox handles the hypervisor layer — VMs get CPU and memory slices, and the Windows workstation VM gets full hardware passthrough for a native experience.

VM layout

The Proxmox host runs seven VMs. Four form the k3s cluster; the other three serve as a workstation, a home-automation hub, and a Linux dev environment.

• k3sserver01 — control-plane node. Runs the k3s API server, etcd, and scheduler.
• k3sagent01 / k3sagent02 / k3sagent03 — worker nodes. All workloads run here.
• Windows 11 workstation — NVIDIA RTX 4080 SUPER, Samsung 990 Pro 2TB NVMe, keyboard, mouse, and monitors all passed through. From the outside it looks like a bare-metal PC.
• Home Assistant OS — handles ESP32 IoT sensors, NOAA aurora alerts, and smart-home automation. Provisioned via the HAOS community script.
• Ubuntu Server (dev) — headless. Used for Linux-native workloads and experiments. I SSH in with Warp terminal.

Why k3s instead of full Kubernetes

k3s is Rancher's lightweight Kubernetes distribution — single binary, SQLite by default, ships with Traefik and a service load balancer built in. For a homelab it's ideal: you get a fully conformant cluster without spending half your RAM on control plane overhead.

Provisioning the cluster

I provision VMs from a Proxmox cloud-init template. Each VM gets static IPs configured in /etc/network/interfaces — ens18 on the home subnet for management, ens19 on the cluster network for pod traffic. Rather than passing flags to the install script, I write a config.yaml first so the config is version-controlled and reproducible:

# k3sserver01
cluster-domain: "arkube.local"
tls-san:
    - "192.168.1.10"
    - "k3sserver01.aruiznet.lan"
flannel-backend: host-gw
flannel-iface: "ens19"
bind-address: "0.0.0.0"
https-listen-port: 6443
advertise-address: "10.0.0.1"
advertise-port: 6443
node-ip: "10.0.0.1"
node-external-ip: "192.168.1.10"
log: "/var/log/k3s.log"
kubelet-arg: "config=/etc/rancher/k3s/kubelet.config"
disable:
    - metrics-server
    - servicelb
protect-kernel-defaults: true
secrets-encryption: true

A few things worth calling out: servicelb is disabled because MetalLB handles load balancer IPs. metrics-server is disabled because kube-prometheus-stack already ships one. secrets-encryption: true enables envelope encryption for Kubernetes Secrets at rest. With the config in place, the install is just:

curl -sfL https://get.k3s.io | sh -

# Grab the join token for agent nodes
cat /var/lib/rancher/k3s/server/node-token

Each agent node gets a matching config before running the install script:

# k3sagent01
server: "https://10.0.0.1:6443"
token: "<node-token>"
flannel-iface: "ens19"
node-ip: "10.0.0.21"
node-external-ip: "192.168.1.21"
log: "/var/log/k3s.log"
kubelet-arg: "config=/etc/rancher/k3s/kubelet.config"
protect-kernel-defaults: true

curl -sfL https://get.k3s.io | sh -s - agent

Flannel uses host-gw backend, which means it programs static routes between nodes rather than creating an overlay network. Slightly more performant for a homelab where all nodes are on the same L2 segment (10.0.0.x). After joining all three agents:

kubectl get nodes
# NAME          STATUS   ROLES                  AGE   VERSION
# k3sserver01   Ready    control-plane,master   99d   v1.34.1+k3s1
# k3sagent01    Ready    <none>                 99d   v1.34.1+k3s1
# k3sagent02    Ready    <none>                 99d   v1.34.1+k3s1
# k3sagent03    Ready    <none>                 99d   v1.34.1+k3s1

Networking: MetalLB + Cloudflare Tunnel

k3s ships with a built-in load balancer called servicelb, but I disabled it in config.yaml and replaced it with MetalLB v0.14.9 for more control over IP allocation. I reserved roughly 20 IPs on my home subnet and defined them as an IPAddressPool. MetalLB announces these IPs via L2 advertisement, so any LoadBalancer-type service gets a stable home-network IP automatically. I occasionally flip a service to LoadBalancer type to test it locally before wiring it through the Cloudflare Tunnel.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default
  namespace: metallb-system
spec:
  addresses:
    - 192.168.1.60-192.168.1.79   # ~20 IPs reserved on home subnet
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default
  namespace: metallb-system
spec:
  ipAddressPools:
    - default

For external access, I use Cloudflare Tunnel instead of port forwarding or a static IP. The cloudflared pod dials out to Cloudflare's edge and holds a persistent encrypted connection. Traffic flows Cloudflare → tunnel → Traefik ingress → pod. My router firewall has zero inbound rules.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
  namespace: kube-system
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: cloudflared
          image: cloudflare/cloudflared:latest
          args:
            - tunnel
            - --config
            - /etc/cloudflared/config.yaml
            - run

GitOps with Argo CD

I started with manual kubectl apply deploys, but that gets old fast. I ported my workloads over to a private GitHub repo and installed Argo CD to watch it. Now I push to main and Argo CD reconciles the cluster automatically. No credentials on the build server, no kubectl in CI.

kubectl create namespace argocd
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Each application gets an Application CRD in the repo. Argo CD detects changes on push and syncs automatically with prune: true and selfHeal: true.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: anthonypaulruiz
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/anthonypruiz/anthonypaulruiz
    targetRevision: main
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: apr
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Observability

The cluster runs a full observability stack in the monitoring namespace: Prometheus for metrics, Loki for logs, and Grafana for dashboards. This portfolio site ships OpenTelemetry spans and structured Pino logs to Loki on every request, with trace-to-log correlation via traceId injection.

• kube-prometheus-stack via Helm — Prometheus v3.11.1, Alertmanager, node-exporter, kube-state-metrics
• Loki distributed stack (write/read/backend components) with MinIO for chunk storage
• Grafana 11.2.0 — dashboards for cluster health, pod resources, and application logs
• OpenTelemetry Collector contrib — receives OTLP spans from the portfolio app and forwards to Loki
• Custom RingBufferExporter in the portfolio app ships OTEL spans to Loki for trace-to-log correlation

What's running in the cluster

Beyond the monitoring stack and this portfolio site, the cluster has become a general-purpose platform. Some notable workloads:

• ARAI (ai namespace) — a personal AI assistant built on a FastAPI backend with a Next.js frontend, backed by CloudNative-PG (PostgreSQL 18)
• Lobe Chat (lobe-chat namespace) — self-hosted AI chat with SearXNG for private web search and MinIO for file storage
• Private container registry (registry namespace) — self-hosted GHCR mirror for internal images
• MariaDB — general-purpose relational database for apps that need it

Automatic TLS with cert-manager

Even without a public IP, internal services still need TLS. cert-manager handles certificate provisioning and renewal automatically. For services exposed through Cloudflare Tunnel, Cloudflare terminates TLS at the edge — but cert-manager covers internal service-to-service traffic and any services accessed directly over the home subnet.

CI/CD pipeline

GitHub Actions builds a Docker image on every push to main, tags it with the commit SHA, pushes to GHCR, then patches the image tag in k8s/deployment.yaml and commits back to the repo. Argo CD detects the manifest change and rolls the cluster forward.

git push → GitHub Actions
  → docker build + push (ghcr.io/anthonypruiz/anthonypaulruiz:sha-<sha>)
  → sed image tag into k8s/deployment.yaml
  → git commit + push
  → Argo CD detects diff → sync
  → cluster rolls to new image

Gotchas I hit along the way

Nothing in this post came from a first attempt. A few things that cost me time and would have been useful to know upfront:

• Flannel `host-gw` requires L2 adjacency. All nodes must be on the same network segment for host-gw routing to work. If you put nodes on different subnets, Flannel silently falls back to vxlan or routes don't propagate. The dedicated 10.0.0.x cluster NIC on ens19 solves this — every node is L2-adjacent on that interface.
• **Disable servicelb *before* installing MetalLB, not after.** If both are running simultaneously they fight over LoadBalancer service IPs. Set disable: [servicelb] in config.yaml before the first curl | sh, then install MetalLB clean.
• `protect-kernel-defaults: true` will fail if sysctl values aren't pre-set on the VM. k3s checks kernel parameters on startup and exits if they don't match expected values. Set vm.panic_on_oom=0, kernel.panic=10, and kernel.panic_on_oops=1 in /etc/sysctl.d/ on each VM before running the install script.
• Always specify `flannel-iface` explicitly. Without it, Flannel picks an interface on its own — usually the wrong one. If it binds to ens18 (the home subnet NIC) instead of ens19, pod-to-pod traffic leaks onto your home network and MTU issues follow.
• `secrets-encryption: true` is a one-way door without planning. Enabling envelope encryption for Secrets at rest is good practice, but if you ever need to rotate the encryption key or restore from backup, you need the key rotation procedure ready. Back up /var/lib/rancher/k3s/server/cred/ before enabling it.
• The node token must match across all agents. If you re-initialize the server node (even just to change a config option), the token regenerates and existing agents can no longer join. Always snapshot the token before any server-node changes.

What's next

Part 2 will cover the observability stack in detail — how I configured Prometheus scrape targets, built the Grafana dashboards, and set up trace-to-log correlation between OTEL and Loki. Part 3 will cover the Cloudflare Tunnel setup end-to-end: tunnel creation, DNS records, and the Traefik ingress config that ties it all together.