Homelab + workstation
The lab.
What runs at home, drawn to spec. It’s mid-rebuild, so anything dashed is next.
Rebuild in progress · revised
- Threads
- 64
- threads, two Xeons, one R730
- Memory
- 512GB
- of RAM in the same box
- ZFS pools
- 3
- ZFS pools: rpool, fast, bulk
- Cluster
- 3
- Talos 1.14 nodes on Proxmox
- Cabinet
- 9U
- of 42U in the cabinet
The cabinet.
9U of 42U in use. The rest is air, for now.
Server · U13–14
Dell PowerEdge R730
- 2 × Xeon E5-2683 v4: 64 threads, 512 GB RAM.
- Proxmox VE 9.2 on ZFS, ten drives in 16 bays.
- The RAID card does HBA mode itself: no surgery needed.
Gateway · U10
UniFi Dream Machine Pro
- Router and firewall. Everything else hangs off it.
- WAN on port 9, to the ONT.
- Port 2: the Pi. Port 3: the server.
- SFP+ port 11: the NAS, at 10 GbE.
Access point · off-sheet
UniFi U7 Pro
- The Wi-Fi 7 access point.
- Lives outside the cabinet; the PoE injector powers it.
From the ISP · top shelf
Fiber ONT
- Optical network terminal: fiber in, Ethernet out.
- Feeds the gateway’s WAN port.
Power over Ethernet · top shelf
PoE injector
- Powers the U7 Pro down its network cable.
- Network in from the gateway, power and network out.
NAS · U8–9
UniFi UNAS Pro 8
- Eight drive bays of shared storage.
- Serves NFS to the lab.
- 10 GbE to the gateway over SFP+.
Control node · top shelf
Raspberry Pi 4 Model B
- 8 GB, in a black case.
- The control node. Ansible configures it too.
- Plugged into the gateway’s port 2.
| No | Device | U |
|---|---|---|
| Compute | ||
| 1 | U13–14 | |
| Network | ||
| 2 | U10 | |
| 3 | Off sheet | |
| 4 | U1–2 | |
| 5 | U1–2 | |
| Storage | ||
| 6 | U8–9 | |
| Edge | ||
| 7 | U1–2 | |
Everything lives in the top 14U, except a shelf of spares at the bottom. Sheet 01 · The cabinet
The stack.
Metal at the bottom, Kubernetes on top. The top floor is still going up.
Metal
Dell PowerEdge R730
- 64 threads, 512 GB RAM, ten drives.
- Its RAID card runs in HBA mode, so every disk goes straight to ZFS.
Storage
ZFS pools
- rpool: boot, a 3-way mirror.
- fast: VM disks, a 3-way mirror with a hot spare.
- bulk: scratch, one disk.
Hypervisor
Proxmox VE 9.2
- Installed by hand, on ZFS.
- Ansible configures it after that.
Ansible playbook
Host config
- Hourly and daily ZFS snapshots of the VM pool.
- SMART self-tests and ZFS event alerts.
- Push alerts to my phone, through ntfy.
- Metrics exporters for the host and its hardware.
Control node
Raspberry Pi 4
- The control node, on the top shelf.
- Configured by Ansible, like the host.
Gateway
UniFi Dream Machine Pro
- Router and firewall. Everything else hangs off it.
Network storage
UniFi UNAS Pro 8
- NFS storage, 10 GbE from the gateway.
Kubernetes cluster
Talos nodes ×3
- Three VMs, each 8 vCPU, 24 GiB RAM and 64 GB on fast.
- Talos 1.14, running Kubernetes 1.37.
- All three are control plane, and all three run workloads.
- The OS image comes from Sidero’s Image Factory.
Standalone VM
GitHub runner
- Self-hosted GitHub Actions runners, in their own VM.
- 8 vCPU, 16 GiB RAM, 200 GB on bulk.
Standalone VM · planned
robomp
- The oh-my-pi GitHub bot, in its own VM.
- It will triage my personal projects and fix them through PRs.
Networking
Cilium 1.20
- The cluster’s network (CNI).
- Replaces kube-proxy.
- Hubble shows the traffic.
- Its Gateway API serves my apps over HTTPS.
Metrics
metrics-server 0.9
- CPU and memory use per node and pod.
- It’s what makes kubectl top work.
GitOps
Flux
- Keeps the cluster in step with the repo.
- Took over Cilium from Helm, and runs everything after it.
Certificates
cert-manager 1.21
- Issues and renews certificates.
- Let’s Encrypt ones, so my apps get real HTTPS.
Secrets
External Secrets
- External Secrets with 1Password Connect.
- Apps get their secrets from 1Password.
Block storage
Proxmox CSI
- Volumes carved out of the fast pool by default, bulk on request.
- Attached to whichever node runs the pod.
Metrics and logs
Observability
- VictoriaMetrics for metrics, VictoriaLogs for logs.
- Grafana for looking at both.
- The alert rules run, and they page my phone through ntfy.
Databases
CloudNativePG 1.30
- Postgres, for the apps that want one.
- Its backups land in RustFS.
Object storage
RustFS 1.0
- S3-compatible storage inside the cluster.
- Database backups land here.
Guardrails
Admission policies
- Built-in Kubernetes rules that turn risky changes away.
Shared storage · planned
NFS CSI
- Volumes on the UNAS, over NFS.
Backups · planned
VolSync
- Backs up the cluster’s volumes to the UNAS.
Cluster UI
Headlamp
- A web UI for the cluster.
- The first app in, mostly so I can click instead of type.
Dashboard
Homepage 2.4
- A start page for the lab.
- Everything I run, one click away.
Status page
Gatus 5.37
- Checks that everything answers.
- Pages my phone when something doesn’t.
Registry
Zot 2.1
- A private container registry.
- Container images stay in the lab.
Hover or tap anything numbered.
Drawn to spec.
The lab as engineering drawings, all generated from one set of data. Two are as built. The stack is still going up.
- TD-LAB-01As built The cabinet The 42U, top to bottom: a shelf of small stuff, the NAS, the gateway, the server, the spares.
- TD-LAB-02As built The server The R730 inside and out: sixteen bays, ten drives, three pools.
- TD-LAB-03In progress The stack What runs on the metal, in three layers. A few dashed lines left.
Changelog.
- · latest
CloudNativePG, RustFS
Postgres for the apps that want one, and an S3 store inside the cluster for its backups.
-
R720xd → R730
Swapped the server for an R730: 64 threads, 512 GB of RAM, and a RAID card that does HBA mode on its own. Then rebuilt the whole cluster on it from git.
-
A registry, a GitHub runner
Zot keeps my container images in the lab, and a runner VM builds them.
-
Homepage, Gatus
A start page for the lab, a status page that checks it, and alerts that finally reach my phone.
-
kubectl top → Grafana
Metrics and logs land in VictoriaMetrics and VictoriaLogs now, and Grafana draws them. The alert rules run too, but nothing pages me yet. On purpose.
-
Real HTTPS, first app
Apps on my network get proper HTTPS now: cert-manager fetches the certificates and Cilium serves them. First one in: Headlamp, a web UI for the cluster.
-
Talos cluster, Flux and friends
Three Talos VMs on Proxmox, built with OpenTofu. Cilium runs the network, and Flux runs the cluster from git.
-
PERC H710P → HBA330, Proxmox on ZFS
Pulled the RAID card, put in an HBA, and reinstalled Proxmox 9.2 on three ZFS pools.
-
Cloudflare Tunnels → Tailscale Operator
Swapped Cloudflare Tunnels for the Tailscale Kubernetes Operator, so a new service is one YAML file in git.
-
Talos Kubernetes on Proxmox, with Terraform
An immutable, Kubernetes-only OS on Proxmox, built end to end with Terraform.
The desk.
