quorate/docs/spec/homelab-spec.md

Home Infrastructure Specification

1. Overview

This document specifies a small home infrastructure environment built around a multi-node K3s cluster, a dedicated north-south boundary, and three complementary storage layers. The design prioritizes declarative operations, clear trust boundaries, and pragmatic tradeoffs appropriate for a single-operator environment.

2. Terminology

This document uses a small number of context-specific terms:

Term	Meaning
Boundary	The home network edge, implemented by OPNsense, where WAN egress, remote-client WireGuard, routing, and firewall policy are enforced.
ZFS node	The 7840HS K3s agent that hosts the cluster’s bulk HDD storage and all ZFS-backed workloads.
Tier H / O / R / S / I / D / C	Classifications defined in §6. They describe the path by which a service is reached.
Storage-heavy workload	A workload whose primary persistence depends on ZFS-backed storage and which is therefore pinned to the ZFS node via node affinity.

---

3. Hardware Inventory

Role	CPU	RAM	Boot Storage	Bulk Storage	OS	K3s Role
Control Plane 1	Intel Xeon E5-2697 v3	64 GB	512 GB SATA SSD	512 GB SATA SSD + 320 GB HDD	Debian (headless, LUKS)	Server
Control Plane 2	Intel Core i7-7700HQ	24 GB	128 GB NVMe SSD	1 TB HDD	Debian (headless, LUKS)	Server
Control Plane 3	Rockchip RK3588	16 GB	512 GB NVMe (PCIe 2.0 x1)	—	Armbian (headless, LUKS)	Server
Agent — ZFS node	AMD 7840HS	48 GB	1 TB SSD	12×3 TB HDD (2× ZFS Z2 vdevs)	Debian (headless, LUKS)	Agent
Network Boundary	Intel Pentium Gold 8505	32 GB	128 GB SSD	—	OPNsense	N/A

3.1 Architecture Notes

• The control plane is a 3-node quorum spanning mixed x86_64 and arm64 systems.
• All K3s nodes boot from LUKS-encrypted root filesystems.
• The ZFS node is the only node with bulk HDD storage and is the scheduling target for all storage-heavy workloads.
• Multi-architecture container images are required for workloads that may schedule onto the RK3588.

3.2 Host Runtime Assumptions

K3s nodes are assumed to start from an already bootstrapped Linux host with hostname, LAN addressing, SSH access, time sync, and LUKS root encryption handled outside this specification. This section only documents host dependencies required by the cluster runtime and its deployed infrastructure.

All K3s nodes need the following host-level capabilities before joining the cluster:

• Cilium/K3s networking primitives: cgroups, overlay filesystems, VXLAN, WireGuard, eBPF, netfilter/conntrack kernel support, policy routing, and a mounted BPF filesystem.
• All K3s nodes require persistent net.ipv4.conf.all.rp_filter=0 and net.ipv4.conf.default.rp_filter=2 so OCI relay tunnel interfaces inherit loose reverse-path filtering.
• Longhorn storage prerequisites: iSCSI initiator userspace, NFS client support, and XFS formatting tools.
• Disk health prerequisites: SMART userspace (smartmontools) for smartctl-exporter.
• Swap disabled.
• K3s is installed without the bundled networking stack that Cilium replaces: flannel, kube-proxy, K3s network-policy, Traefik, and ServiceLB are disabled.

Debian/Armbian/Ubuntu runtime packages:

sudo apt install -y \
  open-iscsi nfs-common xfsprogs smartmontools

RHEL/Fedora-family runtime packages:

sudo dnf install -y \
  iscsi-initiator-utils nfs-utils xfsprogs smartmontools

Userspace iptables/nftables and WireGuard tools are useful for diagnostics and emergency cleanup, but they are not treated as required node runtime packages because Cilium is deployed with kube-proxy replacement, BPF masquerading, and WireGuard encryption managed by Cilium.

Longhorn expects /var/lib/longhorn/ to live on node-local SSD/NVMe storage and to be available before Longhorn starts. Nodes that participate in Longhorn scheduling are tagged with the ssd disk selector.

The ZFS node has additional host requirements:

• ZFS userspace and kernel support installed.
• The pool ******** imported at boot.
• The dataset ******** present for OpenEBS LocalPV ZFS.
• The Kubernetes node labeled has-zfs=true.

---

4. Design Constraints and Tradeoffs

This environment is intentionally designed around pragmatic home-lab constraints rather than datacenter-style redundancy.

• The ZFS node is a deliberate single-node failure domain for all ZFS-backed workloads.
• Local ZFS redundancy mitigates disk failure, but not host failure.
• Replicating the bulk-storage tier on a second host is currently out of scope due to cost, particularly HDD pricing.
• VersityGW is intentionally coupled to the ZFS node and is therefore also a deliberate single-node availability tradeoff.

These are accepted architectural constraints, not accidental omissions.

---

5. Network Architecture

5.1 Topology

%%{init: {'theme': 'neutral'}}%%
graph TD
    WAN["WAN (ISP Public IP)"]
    CF["Cloudflare Edge"]
    OCI["OCI L4 Relay\nReserved Public IP"]
    NLB["OCI NLB"]

    subgraph boundary["Home Boundary"]
        OPN["OPNsense\nPentium Gold 8505"]
        WG["Remote-client WireGuard"]
    end

    subgraph cluster["K3s Cluster (LAN)"]
        subgraph cp["Control Plane"]
            CP1["Xeon E5-2697 v3"]
            CP2["Core i7-7700HQ"]
            CP3["RK3588"]
        end
        subgraph agents["Agent"]
            A1["7840HS (ZFS node)"]
        end
        CIL["Cilium CNI"]
        GW_R["Gateway R\n(Relay Services)"]
        LB_R["LoadBalancer R\n(L4-passed Services)"]
        Short["Short tunnel"]
        GW_S["Gateway S\n(Short Services)"]
        SVC_S["Services"]
        GW_I["Gateway I\n(Internal Services)"]
        LB_D["LoadBalancer D\n(Direct Services)"]
    end

    WAN --> OPN
    CF -->|Cloudflare Tunnel| CIL
    WAN -->|Public relay traffic| OCI
    OCI -->|"WireGuard: web"| GW_R
    OCI -->|"WireGuard: mail"| LB_R
    WAN -->|Public NLB traffic| NLB
    NLB -->|"Site-to-Site VPN"| Short --> GW_S --> SVC_S
    OPN <-->|BGP Peering| CIL
    WG --> OPN
    OPN -->|LAN route| GW_I
    OPN -->|LAN route| LB_D
    CIL --- GW_R
    CIL --- LB_R
    CIL --- GW_S
    CIL --- GW_I
    CIL --- LB_D
    CIL --- CP1
    CIL --- CP2
    CIL --- CP3
    CIL --- A1

5.2 Cilium Configuration

Feature	Status
Kube-proxy replacement	Enabled
BGP peering with OPNsense	Enabled
Gateway API	Enabled
Global default deny	Enabled
Host firewall	Enabled
Pod traffic encryption	WireGuard
IPv6	Disabled at boundary

5.3 OPNsense Configuration

Feature	Detail
WAN interface	ISP-assigned public IP
LAN interface	Private subnet
WireGuard	Remote clients behave as LAN-local
BGP	Peered with Cilium for LoadBalancerIPPool advertisement
Port forwarding	Service-specific rules only; no shared public Gateway
Relay egress	Outbound WireGuard to OCI from the elected K3s node

5.4 IPv6 Policy

IPv6 is disabled for north-south traffic at the network boundary; the environment operates IPv4-only.

5.5 Split-Horizon DNS

To avoid WAN hairpinning, selected public service hostnames resolve differently for internal and external clients.

• External clients resolve public hostnames according to public DNS and reach services through Tier O, Tier R, or Tier S, depending on the service.
• LAN and WireGuard clients resolve selected public service hostnames to private ingress addresses so the same services are reached through internal paths instead of via the ISP-facing WAN address.

This allows Internet-facing services to remain externally reachable while internal clients prefer private ingress paths.

A dedicated CoreDNS instance runs in-cluster, separate from the cluster-internal K3s CoreDNS, and is authoritative for the split-horizon zones. OPNsense Unbound forwards those zones to it over DNS-over-TLS. This keeps DNS overrides version-controlled and avoids click-ops in OPNsense.

---

6. Ingress Tiers

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    subgraph external["External"]
        Internet
        CFEdge["Cloudflare Edge"]
    end

    subgraph oci["OCI"]
        Admission["Address Admission"]
        Relay["L4 Relay"]
        NLB["OCI NLB"]
    end

    subgraph cluster["K3s Cluster"]
        GW_R["Gateway R\n(Relay)"]
        LB_R["LoadBalancer R\n(L4-passed)"]
        Short["Short tunnel"]
        GW_S["Gateway S\n(Short)"]
        GW_I["Gateway I\n(Internal)"]
        LB_D["LoadBalancer D\n(Direct)"]
        SVC["Cluster Services"]
    end

    Internet -->|"O: Cloudflare Tunnel"| CFEdge --> SVC
    Internet -->|"R: declared TCP ports"| Admission --> Relay
    Relay -->|"80/443"| GW_R --> SVC
    Relay -->|"L4-passed service"| LB_R --> SVC
    Internet -->|"S"| NLB
    NLB -->|"Site-to-Site VPN"| Short --> GW_S --> SVC
    LAN["LAN / WireGuard"] -->|"I: Private CIDR"| GW_I --> SVC
    LAN["LAN / WireGuard"] -->|"D: Dedicated IP"| LB_D --> SVC
    SVC -->|"C: ClusterIP only"| SVC

Tier	Label	Path	TLS Termination	Certificate Authority	Auth Requirement
H	Hypertext	Cloudflare Pages; no cluster ingress	Cloudflare Edge	Cloudflare	Anonymous
O	Orange Cloud	Cloudflare Tunnel → Service	Cloudflare Edge	Cloudflare	Cloudflare-managed
R	Relay	OCI L4 relay → WireGuard → Gateway R → Service	Gateway R or L4-passed service	Let’s Encrypt (DNS-01)	Required per service
S	Short	OCI NLB → Site-to-Site VPN → Gateway S	Gateway S or none	Let’s Encrypt (DNS-01)	Service-native
I	Internal	LAN / WireGuard → Gateway I → Service	Gateway I	Vault PKI	Network-level (LAN/WG)
D	Direct	LAN / WireGuard → LoadBalancer IP → Service	Service-native (or none)	Service-native	Service-native
C	Cluster Internal	ClusterIP only	Service-specific (or none)	Service-specific	Cilium network policy

The OCI L4 Relay Specification is authoritative for the Tier R design. The Tier S model is authoritative for the Tier S design.

6.1 Gateway IP Allocation

Gateway	CiliumLoadBalancerIPPool	Advertised via
Gateway R	Dedicated private CIDR pool	Routed over the OCI WireGuard path
Gateway S	ClusterIP; no dedicated pool	Reached through the Site-to-Site VPN
Gateway I	Dedicated private CIDR pool	BGP → LAN
Tier D	Dedicated private CIDR pool	BGP → LAN

Known and reserved LoadBalancer addresses:

Address	Owner
10.90.141.119	Gateway I
10.104.40.127	Gateway R
10.190.165.53	Forgejo SSH
10.28.147.227	Network DNS / CoreDNS
172.28.129.182	Zot direct registry endpoint
10.108.252.250	Quay direct registry endpoint
10.66.127.81	VersityGW S3 API direct endpoint
10.59.91.218	VersityGW WebUI direct endpoint
10.69.181.158	Vault active API direct endpoint
10.47.201.154	Public Forgejo SSH
10.66.153.202	Stalwart mail and account HTTPS
10.80.158.106	Short VPN endpoint
10.157.251.228	PostgreSQL read-only service
10.234.158.153	PostgreSQL read/write service

---

7. Workload Inventory

Workload	Purpose	Tier	Replicas	ZFS Node	Storage Backend
VictoriaMetrics	Metrics (time-series)	I	split	—	OpenEBS LocalPV Hostpath
VictoriaLogs	Logs	I	split	—	Longhorn (replica=2)
Grafana	Metrics visualization, alerting	I	≥2	—	—
VersityGW	S3-compatible object gateway	I	=1	Yes	OpenEBS LocalPV ZFS
VersityGW WebGUI	Administrative interface for VersityGW	I	=1	—	—
CloudNativePG	Managed PostgreSQL	D	=3	—	OpenEBS LocalPV Hostpath
Internal Forgejo	Private Git hosting and Flux GitOps origin	I / D	≥2	—	Longhorn RWX, VersityGW (S3), Dragonfly, CloudNativePG
Public Forgejo	Public source development and release distribution	O / R / D	≥2 per root	—	Longhorn RWX, VersityGW (S3), Dragonfly, CloudNativePG
FluxCD	GitOps controller	I	≥2	—	—
Argo Workflows	CI/CD, workflow automation, and curated image mirror pipelines	I	≥2	—	VersityGW (S3, artifacts), CloudNativePG
OLM	Operator Lifecycle Manager (default catalogs disabled)	C	≥2	—	—
Quay	Container registry for curated internal images and scanned mirrors	D	≥2	—	VersityGW (S3), CloudNativePG
Clair	Image vulnerability scanner for Quay	C	≥2	—	CloudNativePG
Zitadel	Identity provider (OIDC)	R	≥2	—	CloudNativePG
Cilium	CNI, kube-proxy replacement, BGP, network policy, host firewall	C	DaemonSet	—	—
Hubble	Cilium network observability (relay + UI)	I	≥2	—	—
Longhorn	Replicated block storage	I	DaemonSet	—	Distributed node disks
OpenEBS LocalPV ZFS	Local ZFS-backed PVs	C	DaemonSet	Yes	ZFS (2× Z2 vdevs)
Dragonfly	In-memory cache (Redis-compatible)	C	=2 each	—	— (memory); dedicated CR per consumer
cert-manager	Certificate lifecycle automation	C	≥2	—	—
Vault	Secrets management, internal CA	I	=3 (Raft)	—	OpenEBS LocalPV Hostpath
oauth2-proxy	Auth proxy for selected web UIs	R / I	≥2	—	—
Vault Secrets Operator	Sync Vault secrets → K8s Secrets	C	≥2	—	—
Stalwart	Authoritative mail server and alert-notification sink	I / R	≥2	—	CloudNativePG, VersityGW (S3), Dragonfly
Frigate	Home security NVR (1×1080p, 7d retention)	I	=1	—	Longhorn

Victoria components: durable storage components run as one replica while the cluster has one ZFS node. Stateless insert/select components may run with multiple replicas. Collection agents are not independent workloads.

Stalwart's service contract is defined by the Authoritative Email Server Specification.

7.1 Forgejo

Forgejo implements separate internal and public repository authorities. The authority, ingress, state, runtime, feature, and verification contracts are in the Forgejo Deployment Specification.

7.2 Container Image Supply Chain

Quay native proxy cache is intentionally disabled. Cold pull-through requires Quay-side organization membership and does not satisfy the desired declarative access model.

Images intended for cluster consumption should be mirrored into Quay from a Git-owned allowlist, scanned by Clair, and promoted only after policy checks pass. Argo Workflows is the planned automation point for this mirror/scan/promote flow. Zot remains available as a lightweight direct registry for bootstrap and locally built infrastructure images.

---

8. Storage Architecture

The cluster uses three complementary storage layers, each serving a distinct purpose: replicated block storage for lightweight stateful workloads, high-capacity local storage for bulk data, and S3-compatible object storage for service backends.

8.1 Storage Layers

Longhorn provides replicated block storage across the K3s nodes. It is used for workloads that need persistent volumes but do not have large capacity requirements.

OpenEBS LocalPV Hostpath exposes node-local host paths as PersistentVolumes. It is used for workloads that implement their own application-level replication or clustering.

OpenEBS LocalPV ZFS exposes local ZFS datasets as PersistentVolumes. It is constrained to the ZFS node, which hosts 2× Z2 vdevs over 12×3 TB HDDs. This layer is used for workloads with large or IO-intensive storage needs.

VersityGW provides an S3-compatible object gateway running on the ZFS node. It exposes storage backed by the ZFS node via an S3 API for workloads that consume S3-compatible storage directly.

8.2 Consumer Mapping

Storage Layer	Consumer	Longhorn Replicas	Notes
OpenEBS LocalPV Hostpath	Vault (Raft)	N/A	Vault handles its own replication via Raft across local host paths
OpenEBS LocalPV Hostpath	CloudNativePG	N/A	CNPG handles its own replication via streaming replication
OpenEBS LocalPV Hostpath	VictoriaMetrics	N/A	Multiple vmstorage replicas handle metrics redundancy across local host paths
Longhorn	VictoriaLogs	2	Highest-volume logs component
Longhorn	Frigate	Default	Single 1080p camera, 7-day retention
Longhorn RWX	Forgejo git/SSH	share-manager	Bare repos and built-in SSH keys; NFS semantics acceptable at homelab scale
VersityGW (S3)	Forgejo blobs	N/A	Avatars and attachments in bucket forgejo (port 7070 in-cluster)
Dragonfly	Forgejo cache	N/A	Dedicated forgejo CR; session, cache, and queue
CloudNativePG	Stalwart state	N/A	Durable account, metadata, queue, and search state
VersityGW (S3)	Stalwart blobs	N/A	Durable message bodies and attachments in bucket stalwart
Dragonfly	Stalwart cache	N/A	Dedicated non-durable Redis-compatible state
OpenEBS LocalPV ZFS	VersityGW	N/A	ZFS node only

8.3 Design Principles

Principle	Implementation
Prefer application-level replication where explicitly deployed	Vault Raft, CNPG streaming replication, and VictoriaMetrics clustering justify using non-replicated local storage (OpenEBS LocalPV Hostpath) instead of replicated block storage
Bulk data on ZFS	Large-capacity workloads are pinned to the ZFS node via node affinity
Object-storage backend	VersityGW provides S3-compatible storage for workloads that require it
Separation of concerns	Longhorn for general PVs; OpenEBS LocalPV Hostpath for high-performance clustered apps; OpenEBS LocalPV ZFS for bulk local storage; VersityGW for object storage

---

9. Security Architecture

9.1 Trust Boundaries

%%{init: {'theme': 'neutral'}}%%
graph TD
    subgraph internet["Untrusted — Internet"]
        ExtClient["External Client"]
    end

    subgraph cf_edge["Semi-Trusted — Cloudflare"]
        CFTunnel["Cloudflare Tunnel + WAF"]
    end

    subgraph relay["Boundary — OCI Relay"]
        Admission["Address Admission + L4 Limits"]
        WGRelay["WireGuard Relay"]
    end

    subgraph boundary["Boundary — OPNsense"]
        WG["Remote-client WireGuard"]
    end

    subgraph lan["Trusted — LAN"]
        subgraph k3s["K3s Cluster"]
            CiliumFW["Cilium Host Firewall"]
            DefaultDeny["Global Default Deny"]
            GW_R2["Gateway R"]
            GW_I2["Gateway I"]
            LB_R2["LoadBalancer R"]
            Workloads["Workloads"]
        end
    end

    ExtClient -->|"TLS (CF-terminated)"| CFTunnel -->|"Tunnel"| Workloads
    ExtClient -->|"Tier R"| Admission --> WGRelay
    WGRelay -->|"HTTP/TLS"| GW_R2
    WGRelay -->|"L4 pass-through"| LB_R2 --> Workloads
    ExtClient -->|"WireGuard"| WG -->|"LAN access"| GW_I2
    GW_R2 -->|"Cilium policy"| Workloads
    GW_I2 -->|"Cilium policy"| Workloads

9.2 Certificate Authorities

Scope	CA	Issuance	Managed By
Tier R (public-facing)	Let’s Encrypt	DNS-01 challenge (Cloudflare DNS)	cert-manager
Tier I (internal)	Vault PKI	Vault-issued	cert-manager + Vault
Tier O	Cloudflare	Cloudflare-managed	Cloudflare

9.3 Authentication Flow (Tier R Web Services)

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    actor User
    participant GW as Gateway R
    participant OAP as oauth2-proxy
    participant ZIT as Zitadel (IdP)
    participant SVC as Backend Service

    User->>GW: HTTPS request
    GW->>SVC: Route (if native OIDC)
    GW->>OAP: Route (if no native OIDC)
    OAP->>ZIT: OIDC redirect
    ZIT->>User: Login prompt
    User->>ZIT: Credentials
    ZIT->>OAP: ID token
    OAP->>SVC: Authenticated request
    SVC->>User: Response

Service	Auth Method
Forgejo	See deployment specification
Quay	Native OIDC → Zitadel
Zitadel	Self
VersityGW & UI	IAM-style auth (S3 access keys)
Hubble UI	oauth2-proxy → Zitadel
Longhorn UI	oauth2-proxy → Zitadel

9.4 Encryption at Rest

Layer	Mechanism
Node root filesystem	LUKS
K3s secrets	K3s secrets encryption
Application secrets	Vault → Vault Secrets Operator → K8s Secrets
ZFS data	ZFS native AES-256-GCM encryption inherited from ********