quorate/docs/spec/homelab-spec.md

# Home Infrastructure Specification

## 1. Overview

This document specifies a small home infrastructure environment built around a multi-node K3s cluster, a dedicated north-south boundary, and three complementary storage layers. The design prioritizes declarative operations, clear trust boundaries, and pragmatic tradeoffs appropriate for a single-operator environment.

## 2. Terminology

This document uses a small number of context-specific terms:

| Term | Meaning |
| --- | --- |
| **Boundary** | The home network edge, implemented by OPNsense, where WAN egress, remote-client WireGuard, routing, and firewall policy are enforced. |
| **ZFS node** | The 7840HS K3s agent that hosts the cluster’s bulk HDD storage and all ZFS-backed workloads. |
| **Tier H / O / R / S / I / D / C** | Classifications defined in §6. They describe the path by which a service is reached. |
| **Storage-heavy workload** | A workload whose primary persistence depends on ZFS-backed storage and which is therefore pinned to the ZFS node via node affinity. |

______________________________________________________________________


## 3. Hardware Inventory

| Role | CPU | RAM | Boot Storage | Bulk Storage | OS | K3s Role |
| --- | --- | --- | --- | --- | --- | --- |
| Control Plane 1 | Intel Xeon E5-2697 v3 | 64 GB | 512 GB SATA SSD | 512 GB SATA SSD + 320 GB HDD | Debian (headless, LUKS) | Server |
| Control Plane 2 | Intel Core i7-7700HQ | 24 GB | 128 GB NVMe SSD | 1 TB HDD | Debian (headless, LUKS) | Server |
| Control Plane 3 | Rockchip RK3588 | 16 GB | 512 GB NVMe (PCIe 2.0 x1) | — | Armbian (headless, LUKS) | Server |
| Agent — ZFS node | AMD 7840HS | 48 GB | 1 TB SSD | 12×3 TB HDD (2× ZFS Z2 vdevs) | Debian (headless, LUKS) | Agent |
| Network Boundary | Intel Pentium Gold 8505 | 32 GB | 128 GB SSD | — | OPNsense | N/A |

### 3.1 Architecture Notes

- The control plane is a 3-node quorum spanning mixed `x86_64` and `arm64` systems.
- All K3s nodes boot from LUKS-encrypted root filesystems.
- The **ZFS node** is the only node with bulk HDD storage and is the scheduling target for all storage-heavy workloads.
- Multi-architecture container images are required for workloads that may schedule onto the RK3588.

### 3.2 Host Runtime Assumptions

K3s nodes are assumed to start from an already bootstrapped Linux host with hostname, LAN addressing, SSH access, time sync, and LUKS root encryption handled outside this specification. This section only documents host dependencies required by the cluster runtime and its deployed infrastructure.

All K3s nodes need the following host-level capabilities before joining the cluster:

- Cilium/K3s networking primitives: cgroups, overlay filesystems, VXLAN, WireGuard, eBPF, netfilter/conntrack kernel support, policy routing, and a mounted BPF filesystem.
- All K3s nodes require persistent `net.ipv4.conf.all.rp_filter=0` and `net.ipv4.conf.default.rp_filter=2` so OCI relay tunnel interfaces inherit loose reverse-path filtering.
- Longhorn storage prerequisites: iSCSI initiator userspace, NFS client support, and XFS formatting tools.
- Disk health prerequisites: SMART userspace (`smartmontools`) for `smartctl-exporter`.
- Swap disabled.
- K3s is installed without the bundled networking stack that Cilium replaces: flannel, kube-proxy, K3s network-policy, Traefik, and ServiceLB are disabled.

Debian/Armbian/Ubuntu runtime packages:

```bash
sudo apt install -y \
  open-iscsi nfs-common xfsprogs smartmontools
```

RHEL/Fedora-family runtime packages:

```bash
sudo dnf install -y \
  iscsi-initiator-utils nfs-utils xfsprogs smartmontools
```

Userspace iptables/nftables and WireGuard tools are useful for diagnostics and emergency cleanup, but they are not treated as required node runtime packages because Cilium is deployed with kube-proxy replacement, BPF masquerading, and WireGuard encryption managed by Cilium.

Longhorn expects `/var/lib/longhorn/` to live on node-local SSD/NVMe storage and to be available before Longhorn starts. Nodes that participate in Longhorn scheduling are tagged with the `ssd` disk selector.

The ZFS node has additional host requirements:

- ZFS userspace and kernel support installed.
- The pool `********` imported at boot.
- The dataset `********` present for OpenEBS LocalPV ZFS.
- The Kubernetes node labeled `has-zfs=true`.

______________________________________________________________________


## 4. Design Constraints and Tradeoffs

This environment is intentionally designed around pragmatic home-lab constraints rather than datacenter-style redundancy.

- The **ZFS node** is a deliberate single-node failure domain for all ZFS-backed workloads.
- Local ZFS redundancy mitigates disk failure, but not host failure.
- Replicating the bulk-storage tier on a second host is currently out of scope due to cost, particularly HDD pricing.
- **VersityGW** is intentionally coupled to the ZFS node and is therefore also a deliberate single-node availability tradeoff.

These are accepted architectural constraints, not accidental omissions.

______________________________________________________________________


## 5. Network Architecture

### 5.1 Topology

```mermaid
%%{init: {'theme': 'neutral'}}%%
graph TD
    WAN["WAN (ISP Public IP)"]
    CF["Cloudflare Edge"]
    OCI["OCI L4 Relay\nReserved Public IP"]
    NLB["OCI NLB"]

    subgraph boundary["Home Boundary"]
        OPN["OPNsense\nPentium Gold 8505"]
        WG["Remote-client WireGuard"]
    end

    subgraph cluster["K3s Cluster (LAN)"]
        subgraph cp["Control Plane"]
            CP1["Xeon E5-2697 v3"]
            CP2["Core i7-7700HQ"]
            CP3["RK3588"]
        end
        subgraph agents["Agent"]
            A1["7840HS (ZFS node)"]
        end
        CIL["Cilium CNI"]
        GW_R["Gateway R\n(Relay Services)"]
        LB_R["LoadBalancer R\n(L4-passed Services)"]
        Short["Short tunnel"]
        GW_S["Gateway S\n(Short Services)"]
        SVC_S["Services"]
        GW_I["Gateway I\n(Internal Services)"]
        LB_D["LoadBalancer D\n(Direct Services)"]
    end

    WAN --> OPN
    CF -->|Cloudflare Tunnel| CIL
    WAN -->|Public relay traffic| OCI
    OCI -->|"WireGuard: web"| GW_R
    OCI -->|"WireGuard: mail"| LB_R
    WAN -->|Public NLB traffic| NLB
    NLB -->|"Site-to-Site VPN"| Short --> GW_S --> SVC_S
    OPN <-->|BGP Peering| CIL
    WG --> OPN
    OPN -->|LAN route| GW_I
    OPN -->|LAN route| LB_D
    CIL --- GW_R
    CIL --- LB_R
    CIL --- GW_S
    CIL --- GW_I
    CIL --- LB_D
    CIL --- CP1
    CIL --- CP2
    CIL --- CP3
    CIL --- A1
```

### 5.2 Cilium Configuration

| Feature | Status |
| --- | --- |
| Kube-proxy replacement | Enabled |
| BGP peering with OPNsense | Enabled |
| Gateway API | Enabled |
| Global default deny | Enabled |
| Host firewall | Enabled |
| Pod traffic encryption | WireGuard |
| IPv6 | Disabled at boundary |

### 5.3 OPNsense Configuration

| Feature | Detail |
| --- | --- |
| WAN interface | ISP-assigned public IP |
| LAN interface | Private subnet |
| WireGuard | Remote clients behave as LAN-local |
| BGP | Peered with Cilium for LoadBalancerIPPool advertisement |
| Port forwarding | Service-specific rules only; no shared public Gateway |
| Relay egress | Outbound WireGuard to OCI from the elected K3s node |

### 5.4 IPv6 Policy

IPv6 is disabled for north-south traffic at the network boundary; the environment operates IPv4-only.

### 5.5 Split-Horizon DNS

To avoid WAN hairpinning, selected public service hostnames resolve differently for internal and external clients.

- **External clients** resolve public hostnames according to public DNS and reach services through Tier O, Tier R, or Tier S, depending on the service.
- **LAN and WireGuard clients** resolve selected public service hostnames to private ingress addresses so the same services are reached through internal paths instead of via the ISP-facing WAN address.

This allows Internet-facing services to remain externally reachable while internal clients prefer private ingress paths.

A dedicated CoreDNS instance runs in-cluster, separate from the cluster-internal K3s CoreDNS, and is authoritative for the split-horizon zones. OPNsense Unbound forwards those zones to it over DNS-over-TLS. This keeps DNS overrides version-controlled and avoids click-ops in OPNsense.

______________________________________________________________________


## 6. Ingress Tiers

```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart LR
    subgraph external["External"]
        Internet
        CFEdge["Cloudflare Edge"]
    end

    subgraph oci["OCI"]
        Admission["Address Admission"]
        Relay["L4 Relay"]
        NLB["OCI NLB"]
    end

    subgraph cluster["K3s Cluster"]
        GW_R["Gateway R\n(Relay)"]
        LB_R["LoadBalancer R\n(L4-passed)"]
        Short["Short tunnel"]
        GW_S["Gateway S\n(Short)"]
        GW_I["Gateway I\n(Internal)"]
        LB_D["LoadBalancer D\n(Direct)"]
        SVC["Cluster Services"]
    end

    Internet -->|"O: Cloudflare Tunnel"| CFEdge --> SVC
    Internet -->|"R: declared TCP ports"| Admission --> Relay
    Relay -->|"80/443"| GW_R --> SVC
    Relay -->|"L4-passed service"| LB_R --> SVC
    Internet -->|"S"| NLB
    NLB -->|"Site-to-Site VPN"| Short --> GW_S --> SVC
    LAN["LAN / WireGuard"] -->|"I: Private CIDR"| GW_I --> SVC
    LAN["LAN / WireGuard"] -->|"D: Dedicated IP"| LB_D --> SVC
    SVC -->|"C: ClusterIP only"| SVC
```

| Tier | Label | Path | TLS Termination | Certificate Authority | Auth Requirement |
| --- | --- | --- | --- | --- | --- |
| **H** | Hypertext | Cloudflare Pages; no cluster ingress | Cloudflare Edge | Cloudflare | Anonymous |
| **O** | Orange Cloud | Cloudflare Tunnel → Service | Cloudflare Edge | Cloudflare | Cloudflare-managed |
| **R** | Relay | OCI L4 relay → WireGuard → Gateway R → Service | Gateway R or L4-passed service | Let’s Encrypt (DNS-01) | Required per service |
| **S** | Short | OCI NLB → Site-to-Site VPN → Gateway S | Gateway S or none | Let’s Encrypt (DNS-01) | Service-native |
| **I** | Internal | LAN / WireGuard → Gateway I → Service | Gateway I | Vault PKI | Network-level (LAN/WG) |
| **D** | Direct | LAN / WireGuard → LoadBalancer IP → Service | Service-native (or none) | Service-native | Service-native |
| **C** | Cluster Internal | ClusterIP only | Service-specific (or none) | Service-specific | Cilium network policy |

The [OCI L4 Relay Specification](oci-l4-relay-spec.md) is authoritative for the Tier R design. The [Tier S model](tier-s/) is authoritative for the Tier S design.

### 6.1 Gateway IP Allocation

| Gateway | CiliumLoadBalancerIPPool | Advertised via |
| --- | --- | --- |
| Gateway R | Dedicated private CIDR pool | Routed over the OCI WireGuard path |
| Gateway S | ClusterIP; no dedicated pool | Reached through the Site-to-Site VPN |
| Gateway I | Dedicated private CIDR pool | BGP → LAN |
| Tier D | Dedicated private CIDR pool | BGP → LAN |

Known and reserved LoadBalancer addresses:

| Address | Owner |
| --- | --- |
| 10.90.141.119 | Gateway I |
| 10.104.40.127 | Gateway R |
| 10.190.165.53 | Forgejo SSH |
| 10.28.147.227 | Network DNS / CoreDNS |
| 172.28.129.182 | Zot direct registry endpoint |
| 10.108.252.250 | Quay direct registry endpoint |
| 10.66.127.81 | VersityGW S3 API direct endpoint |
| 10.59.91.218 | VersityGW WebUI direct endpoint |
| 10.69.181.158 | Vault active API direct endpoint |
| 10.47.201.154 | Public Forgejo SSH |
| 10.66.153.202 | Stalwart mail and account HTTPS |
| 10.80.158.106 | Short VPN endpoint |
| 10.157.251.228 | PostgreSQL read-only service |
| 10.234.158.153 | PostgreSQL read/write service |

______________________________________________________________________


## 7. Workload Inventory

| Workload | Purpose | Tier | Replicas | ZFS Node | Storage Backend |
| --- | --- | --- | --- | --- | --- |
| VictoriaMetrics | Metrics (time-series) | I | split | — | OpenEBS LocalPV Hostpath |
| VictoriaLogs | Logs | I | split | — | Longhorn (replica=2) |
| Grafana | Metrics visualization, alerting | I | ≥2 | — | — |
| VersityGW | S3-compatible object gateway | I | =1 | Yes | OpenEBS LocalPV ZFS |
| VersityGW WebGUI | Administrative interface for VersityGW | I | =1 | — | — |
| CloudNativePG | Managed PostgreSQL | D | =3 | — | OpenEBS LocalPV Hostpath |
| Internal Forgejo | Private Git hosting and Flux GitOps origin | I / D | ≥2 | — | Longhorn RWX, VersityGW (S3), Dragonfly, CloudNativePG |
| Public Forgejo | Public source development and release distribution | O / R / D | ≥2 per root | — | Longhorn RWX, VersityGW (S3), Dragonfly, CloudNativePG |
| FluxCD | GitOps controller | I | ≥2 | — | — |
| Argo Workflows | CI/CD, workflow automation, and curated image mirror pipelines | I | ≥2 | — | VersityGW (S3, artifacts), CloudNativePG |
| OLM | Operator Lifecycle Manager (default catalogs disabled) | C | ≥2 | — | — |
| Quay | Container registry for curated internal images and scanned mirrors | D | ≥2 | — | VersityGW (S3), CloudNativePG |
| Clair | Image vulnerability scanner for Quay | C | ≥2 | — | CloudNativePG |
| Zitadel | Identity provider (OIDC) | R | ≥2 | — | CloudNativePG |
| Cilium | CNI, kube-proxy replacement, BGP, network policy, host firewall | C | DaemonSet | — | — |
| Hubble | Cilium network observability (relay + UI) | I | ≥2 | — | — |
| Longhorn | Replicated block storage | I | DaemonSet | — | Distributed node disks |
| OpenEBS LocalPV ZFS | Local ZFS-backed PVs | C | DaemonSet | Yes | ZFS (2× Z2 vdevs) |
| Dragonfly | In-memory cache (Redis-compatible) | C | =2 each | — | — (memory); dedicated CR per consumer |
| cert-manager | Certificate lifecycle automation | C | ≥2 | — | — |
| Vault | Secrets management, internal CA | I | =3 (Raft) | — | OpenEBS LocalPV Hostpath |
| oauth2-proxy | Auth proxy for selected web UIs | R / I | ≥2 | — | — |
| Vault Secrets Operator | Sync Vault secrets → K8s Secrets | C | ≥2 | — | — |
| Stalwart | Authoritative mail server and alert-notification sink | I / R | ≥2 | — | CloudNativePG, VersityGW (S3), Dragonfly |
| Frigate | Home security NVR (1×1080p, 7d retention) | I | =1 | — | Longhorn |

**Victoria components:** durable storage components run as one replica while the cluster has one ZFS node. Stateless insert/select components may run with multiple replicas. Collection agents are not independent workloads.

Stalwart's service contract is defined by the [Authoritative Email Server Specification](authoritative-email-server-spec.md).

### 7.1 Forgejo

Forgejo implements separate internal and public repository authorities. The authority, ingress, state, runtime, feature, and verification contracts are in the [Forgejo Deployment Specification](forgejo-deployment-spec.md).

### 7.2 Container Image Supply Chain

Quay native proxy cache is intentionally disabled. Cold pull-through requires Quay-side organization membership and does not satisfy the desired declarative access model.

Images intended for cluster consumption should be mirrored into Quay from a Git-owned allowlist, scanned by Clair, and promoted only after policy checks pass. Argo Workflows is the planned automation point for this mirror/scan/promote flow. Zot remains available as a lightweight direct registry for bootstrap and locally built infrastructure images.

______________________________________________________________________


## 8. Storage Architecture

The cluster uses three complementary storage layers, each serving a distinct purpose: replicated block storage for lightweight stateful workloads, high-capacity local storage for bulk data, and S3-compatible object storage for service backends.

### 8.1 Storage Layers

**Longhorn** provides replicated block storage across the K3s nodes. It is used for workloads that need persistent volumes but do not have large capacity requirements.

**OpenEBS LocalPV Hostpath** exposes node-local host paths as PersistentVolumes. It is used for workloads that implement their own application-level replication or clustering.

**OpenEBS LocalPV ZFS** exposes local ZFS datasets as PersistentVolumes. It is constrained to the ZFS node, which hosts 2× Z2 vdevs over 12×3 TB HDDs. This layer is used for workloads with large or IO-intensive storage needs.

**VersityGW** provides an S3-compatible object gateway running on the ZFS node. It exposes storage backed by the ZFS node via an S3 API for workloads that consume S3-compatible storage directly.

### 8.2 Consumer Mapping

| Storage Layer | Consumer | Longhorn Replicas | Notes |
| --- | --- | --- | --- |
| OpenEBS LocalPV Hostpath | Vault (Raft) | N/A | Vault handles its own replication via Raft across local host paths |
| OpenEBS LocalPV Hostpath | CloudNativePG | N/A | CNPG handles its own replication via streaming replication |
| OpenEBS LocalPV Hostpath | VictoriaMetrics | N/A | Multiple vmstorage replicas handle metrics redundancy across local host paths |
| Longhorn | VictoriaLogs | 2 | Highest-volume logs component |
| Longhorn | Frigate | Default | Single 1080p camera, 7-day retention |
| Longhorn RWX | Forgejo git/SSH | share-manager | Bare repos and built-in SSH keys; NFS semantics acceptable at homelab scale |
| VersityGW (S3) | Forgejo blobs | N/A | Avatars and attachments in bucket `forgejo` (port 7070 in-cluster) |
| Dragonfly | Forgejo cache | N/A | Dedicated `forgejo` CR; session, cache, and queue |
| CloudNativePG | Stalwart state | N/A | Durable account, metadata, queue, and search state |
| VersityGW (S3) | Stalwart blobs | N/A | Durable message bodies and attachments in bucket `stalwart` |
| Dragonfly | Stalwart cache | N/A | Dedicated non-durable Redis-compatible state |
| OpenEBS LocalPV ZFS | VersityGW | N/A | ZFS node only |

### 8.3 Design Principles

| Principle | Implementation |
| --- | --- |
| Prefer application-level replication where explicitly deployed | Vault Raft, CNPG streaming replication, and VictoriaMetrics clustering justify using non-replicated local storage (OpenEBS LocalPV Hostpath) instead of replicated block storage |
| Bulk data on ZFS | Large-capacity workloads are pinned to the ZFS node via node affinity |
| Object-storage backend | VersityGW provides S3-compatible storage for workloads that require it |
| Separation of concerns | Longhorn for general PVs; OpenEBS LocalPV Hostpath for high-performance clustered apps; OpenEBS LocalPV ZFS for bulk local storage; VersityGW for object storage |

______________________________________________________________________


## 9. Security Architecture

### 9.1 Trust Boundaries

```mermaid
%%{init: {'theme': 'neutral'}}%%
graph TD
    subgraph internet["Untrusted — Internet"]
        ExtClient["External Client"]
    end

    subgraph cf_edge["Semi-Trusted — Cloudflare"]
        CFTunnel["Cloudflare Tunnel + WAF"]
    end

    subgraph relay["Boundary — OCI Relay"]
        Admission["Address Admission + L4 Limits"]
        WGRelay["WireGuard Relay"]
    end

    subgraph boundary["Boundary — OPNsense"]
        WG["Remote-client WireGuard"]
    end

    subgraph lan["Trusted — LAN"]
        subgraph k3s["K3s Cluster"]
            CiliumFW["Cilium Host Firewall"]
            DefaultDeny["Global Default Deny"]
            GW_R2["Gateway R"]
            GW_I2["Gateway I"]
            LB_R2["LoadBalancer R"]
            Workloads["Workloads"]
        end
    end

    ExtClient -->|"TLS (CF-terminated)"| CFTunnel -->|"Tunnel"| Workloads
    ExtClient -->|"Tier R"| Admission --> WGRelay
    WGRelay -->|"HTTP/TLS"| GW_R2
    WGRelay -->|"L4 pass-through"| LB_R2 --> Workloads
    ExtClient -->|"WireGuard"| WG -->|"LAN access"| GW_I2
    GW_R2 -->|"Cilium policy"| Workloads
    GW_I2 -->|"Cilium policy"| Workloads
```

### 9.2 Certificate Authorities

| Scope | CA | Issuance | Managed By |
| --- | --- | --- | --- |
| Tier R (public-facing) | Let’s Encrypt | DNS-01 challenge (Cloudflare DNS) | cert-manager |
| Tier I (internal) | Vault PKI | Vault-issued | cert-manager + Vault |
| Tier O | Cloudflare | Cloudflare-managed | Cloudflare |

### 9.3 Authentication Flow (Tier R Web Services)

```mermaid
%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    actor User
    participant GW as Gateway R
    participant OAP as oauth2-proxy
    participant ZIT as Zitadel (IdP)
    participant SVC as Backend Service

    User->>GW: HTTPS request
    GW->>SVC: Route (if native OIDC)
    GW->>OAP: Route (if no native OIDC)
    OAP->>ZIT: OIDC redirect
    ZIT->>User: Login prompt
    User->>ZIT: Credentials
    ZIT->>OAP: ID token
    OAP->>SVC: Authenticated request
    SVC->>User: Response
```

| Service | Auth Method |
| --- | --- |
| Forgejo | See [deployment specification](forgejo-deployment-spec.md) |
| Quay | Native OIDC → Zitadel |
| Zitadel | Self |
| VersityGW & UI | IAM-style auth (S3 access keys) |
| Hubble UI | oauth2-proxy → Zitadel |
| Longhorn UI | oauth2-proxy → Zitadel |

### 9.4 Encryption at Rest

| Layer | Mechanism |
| --- | --- |
| Node root filesystem | LUKS |
| K3s secrets | K3s secrets encryption |
| Application secrets | Vault → Vault Secrets Operator → K8s Secrets |
| ZFS data | ZFS native AES-256-GCM encryption inherited from `********` |