# Infrastructure Repository Organization Standard ______________________________________________________________________ ## 1. Executive Summary This standard defines a clean, honest, and maintainable layout for an infrastructure monorepo that combines: - GitOps (Flux + Kustomize) - First-party applications - Third-party platform services - Custom build artifacts - Terraform/OpenTofu declarative state - Operational scripts and documentation ### Core Philosophy > **Organize by ownership and lifecycle boundary first.** - `vendor/` — Things we only maintain manifests for (third-party + simple internal tools). Further classified by failure-mode impact into `infrastructure/`, `platform/`, and `workloads/`. - `services/` — Things we own source code + manifests for (first-party applications) - `packages/` — Buildable artifacts we produce but do not run as normal services - `terraform/` — Declarative state (external infrastructure + internal application configuration) - `clusters/` — Thin Flux wiring layer only This structure eliminates vague terms like "components", "platform", and "apps", and makes every directory's purpose obvious at a glance. ______________________________________________________________________ ## 2. Final Recommended Structure ```text . ├── clusters/ # Flux entrypoints (thin wiring only) │ └── main/ │ ├── infrastructure.yaml # Points to ./vendor/infrastructure │ ├── platform.yaml # Points to ./vendor/platform │ ├── workloads.yaml # Points to ./vendor/workloads │ └── services.yaml # Points to `k8s` subdirectories in ./services │ ├── vendor/ # Third-party manifests only │ ├── infrastructure/ # Core cluster engines (total meltdown if down) │ │ ├── cilium/ │ │ └── longhorn/ │ ├── platform/ # Shared services & registries (system paralysis if down) │ │ ├── vault/ │ │ ├── zitadel/ │ │ └── forgejo/ │ └── workloads/ # Standalone user-facing apps (service disruption if down) │ ├── forgejo-public/ │ └── canary/ │ ├── services/ # Source code + manifests (first-party apps we develop) │ └── my-gin-app/ │ ├── cmd/ │ ├── internal/ │ ├── Dockerfile │ └── k8s/ │ ├── packages/ # Buildable artifacts only (code/build context) │ ├── versitygw-terraform-provider/ │ └── internal-cli/ │ ├── terraform/ # Terraform/OpenTofu │ ├── external/ # External dependencies (Firewall, GCP resources, etc) │ └── internal/ # Application internal state (Zitadel orgs, policies, etc.) │ ├── scripts/ # Human-run operational scripts └── docs/ ├── spec/ # Normative design-of-record └── journal/ # Symlink to development-journal/ (gitignored) ``` ______________________________________________________________________ ## 3. Directory Responsibilities | Directory | Purpose | Contains | Example | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | `clusters/` | Thin Flux reconciliation wiring | Kustomizations pointing at `vendor/` and `services/` | `clusters/main/platform.yaml` | | `vendor/` | Third-party and simple internal services (manifests only). Organized by failure-mode impact: `infrastructure/` (total meltdown), `platform/` (system paralysis), `workloads/` (individual service disruption). | Kubernetes manifests only | `vendor/infrastructure/cilium/`, `vendor/platform/forgejo/`, `vendor/workloads/forgejo-public/` | | `services/` | First-party applications we develop | Source code + Dockerfile + `k8s/` manifests | `services/my-gin-app/` | | `packages/` | Buildable supporting artifacts | Source/build context only (no manifests) | `packages/versitygw-terraform-provider/` | | `terraform/` | Declarative state | Split into `external/` and `internal/` | `terraform/external/`, `terraform/internal/` | | `scripts/` | Human-run operational utilities | Debugging, migration, and one-off scripts | `scripts/analyze-hubble.py` | | `docs/spec/` | Normative design-of-record | Specs, standards, settled runbooks | `docs/spec/homelab-spec.md` | | `docs/journal/` | Time-sensitive work record (symlink; gitignored) | Investigations, plans, postmortems, trackers | `docs/journal/00_SUMMARY.md` | ______________________________________________________________________ ## 4. Key Design Decisions ### 4.1 Why `vendor/` (with `infrastructure/`, `platform/`, `workloads/`)? - "Platform" and "components" are vague and overloaded. - `vendor/` clearly signals: **"We only maintain the manifests here."** - Content is further classified by **failure mode**: - `infrastructure/` — Total meltdown (nodes go `NotReady`). - `platform/` — System paralysis (workloads crash-loop; CI/CD stops). - `workloads/` — Individual service disruption (users cannot reach one app). - This includes both true third-party services and simple internal tools that have no source code in this repo. - **Internal Forgejo lives in `vendor/platform/`** because Flux depends on it for reconciliation and recovery. **Public Forgejo lives in `vendor/workloads/`** because its failure disrupts the public Git service but leaves cluster reconciliation and shared infrastructure healthy. Product identity does not override failure-mode classification. ### 4.2 Why `services/` for first-party? - Clear contrast with `vendor/`. - Explicitly means: **"We own the source code + deployment contract."** - Preserves the important principle that code and manifests should live together for things we develop. - _Note: To keep Flux reconciliation fast, ensure a `.sourceignore` file is defined at the repository root to prevent Flux from scanning non-Kubernetes files (like `.go`, `go.sum`, etc.) under `services/`._ ### 4.3 Why separate `packages/`? - `packages/` is strictly for **buildable artifacts** that are consumed by something else. - No Kubernetes manifests belong here. - Examples: custom Terraform providers, internal CLIs, code generators. ### 4.4 Terraform Split (`external/` vs `internal/`) This addresses a common real-world need: - `terraform/external/` — External cloud dependencies such as backup destinations on GCS. - `terraform/internal/` — Application configuration state (Zitadel organizations, policies, grants, VersityGW buckets, Vault policies, etc.) This split makes it obvious that Terraform is used for both traditional infrastructure **and** rich application configuration. ### 4.5 Failure-Mode Taxonomy Vendor manifests are classified by what happens when they fail. | Sub-tier | Failure Mode | Impact | Examples | | ----------------- | ---------------------- | ----------------------------------------------------------------- | ------------------------------------- | | `infrastructure/` | **Total Meltdown** | Nodes go `NotReady`; control plane loses quorum; cluster unusable | Cilium, Longhorn, OpenEBS | | `platform/` | **System Paralysis** | Workloads crash-loop; CI/CD stops; source or secret access fails | Vault, Zitadel, internal Forgejo, VSO | | `workloads/` | **Service Disruption** | Individual user-facing services offline; platform remains healthy | oCIS, Frigate, public Forgejo, canary | ______________________________________________________________________ ## 5. Placement Rules | Asset Type | Location | Notes | | -------------------------------------------- | ---------------------------------------- | ---------------------------------------- | | Flux Kustomization for infrastructure | `clusters/main/infrastructure.yaml` | Thin wiring only | | Flux Kustomization for platform | `clusters/main/platform.yaml` | Thin wiring only | | Flux Kustomization for workloads | `clusters/main/workloads.yaml` | Thin wiring only | | Flux Kustomization for first-party | `clusters/main/services.yaml` | Thin wiring only | | Infrastructure Kubernetes manifests | `vendor/infrastructure//` | Total meltdown scope | | Platform Kubernetes manifests | `vendor/platform//` | System paralysis scope | | Workload Kubernetes manifests | `vendor/workloads//` | Service disruption scope | | First-party application (source + manifests) | `services//` | Code + `k8s/` together | | Custom Terraform provider source | `packages/versitygw-terraform-provider/` | Build artifact | | Supporting build artifacts | `packages//` | Build artifact | | External infrastructure state | `terraform/external//` | GCP resources | | Application internal configuration state | `terraform/internal//` | Zitadel orgs, policies, etc. | | Human-run scripts | `scripts/` | Never required for normal reconciliation | | Documentation | `docs/spec/`, `docs/journal/` | Specs vs historical work records | ______________________________________________________________________ ## 6. Application Configuration vs Workload Definition (Important) Terraform is frequently used to manage **application internal state**, not just infrastructure. **Example — Zitadel:** ```text vendor/platform/zitadel/ # Flux-managed workload ├── helm-release.yaml # replicas, image, resources ├── network-policies.yaml └── route.yaml terraform/internal/zitadel/ # Terraform-managed configuration ├── organizations.tf ├── policies.tf ├── grants.tf └── providers.tf ``` **Rule:** - **Flux** manages the _workload_ (replicas, image, probes, network policies). - **Terraform** manages the _application configuration_ (organizations, users, policies, grants). ______________________________________________________________________ ## 7. Decision Tree ```mermaid flowchart TD Start[New thing to add] --> Q1{Do we only maintain Kubernetes manifests?} Q1 -->|Yes| Q_V{What is the failure mode if it goes down?} Q_V -->|Nodes go NotReady / Total meltdown| VI[vendor/infrastructure/] Q_V -->|Cluster paralyzed / Cascading failure| VP[vendor/platform/] Q_V -->|Individual service disruption only| VW[vendor/workloads/] Q1 -->|No| Q2{Do we maintain source code + manifests?} Q2 -->|Yes| S[services/] Q2 -->|No| Q3{Is it a buildable artifact consumed by something else?} Q3 -->|Yes| P[packages/] Q3 -->|No| Q4{Is it Terraform/OpenTofu declarative state?} Q4 -->|Yes| T[terraform/] Q4 -->|No| Q5{Is it a human-run script?} Q5 -->|Yes| Scripts[scripts/] Q5 -->|No| Docs[docs/] ``` ______________________________________________________________________ ## 8. Anti-Patterns - Putting source code in `vendor/` - Putting Kubernetes manifests in `packages/` - Mixing first-party source with third-party manifests in the same directory - Repeating a component suffix across sibling filenames instead of placing those files in a component directory - Using `terraform/` for Kubernetes resource definitions (use Flux instead) - Putting required production logic in `scripts/` - **Classifying by product name instead of failure mode** (e.g., placing both Forgejo authorities in one tier despite different recovery dependencies) - Renaming live Flux `Kustomization` objects or transferring resource ownership in the same prune-enabled change as a path refactor ## 9. Migration Safety Repository layout changes must preserve Flux inventory ownership unless there is an explicit, tested adoption plan. - Preserve existing Flux `Kustomization` `metadata.name` values during path moves. - Move paths first, then verify reconciliation and inventory before renaming owners. - Do not move directly root-owned resources under a new child Kustomization in the same change. - Critical bootstrap resources should use non-pruned Kustomizations unless a deletion procedure has been tested. ______________________________________________________________________ ## 10. Quick Reference | I want to add... | Put it in... | | ------------------------------------------------------- | ------------------------------- | | New infrastructure service (CNI, storage, cert-manager) | `vendor/infrastructure//` | | New platform service (IdP, secrets, source) | `vendor/platform//` | | New third-party user-facing app | `vendor/workloads//` | | New first-party application | `services//` | | Custom Terraform provider | `packages//` | | Supporting build artifact | `packages//` | | External infrastructure | `terraform/external//` | | Application config state (orgs, policies) | `terraform/internal//` | | Operational script | `scripts/` | | Documentation | `docs/spec/`, `docs/journal/` | ______________________________________________________________________ _Last updated: June 26, 2026_