246 lines
16 KiB
Markdown
246 lines
16 KiB
Markdown
|
# Infrastructure Repository Organization Standard
|
||
|
|
|
||
|
|
______________________________________________________________________
|
||
|
|
|
||
|
|
## 1. Executive Summary
|
||
|
|
|
||
|
|
This standard defines a clean, honest, and maintainable layout for an infrastructure monorepo that combines:
|
||
|
|
|
||
|
|
- GitOps (Flux + Kustomize)
|
||
|
|
- First-party applications
|
||
|
|
- Third-party platform services
|
||
|
|
- Custom build artifacts
|
||
|
|
- Terraform/OpenTofu declarative state
|
||
|
|
- Operational scripts and documentation
|
||
|
|
|
||
|
|
### Core Philosophy
|
||
|
|
|
||
|
|
> **Organize by ownership and lifecycle boundary first.**
|
||
|
|
|
||
|
|
- `vendor/` — Things we only maintain manifests for (third-party + simple internal tools). Further classified by failure-mode impact into `infrastructure/`, `platform/`, and `workloads/`.
|
||
|
|
- `services/` — Things we own source code + manifests for (first-party applications)
|
||
|
|
- `packages/` — Buildable artifacts we produce but do not run as normal services
|
||
|
|
- `terraform/` — Declarative state (external infrastructure + internal application configuration)
|
||
|
|
- `clusters/` — Thin Flux wiring layer only
|
||
|
|
|
||
|
|
This structure eliminates vague terms like "components", "platform", and "apps", and makes every directory's purpose obvious at a glance.
|
||
|
|
|
||
|
|
______________________________________________________________________
|
||
|
|
|
||
|
|
## 2. Final Recommended Structure
|
||
|
|
|
||
|
|
```text
|
||
|
|
.
|
||
|
|
├── clusters/ # Flux entrypoints (thin wiring only)
|
||
|
|
│ └── main/
|
||
|
|
│ ├── infrastructure.yaml # Points to ./vendor/infrastructure
|
||
|
|
│ ├── platform.yaml # Points to ./vendor/platform
|
||
|
|
│ ├── workloads.yaml # Points to ./vendor/workloads
|
||
|
|
│ └── services.yaml # Points to `k8s` subdirectories in ./services
|
||
|
|
│
|
||
|
|
├── vendor/ # Third-party manifests only
|
||
|
|
│ ├── infrastructure/ # Core cluster engines (total meltdown if down)
|
||
|
|
│ │ ├── cilium/
|
||
|
|
│ │ └── longhorn/
|
||
|
|
│ ├── platform/ # Shared services & registries (system paralysis if down)
|
||
|
|
│ │ ├── vault/
|
||
|
|
│ │ ├── zitadel/
|
||
|
|
│ │ └── forgejo/
|
||
|
|
│ └── workloads/ # Standalone user-facing apps (service disruption if down)
|
||
|
|
│ ├── forgejo-public/
|
||
|
|
│ └── canary/
|
||
|
|
│
|
||
|
|
├── services/ # Source code + manifests (first-party apps we develop)
|
||
|
|
│ └── my-gin-app/
|
||
|
|
│ ├── cmd/
|
||
|
|
│ ├── internal/
|
||
|
|
│ ├── Dockerfile
|
||
|
|
│ └── k8s/
|
||
|
|
│
|
||
|
|
├── packages/ # Buildable artifacts only (code/build context)
|
||
|
|
│ ├── versitygw-terraform-provider/
|
||
|
|
│ └── internal-cli/
|
||
|
|
│
|
||
|
|
├── terraform/ # Terraform/OpenTofu
|
||
|
|
│ ├── external/ # External dependencies (Firewall, GCP resources, etc)
|
||
|
|
│ └── internal/ # Application internal state (Zitadel orgs, policies, etc.)
|
||
|
|
│
|
||
|
|
├── scripts/ # Human-run operational scripts
|
||
|
|
└── docs/
|
||
|
|
├── spec/ # Normative design-of-record
|
||
|
|
└── journal/ # Symlink to development-journal/<repo> (gitignored)
|
||
|
|
```
|
||
|
|
|
||
|
|
______________________________________________________________________
|
||
|
|
|
||
|
|
## 3. Directory Responsibilities
|
||
|
|
|
||
|
|
| Directory | Purpose | Contains | Example |
|
||
|
|
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
|
||
|
|
| `clusters/` | Thin Flux reconciliation wiring | Kustomizations pointing at `vendor/` and `services/` | `clusters/main/platform.yaml` |
|
||
|
|
| `vendor/` | Third-party and simple internal services (manifests only). Organized by failure-mode impact: `infrastructure/` (total meltdown), `platform/` (system paralysis), `workloads/` (individual service disruption). | Kubernetes manifests only | `vendor/infrastructure/cilium/`, `vendor/platform/forgejo/`, `vendor/workloads/forgejo-public/` |
|
||
|
|
| `services/` | First-party applications we develop | Source code + Dockerfile + `k8s/` manifests | `services/my-gin-app/` |
|
||
|
|
| `packages/` | Buildable supporting artifacts | Source/build context only (no manifests) | `packages/versitygw-terraform-provider/` |
|
||
|
|
| `terraform/` | Declarative state | Split into `external/` and `internal/` | `terraform/external/`, `terraform/internal/` |
|
||
|
|
| `scripts/` | Human-run operational utilities | Debugging, migration, and one-off scripts | `scripts/analyze-hubble.py` |
|
||
|
|
| `docs/spec/` | Normative design-of-record | Specs, standards, settled runbooks | `docs/spec/homelab-spec.md` |
|
||
|
|
| `docs/journal/` | Time-sensitive work record (symlink; gitignored) | Investigations, plans, postmortems, trackers | `docs/journal/00_SUMMARY.md` |
|
||
|
|
|
||
|
|
______________________________________________________________________
|
||
|
|
|
||
|
|
## 4. Key Design Decisions
|
||
|
|
|
||
|
|
### 4.1 Why `vendor/` (with `infrastructure/`, `platform/`, `workloads/`)?
|
||
|
|
|
||
|
|
- "Platform" and "components" are vague and overloaded.
|
||
|
|
- `vendor/` clearly signals: **"We only maintain the manifests here."**
|
||
|
|
- Content is further classified by **failure mode**:
|
||
|
|
- `infrastructure/` — Total meltdown (nodes go `NotReady`).
|
||
|
|
- `platform/` — System paralysis (workloads crash-loop; CI/CD stops).
|
||
|
|
- `workloads/` — Individual service disruption (users cannot reach one app).
|
||
|
|
- This includes both true third-party services and simple internal tools that have no source code in this repo.
|
||
|
|
- **Internal Forgejo lives in `vendor/platform/`** because Flux depends on it for reconciliation and recovery. **Public Forgejo lives in `vendor/workloads/`** because its failure disrupts the public Git service but leaves cluster reconciliation and shared infrastructure healthy. Product identity does not override failure-mode classification.
|
||
|
|
|
||
|
|
### 4.2 Why `services/` for first-party?
|
||
|
|
|
||
|
|
- Clear contrast with `vendor/`.
|
||
|
|
- Explicitly means: **"We own the source code + deployment contract."**
|
||
|
|
- Preserves the important principle that code and manifests should live together for things we develop.
|
||
|
|
- _Note: To keep Flux reconciliation fast, ensure a `.sourceignore` file is defined at the repository root to prevent Flux from scanning non-Kubernetes files (like `.go`, `go.sum`, etc.) under `services/`._
|
||
|
|
|
||
|
|
### 4.3 Why separate `packages/`?
|
||
|
|
|
||
|
|
- `packages/` is strictly for **buildable artifacts** that are consumed by something else.
|
||
|
|
- No Kubernetes manifests belong here.
|
||
|
|
- Examples: custom Terraform providers, internal CLIs, code generators.
|
||
|
|
|
||
|
|
### 4.4 Terraform Split (`external/` vs `internal/`)
|
||
|
|
|
||
|
|
This addresses a common real-world need:
|
||
|
|
|
||
|
|
- `terraform/external/` — External cloud dependencies such as backup destinations on GCS.
|
||
|
|
- `terraform/internal/` — Application configuration state (Zitadel organizations, policies, grants, VersityGW buckets, Vault policies, etc.)
|
||
|
|
|
||
|
|
This split makes it obvious that Terraform is used for both traditional infrastructure **and** rich application configuration.
|
||
|
|
|
||
|
|
### 4.5 Failure-Mode Taxonomy
|
||
|
|
|
||
|
|
Vendor manifests are classified by what happens when they fail.
|
||
|
|
|
||
|
|
| Sub-tier | Failure Mode | Impact | Examples |
|
||
|
|
| ----------------- | ---------------------- | ----------------------------------------------------------------- | ------------------------------------- |
|
||
|
|
| `infrastructure/` | **Total Meltdown** | Nodes go `NotReady`; control plane loses quorum; cluster unusable | Cilium, Longhorn, OpenEBS |
|
||
|
|
| `platform/` | **System Paralysis** | Workloads crash-loop; CI/CD stops; source or secret access fails | Vault, Zitadel, internal Forgejo, VSO |
|
||
|
|
| `workloads/` | **Service Disruption** | Individual user-facing services offline; platform remains healthy | oCIS, Frigate, public Forgejo, canary |
|
||
|
|
|
||
|
|
______________________________________________________________________
|
||
|
|
|
||
|
|
## 5. Placement Rules
|
||
|
|
|
||
|
|
| Asset Type | Location | Notes |
|
||
|
|
| -------------------------------------------- | ---------------------------------------- | ---------------------------------------- |
|
||
|
|
| Flux Kustomization for infrastructure | `clusters/main/infrastructure.yaml` | Thin wiring only |
|
||
|
|
| Flux Kustomization for platform | `clusters/main/platform.yaml` | Thin wiring only |
|
||
|
|
| Flux Kustomization for workloads | `clusters/main/workloads.yaml` | Thin wiring only |
|
||
|
|
| Flux Kustomization for first-party | `clusters/main/services.yaml` | Thin wiring only |
|
||
|
|
| Infrastructure Kubernetes manifests | `vendor/infrastructure/<name>/` | Total meltdown scope |
|
||
|
|
| Platform Kubernetes manifests | `vendor/platform/<name>/` | System paralysis scope |
|
||
|
|
| Workload Kubernetes manifests | `vendor/workloads/<name>/` | Service disruption scope |
|
||
|
|
| First-party application (source + manifests) | `services/<name>/` | Code + `k8s/` together |
|
||
|
|
| Custom Terraform provider source | `packages/versitygw-terraform-provider/` | Build artifact |
|
||
|
|
| Supporting build artifacts | `packages/<name>/` | Build artifact |
|
||
|
|
| External infrastructure state | `terraform/external/<boundary>/` | GCP resources |
|
||
|
|
| Application internal configuration state | `terraform/internal/<name>/` | Zitadel orgs, policies, etc. |
|
||
|
|
| Human-run scripts | `scripts/` | Never required for normal reconciliation |
|
||
|
|
| Documentation | `docs/spec/`, `docs/journal/` | Specs vs historical work records |
|
||
|
|
|
||
|
|
______________________________________________________________________
|
||
|
|
|
||
|
|
## 6. Application Configuration vs Workload Definition (Important)
|
||
|
|
|
||
|
|
Terraform is frequently used to manage **application internal state**, not just infrastructure.
|
||
|
|
|
||
|
|
**Example — Zitadel:**
|
||
|
|
|
||
|
|
```text
|
||
|
|
vendor/platform/zitadel/ # Flux-managed workload
|
||
|
|
├── helm-release.yaml # replicas, image, resources
|
||
|
|
├── network-policies.yaml
|
||
|
|
└── route.yaml
|
||
|
|
|
||
|
|
terraform/internal/zitadel/ # Terraform-managed configuration
|
||
|
|
├── organizations.tf
|
||
|
|
├── policies.tf
|
||
|
|
├── grants.tf
|
||
|
|
└── providers.tf
|
||
|
|
```
|
||
|
|
|
||
|
|
**Rule:**
|
||
|
|
|
||
|
|
- **Flux** manages the _workload_ (replicas, image, probes, network policies).
|
||
|
|
- **Terraform** manages the _application configuration_ (organizations, users, policies, grants).
|
||
|
|
|
||
|
|
______________________________________________________________________
|
||
|
|
|
||
|
|
## 7. Decision Tree
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
flowchart TD
|
||
|
|
Start[New thing to add] --> Q1{Do we only maintain Kubernetes manifests?}
|
||
|
|
Q1 -->|Yes| Q_V{What is the failure mode if it goes down?}
|
||
|
|
Q_V -->|Nodes go NotReady / Total meltdown| VI[vendor/infrastructure/]
|
||
|
|
Q_V -->|Cluster paralyzed / Cascading failure| VP[vendor/platform/]
|
||
|
|
Q_V -->|Individual service disruption only| VW[vendor/workloads/]
|
||
|
|
Q1 -->|No| Q2{Do we maintain source code + manifests?}
|
||
|
|
Q2 -->|Yes| S[services/]
|
||
|
|
Q2 -->|No| Q3{Is it a buildable artifact consumed by something else?}
|
||
|
|
Q3 -->|Yes| P[packages/]
|
||
|
|
Q3 -->|No| Q4{Is it Terraform/OpenTofu declarative state?}
|
||
|
|
Q4 -->|Yes| T[terraform/]
|
||
|
|
Q4 -->|No| Q5{Is it a human-run script?}
|
||
|
|
Q5 -->|Yes| Scripts[scripts/]
|
||
|
|
Q5 -->|No| Docs[docs/]
|
||
|
|
```
|
||
|
|
|
||
|
|
______________________________________________________________________
|
||
|
|
|
||
|
|
## 8. Anti-Patterns
|
||
|
|
|
||
|
|
- Putting source code in `vendor/`
|
||
|
|
- Putting Kubernetes manifests in `packages/`
|
||
|
|
- Mixing first-party source with third-party manifests in the same directory
|
||
|
|
- Repeating a component suffix across sibling filenames instead of placing those files in a component directory
|
||
|
|
- Using `terraform/` for Kubernetes resource definitions (use Flux instead)
|
||
|
|
- Putting required production logic in `scripts/`
|
||
|
|
- **Classifying by product name instead of failure mode** (e.g., placing both Forgejo authorities in one tier despite different recovery dependencies)
|
||
|
|
- Renaming live Flux `Kustomization` objects or transferring resource ownership in the same prune-enabled change as a path refactor
|
||
|
|
|
||
|
|
## 9. Migration Safety
|
||
|
|
|
||
|
|
Repository layout changes must preserve Flux inventory ownership unless there is an explicit, tested adoption plan.
|
||
|
|
|
||
|
|
- Preserve existing Flux `Kustomization` `metadata.name` values during path moves.
|
||
|
|
- Move paths first, then verify reconciliation and inventory before renaming owners.
|
||
|
|
- Do not move directly root-owned resources under a new child Kustomization in the same change.
|
||
|
|
- Critical bootstrap resources should use non-pruned Kustomizations unless a deletion procedure has been tested.
|
||
|
|
|
||
|
|
______________________________________________________________________
|
||
|
|
|
||
|
|
## 10. Quick Reference
|
||
|
|
|
||
|
|
| I want to add... | Put it in... |
|
||
|
|
| ------------------------------------------------------- | ------------------------------- |
|
||
|
|
| New infrastructure service (CNI, storage, cert-manager) | `vendor/infrastructure/<name>/` |
|
||
|
|
| New platform service (IdP, secrets, source) | `vendor/platform/<name>/` |
|
||
|
|
| New third-party user-facing app | `vendor/workloads/<name>/` |
|
||
|
|
| New first-party application | `services/<name>/` |
|
||
|
|
| Custom Terraform provider | `packages/<name>/` |
|
||
|
|
| Supporting build artifact | `packages/<name>/` |
|
||
|
|
| External infrastructure | `terraform/external/<name>/` |
|
||
|
|
| Application config state (orgs, policies) | `terraform/internal/<name>/` |
|
||
|
|
| Operational script | `scripts/` |
|
||
|
|
| Documentation | `docs/spec/`, `docs/journal/` |
|
||
|
|
|
||
|
|
______________________________________________________________________
|
||
|
|
|
||
|
|
_Last updated: June 26, 2026_
|