ms2k/quorate
SHA256
Watch
0
Code Releases Activity

ce7cb702218693d08ea6d83464d2457c9250027c

Browse source
This commit is contained in:
ms2k 2026-11-12 20:46:28 +09:00
commit 34c5f8963c
45 changed files with 145336 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

177
docs/spec/authoritative-email-server-spec.md Normal file
View file

@ -0,0 +1,177 @@
# Authoritative Email Server Specification
## 1. Scope
Stalwart is the authoritative email server for `quorate.dev`. It keeps hosted delivery inside the cluster and submits external delivery through OCI Email Delivery.
This specification owns the Stalwart deployment, configuration, routing, storage, protocol, identity, and filtering boundaries.
## 2. Topology
```mermaid
flowchart LR
Ingress["Declared ingress surfaces"]
Stalwart["Stalwart\n≥2 replicas"]
Controller["Configuration controller\nexactly one owner"]
PG["PostgreSQL\ndurable state"]
S3["VersityGW\ndurable blobs"]
Cache["DragonflyDB\nnon-durable state"]
OCI["OCI Email Delivery"]
Hosted["Hosted mailbox"]
External["External recipient"]
Ingress --> Stalwart
Controller --> Stalwart
Stalwart --> PG
Stalwart --> S3
Stalwart --> Cache
Stalwart --> Hosted
Stalwart -->|"authenticated TLS"| OCI
OCI -->|"best-effort TLS"| External
```
## 3. Configuration Ownership
```mermaid
stateDiagram-v2
[*] --> Empty
Empty --> RecoveryAccess: one Job starts
RecoveryAccess --> RecoveryAccess: retry after interruption
RecoveryAccess --> Reconciled: controller applies ordinary desired state
Reconciled --> Ownerless: Job, controller, and recovery workload removed
Ownerless --> Steady: replicas and controller start
Steady --> Steady: continuous reconciliation
```
| Owner | Authority |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Git | All readable, non-secret desired state |
| Vault | Service credentials |
| Bootstrap Job | At most one one-shot Job; owns only temporary recovery access and waits for controller readiness |
| Controller | Sole owner of declared Stalwart configuration during initialization and steady operation, plus bounded certificate-ID derivation |
| Terraform | Dependency and external-service state |
| Flux | Bootstrap or normal workload state, never both |
The bootstrap Job contains no Stalwart desired-state plan and performs no configuration mutation. It provides the recovery-authenticated management endpoint while the controller applies the ordinary desired-state plan, then exits after the controller reports successful convergence. No Stalwart `Bootstrap` object is used. The recovery credential and recovery workload are absent before the steady controller starts, where the permanent controller identity is authenticated independently.
The deployment has the following minimum identities:
| Identity | Kind | Authority or use | Identity authority | Hosted mailbox |
| ------------------------- | -------------------- | ------------------------------------------------------------------- | ----------------------------- | -------------- |
| Recovery administrator | Temporary automation | Recovery-authenticated management during bootstrap only | — | None |
| Configuration controller | Automation | Declarative configuration reconciliation | Zitadel | None |
| Alert submitter | Automation | Authenticated cluster-local SMTP submission to the operator mailbox | Zitadel | None |
| Authenticated client test | Automation | TCP/465 and TCP/993 protocol qualification as `[email protected]` | Zitadel | Required |
| Human administrator | Human | Administration and operator mailbox membership | Zitadel | — |
| `[email protected]` | Shared mailbox | Operational alerts and Android shared-folder access | Zitadel-authenticated members | Required |
| `[email protected]` | Test mailbox | Authenticated client, inbound delivery, and routing acceptance | Authenticated client test | Required |
Zitadel provides the human-administrator role. The human administrator is a member of the operator shared mailbox.
Human identity remains Zitadel-owned. Mail clients that cannot use the Zitadel access token through `OAUTHBEARER` or `XOAUTH2` use a per-user Stalwart app password as a revocable protocol credential.
The controller:
| Field | Contract |
| ------------------ | ------------------------------------------------------------------------ |
| Ownership | PostgreSQL advisory lock; a competing owner performs no reconciliation |
| Interval | 5 minutes |
| Timeout | 2 minutes per reconciliation |
| Retry | Exponential backoff from 5 seconds to 5 minutes |
| Persistent failure | Unready after 5 consecutive failures while retries continue |
| Desired input | Git state plus runtime-projected secret values |
| Derived input | Server certificate ID resolved from its declared SAN |
| Secret rotation | Vault-owned service credentials converge after rotation |
| Pruning | Exhaustive declared configuration is pruned; accounts are upserted |
| Status | Liveness, readiness, last success, persistent failure, and plan revision |
Manual API patches, generated snapshot adoption, and live `kubectl` mutation are not deployment or configuration mechanisms. Break-glass mutation is limited to failure response and teardown.
| App-password frontend | Contract |
| --------------------- | --------------------------------------------------------------- |
| Implementation | Custom UI backed by Zitadel login and Stalwart app-password API |
| Scope | Signed-in user's app passwords |
| Endpoint | `mail.quorate.dev/account/` |
| Ingress | Gateway R and direct Stalwart frontend |
| TLS | Gateway R certificate on both ingress paths |
| Root redirect | HTTPS `/` → `/account/` |
## 4. Protocol Boundary
| Surface | Required listener availability | TLS | Authentication |
| --------------------------------------------- | -------------------------------- | ----------------------------- | -------------------------------------- |
| SMTP TCP/25 | Public-ingress target | Opportunistic STARTTLS | AUTH prohibited |
| Submissions TCP/465 | Public-ingress target | Implicit TLS | Zitadel token or per-user app password |
| Submission TCP/587 | Cluster-local | Cilium transparent encryption | Vault-backed automation authentication |
| IMAPS TCP/993 | Public-ingress target | Implicit TLS | Zitadel token or per-user app password |
| JMAP and user HTTPS TCP/443 | Public-ingress and direct target | Required | Zitadel-backed human authentication |
| Administration API at `mail.quorate.internal` | Internal-ingress target | Required outside the cluster | Zitadel-backed human authentication |
## 5. Routing
```mermaid
flowchart LR
Copy["Accepted recipient copy"]
Classify{"In Stalwart's\nhosted-domain set?"}
Local["Local delivery\nPostgreSQL + VersityGW"]
Explicit{"Explicit external\nrecipient or disclosure?"}
Relay["OCI Email Delivery"]
Reject["Reject"]
Copy --> Classify
Classify -->|yes| Local
Classify -->|no| Explicit
Explicit -->|yes| Relay
Explicit -->|no| Reject
```
| Condition | Result |
| -------------------------------------- | -------------------------------------------------------------------------------------------- |
| Hosted account, alias, or mailbox | Deliver or queue entirely inside the cluster |
| Unknown hosted recipient | Reject during SMTP without a later bounce |
| External authenticated recipient | Submit through OCI Email Delivery |
| Mixed recipients | Keep the hosted copy local; submit only the external copy |
| Explicit external forwarding or report | Treat as an external disclosure and minimize quoted original content |
| OCI failure | Keep hosted delivery available and queue external delivery |
| Local-store failure | Queue locally or return `451`; never reclassify as external |
| SMTP pressure | Apply bounded session and concurrency controls; return retryable `451` where SMTP permits it |
| Routing loop | Never convert a hosted destination into external delivery |
Hosted-domain classification comes from Stalwart's authoritative domain set, not suffix matching or a duplicated routing list. Authenticated cluster-local submission to hosted recipients never contacts OCI.
## 6. Storage and Credentials
| Component | Role |
| ----------- | -------------------------------------------------- |
| PostgreSQL | Durable account, metadata, queue, and search state |
| VersityGW | Durable message bodies and attachments |
| DragonflyDB | Disposable Redis-compatible state |
| Vault | Source authority for service credentials |
Stalwart reports SMTP success only after the required PostgreSQL state and VersityGW blob are durable. Dependency failure causes local queueing or a temporary failure without changing the recipient's routing class.
Coordinated PostgreSQL and VersityGW recovery is outside this specification.
## 7. Encryption and Filtering
The server-wide application-encryption capability is enabled.
| Identity | Application-layer encryption policy |
| ---------------------------------------------------- | ------------------------------------------------ |
| Recovery, controller, and alert-submitter identities | Not applicable; none has a mailbox |
| `[email protected]` | Disabled |
| External human account | Enabled by default; explicit per-account opt-out |
Human onboarding records the OpenPGP public key or S/MIME certificate, recovery implications, supported search boundary, spam-training consequences, and append-encryption choice. Delivery and append encryption are verified independently.
Encrypted and plaintext accounts coexist without changing each other's policy. Plaintext accounts retain body search. Encrypted accounts expose only supported header search. Plaintext spam training requires recorded consent.
Stalwart's built-in spam and phishing filter is the only v1 content filter. It runs before application-layer encryption for unauthenticated inbound mail. Authenticated alert submission bypasses inbound spam classification.
Resolver and filter-update concurrency, cache, and timeouts are bounded. Resolver failure does not reject all inbound mail. Filter-update failure keeps the last valid data or enters the declared bounded degraded behavior without opening general egress.
OpenPGP and S/MIME remain available when message content must stay opaque to OCI Email Delivery and subsequent relays.
## 8. Mail Identity
SPF, OCI DKIM, DMARC, the envelope sender, bounce handling, MTA-STS, and TLS reporting align with the `quorate.dev` authority.

272
docs/spec/forgejo-deployment-spec.md Normal file
View file

@ -0,0 +1,272 @@
# Forgejo Deployment Specification
## 1. Scope
This document specifies every Forgejo deployment in the cluster. Two repository authorities are implemented by three frontend groups.
| Authority | Frontend groups | Role |
| ---------------- | --------------------------------- | ------------------------------------------------------- |
| Internal Forgejo | One private root | Private development, operations, and Flux GitOps origin |
| Public Forgejo | Anonymous and authenticated roots | Public project development and release origin |
## 2. Authority and Topology
```mermaid
flowchart LR
Operators[Operators and Flux]
Internet[Internet]
Trusted[LAN or WireGuard]
Internal[Internal frontend]
Facade[Envoy anonymous facade]
Code[Public anonymous frontend]
Git[Public authenticated frontend]
InternalState[(Internal Forgejo state)]
PublicState[(Public Forgejo state)]
Operators --> Internal --> InternalState
Internet -->|code.quorate.dev| Facade --> Code --> PublicState
Internet -->|git.quorate.dev| Git --> PublicState
Trusted -->|git-ssh.quorate.internal| Git
InternalState -. sanitized quorate tip .-> PublicState
```
| Relationship | Contract |
| -------------------------------------- | --------------------------------------------------------- |
| Internal ↔ public authority | No shared application state or runtime credential |
| Public anonymous ↔ authenticated roots | Two roots share one public-forge application state |
| Repository ↔ authority | Exactly one authoritative forge per repository |
| Backup ↔ authority | Backups are recovery artifacts, not alternate Git remotes |
## 3. Common Deployment Contract
### 3.1 Best-Effort HA Profile
Forgejo does not provide a coordinated multi-replica HA mode. The deployment uses shared state and removes replica-local workers so interchangeable frontends can serve the same authority.
| Component | Durable HA contract |
| ------------------------------- | ---------------------------------------------------------------------------------- |
| Forgejo frontend group | Multiple replicas run on distinct eligible nodes |
| Anonymous Envoy facade | Multiple replicas run on distinct eligible nodes |
| Rollout | A replacement settles and becomes ready before an old replica is removed |
| Voluntary disruption | Disruption policy preserves a ready replica for each group |
| Insufficient placement capacity | Availability requirements take precedence over co-locating redundant replicas |
| Manifest-owned tuning | Replica counts, resource requests and limits, rollout thresholds, and probe timing |
| Shared state | Placement | HA purpose |
| ------------------------------------------- | -------------------------------------------------------------- | ----------------------------------------------------------------------- |
| Git repositories and SSH host material | One Longhorn RWX claim per authority | Every replica reads and writes the same Git namespace |
| Users, permissions, and repository metadata | Dedicated database and role in CloudNativePG | No replica-local metadata authority |
| Sessions, cache, and queue | Dedicated Dragonfly instance and ACL per authority | Shared sessions, cache invalidation, and queued work |
| Attachments and avatars | Dedicated private VersityGW bucket and principal per authority | No replica-local blob state |
| Public repository archives | Dedicated public-forge object prefix | Either public root can retrieve a generated archive |
| Runtime secrets | Authority-scoped Vault projections | Every replica in one authority receives compatible application material |
| Failure or rollout condition | Contract |
| ------------------------------- | ------------------------------------------------------------------------------------------------ |
| One frontend pod fails | Another ready replica continues serving that group |
| One public frontend group stops | The other group continues its permitted operations against shared state |
| Voluntary disruption | Disruption policy protects a ready replica; it does not protect against involuntary node failure |
| Missing spread capacity | Redundant replicas do not collapse onto one node merely to satisfy replica count |
| Rolling update | A replacement becomes settled and ready before an old replica is removed |
| Shared dependency failure | All frontends using that authority may fail together |
| In-flight queued operation loss | Automatic takeover is not guaranteed; a client or controller must be able to retry |
The HA claim covers redundant stateless frontends. It does not claim seamless failover for every in-flight operation or remove the availability boundaries of the RWX volume, PostgreSQL, Dragonfly, VersityGW, DNS, or ingress path.
### 3.2 Security Profile
| Control | Requirement |
| ------------------ | ------------------------------------------------------------------------------------------ |
| Process identity | Non-root user and group |
| Privilege | No privilege escalation; all capabilities dropped except an explicitly required capability |
| Root filesystem | Read-only |
| Database transport | PostgreSQL TLS required |
| Internal trust | Dedicated internal CA projection for private service TLS |
| Network | Default-deny Cilium policy with explicit ingress and dependency egress |
### 3.3 Common Feature Profile
| Area | Required state |
| -------------------- | -------------------------------------------------------------------------------- |
| Repository units | Code and releases enabled |
| Disabled units | Issues, pull requests, wikis, projects, packages, Actions, and external trackers |
| Repository migration | Disabled |
| Repository indexer | Disabled to avoid divergent replica-local indexes |
| Forgejo cron | Disabled to avoid independently scheduled duplicate work |
| Forgejo Actions | Disabled to keep execution in the external workflow plane |
| Native mirrors | Disabled to avoid replica-local workers and schedules |
| Metrics endpoint | Disabled |
Argo Workflows owns CI, release orchestration, scheduled reconciliation, and publication automation. Forgejo-native workers remain disabled to avoid duplicate execution across replicas.
## 4. Internal Authority
### 4.1 Access and Purpose
| Surface | Hostname | Tier | Use |
| ------- | ---------------------------- | ----- | ---------------------------------------- |
| HTTPS | `forge.quorate.internal` | I | Authenticated browser and API access |
| SSH | `forge-ssh.quorate.internal` | D / C | Operator Git and in-cluster Flux fetches |
| Property | Contract |
| -------------------- | -------------------------------------------------------------------------- |
| Repository classes | Private, operational, and GitOps repositories |
| Flux origin | Internal Forgejo is authoritative for the cluster configuration repository |
| Anonymous access | Sign-in required |
| Interactive identity | Invite-only Zitadel OIDC |
| SSH clients | Private CIDRs and explicitly admitted in-cluster identities |
| Internet exposure | None |
| Runtime | Default container runtime; not a Tier O or Tier R application backend |
### 4.2 State Boundary
| Dependency | Internal allocation |
| ---------------- | --------------------------------------------------- |
| PostgreSQL | Dedicated internal-forge database and role |
| Dragonfly | Dedicated internal-forge instance and ACL |
| Git repositories | Dedicated RWX claim |
| Object storage | Dedicated private bucket and principal |
| Vault | Dedicated application subtree and projected secrets |
| Zitadel | Dedicated confidential OIDC application |
## 5. Public Authority
### 5.1 Repository Authority
| Repository class | Sole authority | Publication contract |
| ------------------------------------ | ---------------- | ---------------------------------------------------- |
| Private and operational repositories | Internal Forgejo | Absent from public Forgejo |
| Inherently public projects | Public Forgejo | Full public history; developed and released in place |
| `quorate` | Internal Forgejo | Sanitized, history-free snapshot of the current tip |
| Transition | Requirement |
| ---------------------------- | ------------------------------------------------------------ |
| Internal → public | Review the complete history and current tree before transfer |
| Public → internal | Treat all previously public objects as irretractable |
| Routine cross-forge transfer | Only the `quorate` snapshot |
```mermaid
flowchart LR
Tree[Approved quorate export tree]
Orphan[Construct orphan commit]
Scan{Secret scan clean?}
Reject[Fail closed]
Publish[Force-push public branch]
Tree --> Orphan --> Scan
Scan -->|No| Reject
Scan -->|Yes| Publish
```
The published branch has a one-commit history, no internal ancestry, and no tags. Its public commit identity is not stable across exports.
### 5.2 Ingress Boundaries
| Property | Anonymous root | Authenticated root | Trusted SSH |
| ----------------------- | ------------------------------------------------- | ------------------------------------------------------- | -------------------------- |
| Hostname | `code.quorate.dev` | `git.quorate.dev` | `git-ssh.quorate.internal` |
| Tier | O | R | D |
| Termination | Cloudflare TLS | Gateway R TLS | SSH transport |
| Path | Cloudflare Tunnel → Envoy | OCI relay → WireGuard → Gateway R | Private LoadBalancer |
| Supported use | Browse, public API reads, downloads, clone, fetch | OIDC, authenticated API, push, releases | Git SSH |
| Secret-bearing workflow | None | OIDC codes, sessions, API tokens, HTTPS Git credentials | SSH keys and sessions |
| Frontend | Anonymous root through Envoy only | Authenticated root | Authenticated root |
| Internet publication | HTTPS | HTTPS | None |
Split-horizon DNS for `git.quorate.dev` may select the private Gateway R address, but preserves the hostname, certificate identity, cookie origin, and OIDC callback.
### 5.3 Anonymous Envoy Facade
| Boundary | Requirement |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Authorization | Default deny by normalized method and path |
| Path handling | Normalize paths, merge slashes, and reject ambiguous escaped separators before policy evaluation |
| Admitted operations | Only qualified anonymous browser, API, archive, release, and smart-HTTP fetch operations |
| Denied operations | Login, OIDC, account, administration, notifications, repository mutation, `git-receive-pack`, and unclassified methods |
| Request stripping | Credentials, cookies, proxy authentication, signature credentials, and trusted reverse-proxy identity headers |
| Response stripping | Session headers and authentication challenges |
| Query credentials | Rejected |
| Logging | No request paths or credential values |
| Network policy | Tunnel identity → facade only; facade → anonymous Forgejo HTTP only |
| Bypass | No alternate anonymous route to either Forgejo frontend |
The facade remains mandatory because distinct browser roots do not partition database-backed access tokens or HTTPS Git credentials shared by the public frontends.
### 5.4 Frontend and Session Isolation
| Property | Anonymous root | Authenticated root |
| ----------------------- | --------------------------- | --------------------------------------------------------- |
| Canonical root | `https://code.quorate.dev/` | `https://git.quorate.dev/` |
| Interactive login | Absent | Invite-only Zitadel OIDC |
| First login | Not applicable | Automatic provisioning for an authorized Zitadel identity |
| SSH listener | Disabled | Internal-only endpoint |
| Session cookie | Distinct, secure, host-only | Distinct, secure, host-only |
| Session-store keys | Distinct prefix | Distinct prefix |
| Generated configuration | Private ephemeral volume | Private ephemeral volume |
| Shared public state | Isolated per frontend |
| ---------------------------------- | ------------------------------------ |
| PostgreSQL data | Root URL and generated configuration |
| Git repository volume | Session cookie name and store prefix |
| Cache and queue | OIDC source and callbacks |
| Object storage | Exposed network path |
| Application cryptographic material | |
Generated configuration never resides on the shared repository volume. Each root emits its own hostname in canonical links, clone URLs, API pagination, redirects, releases, attachments, and archives.
### 5.5 Public Feature Profile
| Area | Required state |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Anonymous use | Repository and organization exploration, source, commits, branches, tags, releases, downloads, archives, feeds, sitemap, public API reads, HTTPS clone, and fetch |
| Maintainer use | OIDC, authenticated API, HTTPS push, trusted SSH, repository administration, and release publication |
| Repository creation | Administrator-created and public by default |
| Disabled creation paths | Regular-user creation, push-to-create, migration, and forks |
| Search | Bounded repository-local Git search; global code search disabled |
| Git and artifacts | HTTPS, source archives, and release attachments enabled; LFS and packages disabled |
| API | Public and authenticated API enabled; Swagger and query-string tokens disabled |
| Webhooks | Disabled unless restricted to an exact in-cluster Argo Events receiver by application and network policy |
A newly enabled feature remains denied on Tier O until its routes, methods, resource bounds, and credential behavior pass the facade acceptance suite.
### 5.6 Public State and Runtime
| Dependency | Public allocation | Isolation from internal Forgejo |
| ---------------- | ------------------------------------------------- | ------------------------------------------------------------ |
| PostgreSQL | Dedicated public-forge database and role | Database and role sharing prohibited |
| Dragonfly | Dedicated public-forge instance and ACL | Instance, ACL, and credential sharing prohibited |
| Git repositories | Dedicated RWX claim | Volume sharing prohibited |
| Object storage | Dedicated private bucket, principal, and prefixes | Bucket, prefix, principal, and credential sharing prohibited |
| Vault | Dedicated subtree and projected secrets | Runtime-secret sharing prohibited |
| Kubernetes | Dedicated namespace and service account | API authority is not inherited from internal Forgejo |
| Zitadel | Dedicated confidential OIDC application | Client and callback sharing prohibited |
| Runtime control | Requirement |
| ------------------------ | ------------------------------------------------------------------------------------- |
| Namespace classification | `security.quorate.internal/public-runtime: gvisor` |
| Runtime | Explicit `gvisor` RuntimeClass for Forgejo, facade, init, upgrade, and setup pods |
| Admission | Reject a public backend outside a labeled namespace and any non-gVisor pod inside one |
| Service accounts | No Kubernetes API token without a documented need |
| Git hooks | Disabled |
| Images | Reviewed immutable digests |
| Public storage bootstrap | Requirement |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------- |
| New RWX volume | Create data directories with ownership compatible with the rootless Forgejo process before either frontend starts |
| Generated configuration | Keep it off the shared volume; bootstrap creates storage structure only |
A public-forge compromise must not provide a data path or credential path into internal Forgejo.
## 6. Change and Verification Contract
| Change or event | Required verification before acceptance |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Any Forgejo upgrade | Effective configuration, startup, Git read/write paths, release paths, dependency access, and replica replacement |
| Internal ingress or identity change | Sign-in requirement, Zitadel login, Flux fetch, private SSH admission, and absence of Internet exposure |
| Forgejo or Envoy change affecting Tier O | Complete admitted method and route inventory; clean reproducible facade suite |
| Facade change | Anonymous positives; authenticated denials; credential stripping; malformed, encoded, and query bypass rejection |
| Public network-policy change | Tunnel-to-facade path succeeds; direct tunnel-to-Forgejo path fails |
| Public frontend change | Distinct roots, private configuration, host-only disjoint sessions, correct generated links, and cross-root visibility |
| Dependency or replica restart | Affected frontends recover without divergent state |
| New publication class | Repository-specific clone, fetch, archive, release, and cross-root update checks |
| `quorate` publication | One orphan snapshot, no tags, no internal ancestry, and no secret-scan findings |

452
docs/spec/homelab-spec.md Normal file
View file

@ -0,0 +1,452 @@
# Home Infrastructure Specification
## 1. Overview
This document specifies a small home infrastructure environment built around a multi-node K3s cluster, a dedicated north-south boundary, and three complementary storage layers. The design prioritizes declarative operations, clear trust boundaries, and pragmatic tradeoffs appropriate for a single-operator environment.
## 2. Terminology
This document uses a small number of context-specific terms:
| Term | Meaning |
| --- | --- |
| **Boundary** | The home network edge, implemented by OPNsense, where WAN egress, remote-client WireGuard, routing, and firewall policy are enforced. |
| **ZFS node** | The 7840HS K3s agent that hosts the cluster’s bulk HDD storage and all ZFS-backed workloads. |
| **Tier H / O / R / S / I / D / C** | Classifications defined in §6. They describe the path by which a service is reached. |
| **Storage-heavy workload** | A workload whose primary persistence depends on ZFS-backed storage and which is therefore pinned to the ZFS node via node affinity. |
______________________________________________________________________
## 3. Hardware Inventory
| Role | CPU | RAM | Boot Storage | Bulk Storage | OS | K3s Role |
| --- | --- | --- | --- | --- | --- | --- |
| Control Plane 1 | Intel Xeon E5-2697 v3 | 64 GB | 512 GB SATA SSD | 512 GB SATA SSD + 320 GB HDD | Debian (headless, LUKS) | Server |
| Control Plane 2 | Intel Core i7-7700HQ | 24 GB | 128 GB NVMe SSD | 1 TB HDD | Debian (headless, LUKS) | Server |
| Control Plane 3 | Rockchip RK3588 | 16 GB | 512 GB NVMe (PCIe 2.0 x1) | — | Armbian (headless, LUKS) | Server |
| Agent — ZFS node | AMD 7840HS | 48 GB | 1 TB SSD | 12×3 TB HDD (2× ZFS Z2 vdevs) | Debian (headless, LUKS) | Agent |
| Network Boundary | Intel Pentium Gold 8505 | 32 GB | 128 GB SSD | — | OPNsense | N/A |
### 3.1 Architecture Notes
- The control plane is a 3-node quorum spanning mixed `x86_64` and `arm64` systems.
- All K3s nodes boot from LUKS-encrypted root filesystems.
- The **ZFS node** is the only node with bulk HDD storage and is the scheduling target for all storage-heavy workloads.
- Multi-architecture container images are required for workloads that may schedule onto the RK3588.
### 3.2 Host Runtime Assumptions
K3s nodes are assumed to start from an already bootstrapped Linux host with hostname, LAN addressing, SSH access, time sync, and LUKS root encryption handled outside this specification. This section only documents host dependencies required by the cluster runtime and its deployed infrastructure.
All K3s nodes need the following host-level capabilities before joining the cluster:
- Cilium/K3s networking primitives: cgroups, overlay filesystems, VXLAN, WireGuard, eBPF, netfilter/conntrack kernel support, policy routing, and a mounted BPF filesystem.
- All K3s nodes require persistent `net.ipv4.conf.all.rp_filter=0` and `net.ipv4.conf.default.rp_filter=2` so OCI relay tunnel interfaces inherit loose reverse-path filtering.
- Longhorn storage prerequisites: iSCSI initiator userspace, NFS client support, and XFS formatting tools.
- Disk health prerequisites: SMART userspace (`smartmontools`) for `smartctl-exporter`.
- Swap disabled.
- K3s is installed without the bundled networking stack that Cilium replaces: flannel, kube-proxy, K3s network-policy, Traefik, and ServiceLB are disabled.
Debian/Armbian/Ubuntu runtime packages:
```bash
sudo apt install -y \
open-iscsi nfs-common xfsprogs smartmontools
```
RHEL/Fedora-family runtime packages:
```bash
sudo dnf install -y \
iscsi-initiator-utils nfs-utils xfsprogs smartmontools
```
Userspace iptables/nftables and WireGuard tools are useful for diagnostics and emergency cleanup, but they are not treated as required node runtime packages because Cilium is deployed with kube-proxy replacement, BPF masquerading, and WireGuard encryption managed by Cilium.
Longhorn expects `/var/lib/longhorn/` to live on node-local SSD/NVMe storage and to be available before Longhorn starts. Nodes that participate in Longhorn scheduling are tagged with the `ssd` disk selector.
The ZFS node has additional host requirements:
- ZFS userspace and kernel support installed.
- The pool `********` imported at boot.
- The dataset `********` present for OpenEBS LocalPV ZFS.
- The Kubernetes node labeled `has-zfs=true`.
______________________________________________________________________
## 4. Design Constraints and Tradeoffs
This environment is intentionally designed around pragmatic home-lab constraints rather than datacenter-style redundancy.
- The **ZFS node** is a deliberate single-node failure domain for all ZFS-backed workloads.
- Local ZFS redundancy mitigates disk failure, but not host failure.
- Replicating the bulk-storage tier on a second host is currently out of scope due to cost, particularly HDD pricing.
- **VersityGW** is intentionally coupled to the ZFS node and is therefore also a deliberate single-node availability tradeoff.
These are accepted architectural constraints, not accidental omissions.
______________________________________________________________________
## 5. Network Architecture
### 5.1 Topology
```mermaid
%%{init: {'theme': 'neutral'}}%%
graph TD
WAN["WAN (ISP Public IP)"]
CF["Cloudflare Edge"]
OCI["OCI L4 Relay\nReserved Public IP"]
NLB["OCI NLB"]
subgraph boundary["Home Boundary"]
OPN["OPNsense\nPentium Gold 8505"]
WG["Remote-client WireGuard"]
end
subgraph cluster["K3s Cluster (LAN)"]
subgraph cp["Control Plane"]
CP1["Xeon E5-2697 v3"]
CP2["Core i7-7700HQ"]
CP3["RK3588"]
end
subgraph agents["Agent"]
A1["7840HS (ZFS node)"]
end
CIL["Cilium CNI"]
GW_R["Gateway R\n(Relay Services)"]
LB_R["LoadBalancer R\n(L4-passed Services)"]
Short["Short tunnel"]
GW_S["Gateway S\n(Short Services)"]
SVC_S["Services"]
GW_I["Gateway I\n(Internal Services)"]
LB_D["LoadBalancer D\n(Direct Services)"]
end
WAN --> OPN
CF -->|Cloudflare Tunnel| CIL
WAN -->|Public relay traffic| OCI
OCI -->|"WireGuard: web"| GW_R
OCI -->|"WireGuard: mail"| LB_R
WAN -->|Public NLB traffic| NLB
NLB -->|"Site-to-Site VPN"| Short --> GW_S --> SVC_S
OPN <-->|BGP Peering| CIL
WG --> OPN
OPN -->|LAN route| GW_I
OPN -->|LAN route| LB_D
CIL --- GW_R
CIL --- LB_R
CIL --- GW_S
CIL --- GW_I
CIL --- LB_D
CIL --- CP1
CIL --- CP2
CIL --- CP3
CIL --- A1
```
### 5.2 Cilium Configuration
| Feature | Status |
| --- | --- |
| Kube-proxy replacement | Enabled |
| BGP peering with OPNsense | Enabled |
| Gateway API | Enabled |
| Global default deny | Enabled |
| Host firewall | Enabled |
| Pod traffic encryption | WireGuard |
| IPv6 | Disabled at boundary |
### 5.3 OPNsense Configuration
| Feature | Detail |
| --- | --- |
| WAN interface | ISP-assigned public IP |
| LAN interface | Private subnet |
| WireGuard | Remote clients behave as LAN-local |
| BGP | Peered with Cilium for LoadBalancerIPPool advertisement |
| Port forwarding | Service-specific rules only; no shared public Gateway |
| Relay egress | Outbound WireGuard to OCI from the elected K3s node |
### 5.4 IPv6 Policy
IPv6 is disabled for north-south traffic at the network boundary; the environment operates IPv4-only.
### 5.5 Split-Horizon DNS
To avoid WAN hairpinning, selected public service hostnames resolve differently for internal and external clients.
- **External clients** resolve public hostnames according to public DNS and reach services through Tier O, Tier R, or Tier S, depending on the service.
- **LAN and WireGuard clients** resolve selected public service hostnames to private ingress addresses so the same services are reached through internal paths instead of via the ISP-facing WAN address.
This allows Internet-facing services to remain externally reachable while internal clients prefer private ingress paths.
A dedicated CoreDNS instance runs in-cluster, separate from the cluster-internal K3s CoreDNS, and is authoritative for the split-horizon zones. OPNsense Unbound forwards those zones to it over DNS-over-TLS. This keeps DNS overrides version-controlled and avoids click-ops in OPNsense.
______________________________________________________________________
## 6. Ingress Tiers
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart LR
subgraph external["External"]
Internet
CFEdge["Cloudflare Edge"]
end
subgraph oci["OCI"]
Admission["Address Admission"]
Relay["L4 Relay"]
NLB["OCI NLB"]
end
subgraph cluster["K3s Cluster"]
GW_R["Gateway R\n(Relay)"]
LB_R["LoadBalancer R\n(L4-passed)"]
Short["Short tunnel"]
GW_S["Gateway S\n(Short)"]
GW_I["Gateway I\n(Internal)"]
LB_D["LoadBalancer D\n(Direct)"]
SVC["Cluster Services"]
end
Internet -->|"O: Cloudflare Tunnel"| CFEdge --> SVC
Internet -->|"R: declared TCP ports"| Admission --> Relay
Relay -->|"80/443"| GW_R --> SVC
Relay -->|"L4-passed service"| LB_R --> SVC
Internet -->|"S"| NLB
NLB -->|"Site-to-Site VPN"| Short --> GW_S --> SVC
LAN["LAN / WireGuard"] -->|"I: Private CIDR"| GW_I --> SVC
LAN["LAN / WireGuard"] -->|"D: Dedicated IP"| LB_D --> SVC
SVC -->|"C: ClusterIP only"| SVC
```
| Tier | Label | Path | TLS Termination | Certificate Authority | Auth Requirement |
| --- | --- | --- | --- | --- | --- |
| **H** | Hypertext | Cloudflare Pages; no cluster ingress | Cloudflare Edge | Cloudflare | Anonymous |
| **O** | Orange Cloud | Cloudflare Tunnel → Service | Cloudflare Edge | Cloudflare | Cloudflare-managed |
| **R** | Relay | OCI L4 relay → WireGuard → Gateway R → Service | Gateway R or L4-passed service | Let’s Encrypt (DNS-01) | Required per service |
| **S** | Short | OCI NLB → Site-to-Site VPN → Gateway S | Gateway S or none | Let’s Encrypt (DNS-01) | Service-native |
| **I** | Internal | LAN / WireGuard → Gateway I → Service | Gateway I | Vault PKI | Network-level (LAN/WG) |
| **D** | Direct | LAN / WireGuard → LoadBalancer IP → Service | Service-native (or none) | Service-native | Service-native |
| **C** | Cluster Internal | ClusterIP only | Service-specific (or none) | Service-specific | Cilium network policy |
The [OCI L4 Relay Specification](oci-l4-relay-spec.md) is authoritative for the Tier R design. The [Tier S model](tier-s/) is authoritative for the Tier S design.
### 6.1 Gateway IP Allocation
| Gateway | CiliumLoadBalancerIPPool | Advertised via |
| --- | --- | --- |
| Gateway R | Dedicated private CIDR pool | Routed over the OCI WireGuard path |
| Gateway S | ClusterIP; no dedicated pool | Reached through the Site-to-Site VPN |
| Gateway I | Dedicated private CIDR pool | BGP → LAN |
| Tier D | Dedicated private CIDR pool | BGP → LAN |
Known and reserved LoadBalancer addresses:
| Address | Owner |
| --- | --- |
| 10.90.141.119 | Gateway I |
| 10.104.40.127 | Gateway R |
| 10.190.165.53 | Forgejo SSH |
| 10.28.147.227 | Network DNS / CoreDNS |
| 172.28.129.182 | Zot direct registry endpoint |
| 10.108.252.250 | Quay direct registry endpoint |
| 10.66.127.81 | VersityGW S3 API direct endpoint |
| 10.59.91.218 | VersityGW WebUI direct endpoint |
| 10.69.181.158 | Vault active API direct endpoint |
| 10.47.201.154 | Public Forgejo SSH |
| 10.66.153.202 | Stalwart mail and account HTTPS |
| 10.80.158.106 | Short VPN endpoint |
| 10.157.251.228 | PostgreSQL read-only service |
| 10.234.158.153 | PostgreSQL read/write service |
______________________________________________________________________
## 7. Workload Inventory
| Workload | Purpose | Tier | Replicas | ZFS Node | Storage Backend |
| --- | --- | --- | --- | --- | --- |
| VictoriaMetrics | Metrics (time-series) | I | split | — | OpenEBS LocalPV Hostpath |
| VictoriaLogs | Logs | I | split | — | Longhorn (replica=2) |
| Grafana | Metrics visualization, alerting | I | ≥2 | — | — |
| VersityGW | S3-compatible object gateway | I | =1 | Yes | OpenEBS LocalPV ZFS |
| VersityGW WebGUI | Administrative interface for VersityGW | I | =1 | — | — |
| CloudNativePG | Managed PostgreSQL | D | =3 | — | OpenEBS LocalPV Hostpath |
| Internal Forgejo | Private Git hosting and Flux GitOps origin | I / D | ≥2 | — | Longhorn RWX, VersityGW (S3), Dragonfly, CloudNativePG |
| Public Forgejo | Public source development and release distribution | O / R / D | ≥2 per root | — | Longhorn RWX, VersityGW (S3), Dragonfly, CloudNativePG |
| FluxCD | GitOps controller | I | ≥2 | — | — |
| Argo Workflows | CI/CD, workflow automation, and curated image mirror pipelines | I | ≥2 | — | VersityGW (S3, artifacts), CloudNativePG |
| OLM | Operator Lifecycle Manager (default catalogs disabled) | C | ≥2 | — | — |
| Quay | Container registry for curated internal images and scanned mirrors | D | ≥2 | — | VersityGW (S3), CloudNativePG |
| Clair | Image vulnerability scanner for Quay | C | ≥2 | — | CloudNativePG |
| Zitadel | Identity provider (OIDC) | R | ≥2 | — | CloudNativePG |
| Cilium | CNI, kube-proxy replacement, BGP, network policy, host firewall | C | DaemonSet | — | — |
| Hubble | Cilium network observability (relay + UI) | I | ≥2 | — | — |
| Longhorn | Replicated block storage | I | DaemonSet | — | Distributed node disks |
| OpenEBS LocalPV ZFS | Local ZFS-backed PVs | C | DaemonSet | Yes | ZFS (2× Z2 vdevs) |
| Dragonfly | In-memory cache (Redis-compatible) | C | =2 each | — | — (memory); dedicated CR per consumer |
| cert-manager | Certificate lifecycle automation | C | ≥2 | — | — |
| Vault | Secrets management, internal CA | I | =3 (Raft) | — | OpenEBS LocalPV Hostpath |
| oauth2-proxy | Auth proxy for selected web UIs | R / I | ≥2 | — | — |
| Vault Secrets Operator | Sync Vault secrets → K8s Secrets | C | ≥2 | — | — |
| Stalwart | Authoritative mail server and alert-notification sink | I / R | ≥2 | — | CloudNativePG, VersityGW (S3), Dragonfly |
| Frigate | Home security NVR (1×1080p, 7d retention) | I | =1 | — | Longhorn |
**Victoria components:** durable storage components run as one replica while the cluster has one ZFS node. Stateless insert/select components may run with multiple replicas. Collection agents are not independent workloads.
Stalwart's service contract is defined by the [Authoritative Email Server Specification](authoritative-email-server-spec.md).
### 7.1 Forgejo
Forgejo implements separate internal and public repository authorities. The authority, ingress, state, runtime, feature, and verification contracts are in the [Forgejo Deployment Specification](forgejo-deployment-spec.md).
### 7.2 Container Image Supply Chain
Quay native proxy cache is intentionally disabled. Cold pull-through requires Quay-side organization membership and does not satisfy the desired declarative access model.
Images intended for cluster consumption should be mirrored into Quay from a Git-owned allowlist, scanned by Clair, and promoted only after policy checks pass. Argo Workflows is the planned automation point for this mirror/scan/promote flow. Zot remains available as a lightweight direct registry for bootstrap and locally built infrastructure images.
______________________________________________________________________
## 8. Storage Architecture
The cluster uses three complementary storage layers, each serving a distinct purpose: replicated block storage for lightweight stateful workloads, high-capacity local storage for bulk data, and S3-compatible object storage for service backends.
### 8.1 Storage Layers
**Longhorn** provides replicated block storage across the K3s nodes. It is used for workloads that need persistent volumes but do not have large capacity requirements.
**OpenEBS LocalPV Hostpath** exposes node-local host paths as PersistentVolumes. It is used for workloads that implement their own application-level replication or clustering.
**OpenEBS LocalPV ZFS** exposes local ZFS datasets as PersistentVolumes. It is constrained to the ZFS node, which hosts 2× Z2 vdevs over 12×3 TB HDDs. This layer is used for workloads with large or IO-intensive storage needs.
**VersityGW** provides an S3-compatible object gateway running on the ZFS node. It exposes storage backed by the ZFS node via an S3 API for workloads that consume S3-compatible storage directly.
### 8.2 Consumer Mapping
| Storage Layer | Consumer | Longhorn Replicas | Notes |
| --- | --- | --- | --- |
| OpenEBS LocalPV Hostpath | Vault (Raft) | N/A | Vault handles its own replication via Raft across local host paths |
| OpenEBS LocalPV Hostpath | CloudNativePG | N/A | CNPG handles its own replication via streaming replication |
| OpenEBS LocalPV Hostpath | VictoriaMetrics | N/A | Multiple vmstorage replicas handle metrics redundancy across local host paths |
| Longhorn | VictoriaLogs | 2 | Highest-volume logs component |
| Longhorn | Frigate | Default | Single 1080p camera, 7-day retention |
| Longhorn RWX | Forgejo git/SSH | share-manager | Bare repos and built-in SSH keys; NFS semantics acceptable at homelab scale |
| VersityGW (S3) | Forgejo blobs | N/A | Avatars and attachments in bucket `forgejo` (port 7070 in-cluster) |
| Dragonfly | Forgejo cache | N/A | Dedicated `forgejo` CR; session, cache, and queue |
| CloudNativePG | Stalwart state | N/A | Durable account, metadata, queue, and search state |
| VersityGW (S3) | Stalwart blobs | N/A | Durable message bodies and attachments in bucket `stalwart` |
| Dragonfly | Stalwart cache | N/A | Dedicated non-durable Redis-compatible state |
| OpenEBS LocalPV ZFS | VersityGW | N/A | ZFS node only |
### 8.3 Design Principles
| Principle | Implementation |
| --- | --- |
| Prefer application-level replication where explicitly deployed | Vault Raft, CNPG streaming replication, and VictoriaMetrics clustering justify using non-replicated local storage (OpenEBS LocalPV Hostpath) instead of replicated block storage |
| Bulk data on ZFS | Large-capacity workloads are pinned to the ZFS node via node affinity |
| Object-storage backend | VersityGW provides S3-compatible storage for workloads that require it |
| Separation of concerns | Longhorn for general PVs; OpenEBS LocalPV Hostpath for high-performance clustered apps; OpenEBS LocalPV ZFS for bulk local storage; VersityGW for object storage |
______________________________________________________________________
## 9. Security Architecture
### 9.1 Trust Boundaries
```mermaid
%%{init: {'theme': 'neutral'}}%%
graph TD
subgraph internet["Untrusted — Internet"]
ExtClient["External Client"]
end
subgraph cf_edge["Semi-Trusted — Cloudflare"]
CFTunnel["Cloudflare Tunnel + WAF"]
end
subgraph relay["Boundary — OCI Relay"]
Admission["Address Admission + L4 Limits"]
WGRelay["WireGuard Relay"]
end
subgraph boundary["Boundary — OPNsense"]
WG["Remote-client WireGuard"]
end
subgraph lan["Trusted — LAN"]
subgraph k3s["K3s Cluster"]
CiliumFW["Cilium Host Firewall"]
DefaultDeny["Global Default Deny"]
GW_R2["Gateway R"]
GW_I2["Gateway I"]
LB_R2["LoadBalancer R"]
Workloads["Workloads"]
end
end
ExtClient -->|"TLS (CF-terminated)"| CFTunnel -->|"Tunnel"| Workloads
ExtClient -->|"Tier R"| Admission --> WGRelay
WGRelay -->|"HTTP/TLS"| GW_R2
WGRelay -->|"L4 pass-through"| LB_R2 --> Workloads
ExtClient -->|"WireGuard"| WG -->|"LAN access"| GW_I2
GW_R2 -->|"Cilium policy"| Workloads
GW_I2 -->|"Cilium policy"| Workloads
```
### 9.2 Certificate Authorities
| Scope | CA | Issuance | Managed By |
| --- | --- | --- | --- |
| Tier R (public-facing) | Let’s Encrypt | DNS-01 challenge (Cloudflare DNS) | cert-manager |
| Tier I (internal) | Vault PKI | Vault-issued | cert-manager + Vault |
| Tier O | Cloudflare | Cloudflare-managed | Cloudflare |
### 9.3 Authentication Flow (Tier R Web Services)
```mermaid
%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
actor User
participant GW as Gateway R
participant OAP as oauth2-proxy
participant ZIT as Zitadel (IdP)
participant SVC as Backend Service
User->>GW: HTTPS request
GW->>SVC: Route (if native OIDC)
GW->>OAP: Route (if no native OIDC)
OAP->>ZIT: OIDC redirect
ZIT->>User: Login prompt
User->>ZIT: Credentials
ZIT->>OAP: ID token
OAP->>SVC: Authenticated request
SVC->>User: Response
```
| Service | Auth Method |
| --- | --- |
| Forgejo | See [deployment specification](forgejo-deployment-spec.md) |
| Quay | Native OIDC → Zitadel |
| Zitadel | Self |
| VersityGW & UI | IAM-style auth (S3 access keys) |
| Hubble UI | oauth2-proxy → Zitadel |
| Longhorn UI | oauth2-proxy → Zitadel |
### 9.4 Encryption at Rest
| Layer | Mechanism |
| --- | --- |
| Node root filesystem | LUKS |
| K3s secrets | K3s secrets encryption |
| Application secrets | Vault → Vault Secrets Operator → K8s Secrets |
| ZFS data | ZFS native AES-256-GCM encryption inherited from `********` |

85
docs/spec/mirrored-image-spec.md Normal file
View file

@ -0,0 +1,85 @@
# MirroredImage Specification
## Boundary
| Concern | Requirement |
| --------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Controller | A dedicated controller Deployment and service account watch `MirroredImage`; the Project Delivery controller does not. |
| Isolation | Mirrored Image and Project Delivery controller pods remain separate because their permissions and owned resources differ. |
| Reuse | Generic Go libraries and the F# runner may be shared; controller state machines remain separate. |
| Intent | One `MirroredImage` owns one upstream repository. |
| Destination | The corresponding Quay repositories are `candidates/<metadata.name>` and `mirror/<metadata.name>`. |
| Credentials | Two Zitadel-backed Quay personas isolate OAuth authority by organization; four assigned tokens isolate it by privilege. |
| New work | The `MirroredImage` API, controller state machine, Argo Workflows, and tests are independent work. |
| Excluded input | Smeltery specifications, implementation, workflows, and tests are not inputs. |
| Workload policy | Workload image-reference policy is outside the controller; Kyverno governs workload declarations. |
## Declaration
```yaml
kind: MirroredImage
metadata:
name: ingress-nginx
spec:
upstream: registry.k8s.io/ingress-nginx/controller
policy:
rejectAtOrAbove: High
digests:
- digest: sha256:...
tag: v1.13.0
- digest: sha256:...
```
## Quay authorization
OAuth application scopes are privilege boundaries, not organization boundaries. Each token acts as its assigned persona, and each persona has Quay authority only within its corresponding organization.
| Persona | Quay resource boundary | Token | OAuth scope |
| ---------------- | ---------------------- | ------------------- | --------------------------- |
| `mirror-bot` | `mirror` only | `mirror-reader` | `repo:read` |
| `mirror-bot` | `mirror` only | `mirror-writer` | `repo:create`, `repo:write` |
| `candidates-bot` | `candidates` only | `candidates-reader` | `repo:read` |
| `candidates-bot` | `candidates` only | `candidates-writer` | `repo:create`, `repo:write` |
The reader and writer tokens are distinct credentials despite being assigned to the same persona. Neither persona has Quay authority in `quorate` or in the other mirror organization.
Quay treats `repo:write` as including `repo:read`; each writer token therefore has create, read, and write authority inside its persona's organization.
| Runtime | Quay credentials |
| ---------------- | ------------------------------------ |
| Controller | `candidates-reader`, `mirror-reader` |
| Prepare Workflow | `candidates-writer` |
| Promote Workflow | `candidates-reader`, `mirror-writer` |
| Candidate GC | `candidates-writer` |
| Mirror GC | `mirror-writer` |
## Models
- [MirroredImage.tla](mirrored-image/MirroredImage.tla)
- [RetentionTags.tla](mirrored-image/RetentionTags.tla)
- [TagExpiry.tla](mirrored-image/TagExpiry.tla)
## Workflow interface
| Argo Workflow | Stages |
| ------------- | ----------------------- |
| Prepare | Prepare |
| Promotion | Promote |
| Cleanup | Candidate GC, Mirror GC |
| Field | Type |
| -------------------------- | ----------------------- |
| `retryOf` | Workflow-name reference |
| `mirror.quorate.dev/retry` | Workflow-name reference |
| `status.lastHandledRetry` | Workflow-name reference |
## Scan inputs
| Image shape | `ClairInputs` |
| --------------- | ------------------ |
| Single manifest | Candidate manifest |
| OCI index | Child manifests |
```math
\mathrm{Safe} \iff \forall m \in \mathrm{ClairInputs}:\ \mathrm{ReportAvailable}(m) \land \forall v \in \mathrm{Findings}(m):\ \mathrm{severity}(v) < \mathrm{spec.policy.rejectAtOrAbove}
```

3
docs/spec/mirrored-image/Expiry.cfg Normal file
View file

@ -0,0 +1,3 @@
SPECIFICATION Spec
INVARIANTS TypeOK ConfirmedRetention
CHECK_DEADLOCK FALSE

12
docs/spec/mirrored-image/FailureSafety.cfg Normal file
View file

@ -0,0 +1,12 @@
SPECIFICATION Spec
CONSTANTS
MaxEpoch = 2
MaxRetries = 2
Tagged = TRUE
Safe = TRUE
AllowRemoval = TRUE
AllowFailure = TRUE
AllowRetry = TRUE
RetryStageSet = {"prepare", "promotion", "cleanup"}
INVARIANTS TypeOK PromotionRequiresAcceptance DeclaredPromotionRetained DeclaredCandidateRetained CleanupObservedMeansGone RetryPreservesTagIdentity RetryAcknowledgedAfterCreation CleanupExcludesCopy
CHECK_DEADLOCK FALSE

13
docs/spec/mirrored-image/Lifecycle.cfg Normal file
View file

@ -0,0 +1,13 @@
SPECIFICATION FairSpec
CONSTANTS
MaxEpoch = 2
MaxRetries = 1
Tagged = FALSE
Safe = TRUE
AllowRemoval = TRUE
AllowFailure = FALSE
AllowRetry = TRUE
RetryStageSet = {"prepare", "promotion", "cleanup"}
INVARIANTS TypeOK PromotionRequiresAcceptance DeclaredPromotionRetained DeclaredCandidateRetained CleanupObservedMeansGone DeclaredAliasOnlyWhenRequested RetryPreservesTagIdentity RetryAcknowledgedAfterCreation CleanupExcludesCopy
PROPERTIES EventuallyPromoted EventuallyRetired
CHECK_DEADLOCK FALSE

279
docs/spec/mirrored-image/MirroredImage.tla Normal file
View file

@ -0,0 +1,279 @@
---- MODULE MirroredImage ----
EXTENDS Naturals, FiniteSets, TLC
CONSTANTS Tagged, Safe, AllowRemoval, AllowFailure, AllowRetry, RetryStageSet, MaxEpoch, MaxRetries
Stages == {"prepare", "promotion", "cleanup"}
Repos == {"candidate", "mirror"}
Children == {"amd64", "arm64"}
Phases == {"absent", "pending", "queued", "running", "published", "succeeded", "failed"}
Terminal(p) == p \in {"succeeded", "failed"}
VARIABLES desired, managed, retired, attempts, workflows, scanned, verdict,
stored, temporary, durableTags, promoted, copies, retryCount, retryAck,
cleanupObserved, grace, epoch, retryStage, badPromotion
vars == <<desired, managed, retired, attempts, workflows, scanned, verdict,
stored, temporary, durableTags, promoted, copies, retryCount, retryAck,
cleanupObserved, grace, epoch, retryStage, badPromotion>>
Available(r) == stored[r] /\ (temporary[r] \/ durableTags[r] # {})
Active == \E s \in Stages : attempts[s] # "absent" /\ ~Terminal(attempts[s])
Init ==
/\ desired = TRUE
/\ managed = FALSE
/\ retired = FALSE
/\ attempts = [s \in Stages |-> "absent"]
/\ workflows = [s \in Stages |-> "absent"]
/\ scanned = {}
/\ verdict = "pending"
/\ stored = [r \in Repos |-> FALSE]
/\ temporary = [r \in Repos |-> FALSE]
/\ durableTags = [r \in Repos |-> {}]
/\ promoted = FALSE
/\ copies = 0
/\ retryCount = 0
/\ retryAck = FALSE
/\ cleanupObserved = FALSE
/\ grace = [r \in Repos |-> FALSE]
/\ epoch = 1
/\ retryStage = "none" /\ badPromotion = FALSE
Manage ==
/\ desired /\ ~managed /\ ~retired
/\ managed' = TRUE
/\ UNCHANGED <<desired, retired, attempts, workflows, scanned, verdict,
stored, temporary, durableTags, promoted, copies, retryCount,
retryAck, cleanupObserved, grace, epoch, retryStage, badPromotion>>
Plan(s) ==
/\ managed /\ ~Active /\ attempts[s] = "absent"
/\ CASE s = "prepare" -> desired /\ attempts["cleanup"] = "absent"
[] s = "promotion" -> desired /\ attempts["cleanup"] = "absent"
/\ attempts["prepare"] = "succeeded" /\ verdict = "accepted"
[] s = "cleanup" -> ~desired
/\ attempts' = [attempts EXCEPT ![s] = "pending"]
/\ UNCHANGED <<desired, managed, retired, workflows, scanned, verdict,
stored, temporary, durableTags, promoted, copies, retryCount,
retryAck, cleanupObserved, grace, epoch, retryStage, badPromotion>>
Submit(s) ==
/\ attempts[s] = "pending" /\ workflows[s] = "absent"
/\ s = "cleanup" => \A t \in {"prepare", "promotion"} : attempts[t] = "absent" \/ Terminal(attempts[t])
/\ workflows' = [workflows EXCEPT ![s] = "queued"]
/\ retryAck' = (retryAck \/ (retryCount > 0 /\ s = retryStage))
/\ UNCHANGED <<desired, managed, retired, attempts, scanned, verdict,
stored, temporary, durableTags, promoted, copies, retryCount,
cleanupObserved, grace, epoch, retryStage, badPromotion>>
Start(s) ==
/\ workflows[s] = "queued"
/\ ~\E t \in Stages : workflows[t] \in {"running", "published"}
/\ workflows' = [workflows EXCEPT ![s] = "running"]
/\ UNCHANGED <<desired, managed, retired, attempts, scanned, verdict,
stored, temporary, durableTags, promoted, copies, retryCount,
retryAck, cleanupObserved, grace, epoch, retryStage, badPromotion>>
PartialCopy(s) ==
/\ s \in {"prepare", "promotion"} /\ workflows[s] = "running"
/\ s = "promotion" => Available("candidate")
/\ LET r == IF s = "prepare" THEN "candidate" ELSE "mirror"
IN /\ ~stored[r]
/\ stored' = [stored EXCEPT ![r] = TRUE]
/\ temporary' = [temporary EXCEPT ![r] = TRUE]
/\ grace' = [grace EXCEPT ![r] = FALSE]
/\ badPromotion' = (badPromotion \/ (s = "promotion" /\ (~Safe \/ verdict # "accepted" \/ scanned # Children)))
/\ UNCHANGED <<desired, managed, retired, attempts, workflows, scanned, verdict,
durableTags, promoted, copies, retryCount, retryAck,
cleanupObserved, epoch, retryStage>>
Publish(s) ==
/\ s \in {"prepare", "promotion"} /\ workflows[s] = "running"
/\ s = "promotion" => Available("candidate")
/\ LET r == IF s = "prepare" THEN "candidate" ELSE "mirror"
IN /\ stored' = [stored EXCEPT ![r] = TRUE]
/\ temporary' = [temporary EXCEPT ![r] = TRUE]
/\ durableTags' = [durableTags EXCEPT ![r] = @ \cup {"retention"}]
/\ grace' = [grace EXCEPT ![r] = FALSE]
/\ promoted' = (promoted \/ (s = "promotion" /\ attempts["cleanup"] = "absent"))
/\ copies' = IF s = "promotion" THEN copies + 1 ELSE copies
/\ badPromotion' = (badPromotion \/ (s = "promotion" /\ (~Safe \/ verdict # "accepted" \/ scanned # Children)))
/\ workflows' = [workflows EXCEPT ![s] = "published"]
/\ UNCHANGED <<desired, managed, retired, attempts, scanned, verdict,
retryCount, retryAck, cleanupObserved, epoch, retryStage>>
Alias(s) ==
/\ Tagged /\ s \in {"prepare", "promotion"} /\ workflows[s] = "published"
/\ LET r == IF s = "prepare" THEN "candidate" ELSE "mirror"
IN /\ "declared" \notin durableTags[r]
/\ durableTags' = [durableTags EXCEPT ![r] = @ \cup {"declared"}]
/\ UNCHANGED <<desired, managed, retired, attempts, workflows, scanned, verdict,
stored, temporary, promoted, copies, retryCount, retryAck,
cleanupObserved, grace, epoch, retryStage, badPromotion>>
Delete(r) ==
/\ workflows["cleanup"] = "running"
/\ durableTags[r] # {}
/\ durableTags' = [durableTags EXCEPT ![r] = {}]
/\ UNCHANGED <<desired, managed, retired, attempts, workflows, scanned, verdict,
stored, temporary, promoted, copies, retryCount, retryAck,
cleanupObserved, grace, epoch, retryStage, badPromotion>>
Finish(s, success) ==
/\ workflows[s] \in {"running", "published"}
/\ success \/ AllowFailure \/ (s = "promotion" /\ ~Available("candidate"))
/\ success => IF s = "cleanup" THEN \A r \in Repos : durableTags[r] = {}
ELSE /\ workflows[s] = "published"
/\ Tagged => "declared" \in durableTags[IF s = "prepare" THEN "candidate" ELSE "mirror"]
/\ workflows' = [workflows EXCEPT ![s] = IF success THEN "succeeded" ELSE "failed"]
/\ UNCHANGED <<desired, managed, retired, attempts, scanned, verdict,
stored, temporary, durableTags, promoted, copies, retryCount,
retryAck, cleanupObserved, grace, epoch, retryStage, badPromotion>>
Observe(s) ==
/\ attempts[s] = "pending" /\ Terminal(workflows[s])
/\ attempts' = [attempts EXCEPT ![s] = workflows[s]]
/\ UNCHANGED <<desired, managed, retired, workflows, scanned, verdict,
stored, temporary, durableTags, promoted, copies, retryCount,
retryAck, cleanupObserved, grace, epoch, retryStage, badPromotion>>
Scan(c) ==
/\ desired /\ attempts["prepare"] = "succeeded"
/\ attempts["cleanup"] = "absent" /\ verdict = "pending"
/\ Available("candidate") /\ c \notin scanned
/\ scanned' = scanned \cup {c}
/\ verdict' = IF ~Safe /\ c = "arm64" THEN "rejected"
ELSE IF scanned' = Children THEN "accepted" ELSE "pending"
/\ UNCHANGED <<desired, managed, retired, attempts, workflows,
stored, temporary, durableTags, promoted, copies, retryCount,
retryAck, cleanupObserved, grace, epoch, retryStage, badPromotion>>
Expire(r) ==
/\ temporary[r]
/\ temporary' = [temporary EXCEPT ![r] = FALSE]
/\ UNCHANGED <<desired, managed, retired, attempts, workflows, scanned, verdict,
stored, durableTags, promoted, copies, retryCount,
retryAck, cleanupObserved, grace, epoch, retryStage, badPromotion>>
Age(r) ==
/\ stored[r] /\ ~Available(r) /\ ~grace[r]
/\ grace' = [grace EXCEPT ![r] = TRUE]
/\ UNCHANGED <<desired, managed, retired, attempts, workflows, scanned, verdict,
stored, temporary, durableTags, promoted, copies, retryCount,
retryAck, cleanupObserved, epoch, retryStage, badPromotion>>
Collect(r) ==
/\ stored[r] /\ ~Available(r) /\ grace[r]
/\ stored' = [stored EXCEPT ![r] = FALSE]
/\ UNCHANGED <<desired, managed, retired, attempts, workflows, scanned, verdict,
temporary, durableTags, promoted, copies, retryCount,
retryAck, cleanupObserved, grace, epoch, retryStage, badPromotion>>
Remove ==
/\ AllowRemoval /\ desired
/\ desired' = FALSE
/\ UNCHANGED <<managed, retired, attempts, workflows, scanned, verdict,
stored, temporary, durableTags, promoted, copies, retryCount,
retryAck, cleanupObserved, grace, epoch, retryStage, badPromotion>>
Forget ==
/\ managed /\ ~Active /\ attempts["cleanup"] = "succeeded"
/\ \A r \in Repos : ~stored[r]
/\ managed' = desired
/\ retired' = ~desired
/\ cleanupObserved' = ~desired
/\ attempts' = IF desired THEN [s \in Stages |-> "absent"] ELSE attempts
/\ workflows' = IF desired THEN [s \in Stages |-> "absent"] ELSE workflows
/\ scanned' = IF desired THEN {} ELSE scanned
/\ verdict' = IF desired THEN "pending" ELSE verdict
/\ promoted' = IF desired THEN FALSE ELSE promoted
/\ retryAck' = IF desired THEN FALSE ELSE retryAck
/\ UNCHANGED <<desired, stored, temporary, durableTags, copies, retryCount, grace, epoch, retryStage, badPromotion>>
Readd ==
/\ ~desired /\ epoch < MaxEpoch
/\ desired' = TRUE /\ epoch' = epoch + 1
/\ retired' = FALSE /\ promoted' = FALSE /\ cleanupObserved' = FALSE
/\ attempts' = IF managed THEN attempts ELSE [s \in Stages |-> "absent"]
/\ workflows' = IF managed THEN workflows ELSE [s \in Stages |-> "absent"]
/\ scanned' = IF managed THEN scanned ELSE {}
/\ verdict' = IF managed THEN verdict ELSE "pending"
/\ retryAck' = IF managed THEN retryAck ELSE FALSE
/\ UNCHANGED <<managed, stored, temporary, durableTags, copies, retryCount, grace, retryStage, badPromotion>>
ObservePromoted ==
/\ desired /\ ~promoted /\ attempts["promotion"] = "succeeded"
/\ attempts["cleanup"] = "absent" /\ Available("mirror")
/\ promoted' = TRUE
/\ UNCHANGED <<desired, managed, retired, attempts, workflows, scanned, verdict,
stored, temporary, durableTags, copies, retryCount, retryAck,
cleanupObserved, grace, epoch, retryStage, badPromotion>>
Retry(s) ==
/\ AllowRetry /\ retryCount < MaxRetries /\ managed /\ ~Active
/\ Terminal(attempts[s])
/\ attempts' = [t \in Stages |->
IF t = s THEN "pending"
ELSE IF s = "prepare" /\ t = "promotion" THEN "absent"
ELSE IF t = "cleanup" /\ attempts[t] # "absent" THEN "pending"
ELSE attempts[t]]
/\ workflows' = [t \in Stages |->
IF t = s \/ (s = "prepare" /\ t = "promotion") \/ t = "cleanup"
THEN "absent" ELSE workflows[t]]
/\ scanned' = IF s = "prepare" THEN {} ELSE scanned
/\ verdict' = IF s = "prepare" THEN "pending" ELSE verdict
/\ retryCount' = retryCount + 1
/\ retryStage' = s
/\ retryAck' = FALSE
/\ UNCHANGED <<desired, managed, retired,
stored, temporary, durableTags, promoted, copies,
cleanupObserved, grace, epoch, badPromotion>>
Next ==
\/ Manage
\/ \E s \in Stages : Plan(s) \/ Submit(s) \/ Start(s) \/ Observe(s)
\/ \E success \in BOOLEAN : Finish(s, success)
\/ \E s \in {"prepare", "promotion"} : PartialCopy(s) \/ Publish(s) \/ Alias(s)
\/ \E r \in Repos : Delete(r) \/ Expire(r) \/ Age(r) \/ Collect(r)
\/ \E c \in Children : Scan(c)
\/ Remove \/ Forget \/ (\E s \in RetryStageSet : Retry(s)) \/ Readd \/ ObservePromoted
Spec == Init /\ [][Next]_vars
FairSpec == Spec
/\ WF_vars(Manage) /\ WF_vars(Forget) /\ WF_vars(ObservePromoted)
/\ (\A s \in Stages : WF_vars(Plan(s)) /\ WF_vars(Submit(s))
/\ WF_vars(Start(s)) /\ WF_vars(Observe(s))
/\ WF_vars(Finish(s, TRUE) \/ Finish(s, FALSE)))
/\ (\A s \in {"prepare", "promotion"} : WF_vars(Publish(s)) /\ WF_vars(Alias(s)))
/\ (\A r \in Repos : WF_vars(Delete(r)) /\ WF_vars(Expire(r))
/\ WF_vars(Age(r)) /\ WF_vars(Collect(r)))
/\ (\A c \in Children : WF_vars(Scan(c)))
TypeOK ==
/\ desired \in BOOLEAN /\ managed \in BOOLEAN /\ retired \in BOOLEAN
/\ attempts \in [Stages -> Phases] /\ workflows \in [Stages -> Phases]
/\ scanned \subseteq Children /\ verdict \in {"pending", "accepted", "rejected"}
/\ stored \in [Repos -> BOOLEAN] /\ temporary \in [Repos -> BOOLEAN]
/\ durableTags \in [Repos -> SUBSET {"retention", "declared", "retry"}]
/\ promoted \in BOOLEAN /\ copies \in Nat /\ epoch \in 1..MaxEpoch
/\ retryCount \in 0..MaxRetries /\ retryAck \in BOOLEAN
/\ retryStage \in Stages \cup {"none"} /\ badPromotion \in BOOLEAN
/\ cleanupObserved \in BOOLEAN /\ grace \in [Repos -> BOOLEAN]
PromotionRequiresAcceptance == ~badPromotion
DeclaredPromotionRetained == desired /\ promoted => Available("mirror")
DeclaredCandidateRetained == desired /\ attempts["cleanup"] = "absent" /\ attempts["prepare"] = "succeeded" => Available("candidate")
CleanupObservedMeansGone == cleanupObserved => \A r \in Repos : ~stored[r]
DeclaredAliasOnlyWhenRequested == ~Tagged => \A r \in Repos : "declared" \notin durableTags[r]
RetryPreservesTagIdentity == \A r \in Repos : durableTags[r] \subseteq {"retention", "declared"}
RetryAcknowledgedAfterCreation == retryAck => retryStage # "none" /\ workflows[retryStage] # "absent"
CleanupExcludesCopy == workflows["cleanup"] \in {"running", "published"} =>
\A s \in {"prepare", "promotion"} : workflows[s] \notin {"running", "published"}
EventuallyPromoted == (desired /\ managed) ~> (~desired \/ promoted)
EventuallyRetired == (~desired /\ managed) ~> (retired \/ desired)
EventuallyRejected == <>(verdict = "rejected")
====

3
docs/spec/mirrored-image/Ownership.cfg Normal file
View file

@ -0,0 +1,3 @@
SPECIFICATION Spec
INVARIANTS TypeOK RetentionNamesOwned DeclaredDigestsRetained RetryPreservesTagIdentity
CHECK_DEADLOCK FALSE

13
docs/spec/mirrored-image/Rejection.cfg Normal file
View file

@ -0,0 +1,13 @@
SPECIFICATION FairSpec
CONSTANTS
MaxEpoch = 2
MaxRetries = 2
Tagged = FALSE
Safe = FALSE
AllowRemoval = FALSE
AllowFailure = FALSE
AllowRetry = TRUE
RetryStageSet = {"prepare", "promotion", "cleanup"}
INVARIANTS TypeOK PromotionRequiresAcceptance DeclaredCandidateRetained DeclaredAliasOnlyWhenRequested RetryPreservesTagIdentity RetryAcknowledgedAfterCreation CleanupExcludesCopy
PROPERTY EventuallyRejected
CHECK_DEADLOCK FALSE

57
docs/spec/mirrored-image/RetentionTags.tla Normal file
View file

@ -0,0 +1,57 @@
---- MODULE RetentionTags ----
EXTENDS Naturals, FiniteSets, TLC
Digests == {"a", "b"}
RetentionTag(d) == IF d = "a" THEN "retain-a" ELSE "retain-b"
RetentionTags == {RetentionTag(d) : d \in Digests}
Tags == RetentionTags \cup {"stable"}
VARIABLES aliases, tags, phase, copies, retained
vars == <<aliases, tags, phase, copies, retained>>
AliasAllowed(d) == aliases[d] \notin RetentionTags \/ aliases[d] = RetentionTag(d)
Init ==
/\ aliases \in [Digests -> Tags \cup {"none"}]
/\ tags = [t \in Tags |-> "none"]
/\ phase = [d \in Digests |-> "idle"]
/\ copies = [d \in Digests |-> 0]
/\ retained = {}
Publish(d) ==
/\ phase[d] \in {"idle", "finished", "retired"} /\ copies[d] < 2
/\ AliasAllowed(d)
/\ tags' = [tags EXCEPT ![RetentionTag(d)] = d]
/\ phase' = [phase EXCEPT ![d] = "published"]
/\ copies' = [copies EXCEPT ![d] = @ + 1]
/\ retained' = retained \cup {d}
/\ UNCHANGED aliases
Alias(d) ==
/\ phase[d] = "published"
/\ tags' = IF aliases[d] = "none" THEN tags ELSE [tags EXCEPT ![aliases[d]] = d]
/\ phase' = [phase EXCEPT ![d] = "finished"]
/\ UNCHANGED <<aliases, copies, retained>>
Delete(d) ==
/\ phase[d] = "finished"
/\ tags' = [t \in Tags |-> IF tags[t] = d THEN "none" ELSE tags[t]]
/\ retained' = retained \ {d}
/\ phase' = [phase EXCEPT ![d] = "retired"]
/\ UNCHANGED <<aliases, copies>>
Next == \E d \in Digests : Publish(d) \/ Alias(d) \/ Delete(d)
Spec == Init /\ [][Next]_vars
TypeOK ==
/\ aliases \in [Digests -> Tags \cup {"none"}]
/\ tags \in [Tags -> Digests \cup {"none"}]
/\ phase \in [Digests -> {"idle", "published", "finished", "retired"}]
/\ copies \in [Digests -> 0..2] /\ retained \subseteq Digests
RetentionNamesOwned == \A d \in Digests : tags[RetentionTag(d)] \in {"none", d}
DeclaredDigestsRetained == \A d \in retained : tags[RetentionTag(d)] = d
RetryPreservesTagIdentity == \A d \in Digests :
{t \in Tags : tags[t] = d} \subseteq {RetentionTag(d), aliases[d]}
====

13
docs/spec/mirrored-image/Retries.cfg Normal file
View file

@ -0,0 +1,13 @@
SPECIFICATION FairSpec
CONSTANTS
MaxEpoch = 2
MaxRetries = 2
Tagged = FALSE
Safe = TRUE
AllowRemoval = TRUE
AllowFailure = FALSE
AllowRetry = TRUE
RetryStageSet = {"prepare", "promotion", "cleanup"}
INVARIANTS TypeOK PromotionRequiresAcceptance DeclaredPromotionRetained DeclaredCandidateRetained CleanupObservedMeansGone DeclaredAliasOnlyWhenRequested RetryPreservesTagIdentity RetryAcknowledgedAfterCreation CleanupExcludesCopy
PROPERTIES EventuallyPromoted EventuallyRetired
CHECK_DEADLOCK FALSE

34
docs/spec/mirrored-image/TagExpiry.tla Normal file
View file

@ -0,0 +1,34 @@
---- MODULE TagExpiry ----
EXTENDS Naturals, TLC
VARIABLES live, permanent, confirmed, writes
vars == <<live, permanent, confirmed, writes>>
Init == /\ live = FALSE /\ permanent = FALSE /\ confirmed = FALSE /\ writes = 0
Write ==
/\ writes < 2 /\ ~permanent
/\ live' = TRUE /\ permanent' = FALSE /\ writes' = writes + 1
/\ UNCHANGED confirmed
ClearExpiry ==
/\ live /\ ~permanent
/\ permanent' = TRUE
/\ UNCHANGED <<live, confirmed, writes>>
Confirm ==
/\ live /\ permanent /\ ~confirmed
/\ confirmed' = TRUE
/\ UNCHANGED <<live, permanent, writes>>
Expire ==
/\ live /\ ~permanent
/\ live' = FALSE
/\ UNCHANGED <<permanent, confirmed, writes>>
Next == Write \/ ClearExpiry \/ Confirm \/ Expire
Spec == Init /\ [][Next]_vars
TypeOK == live \in BOOLEAN /\ permanent \in BOOLEAN /\ confirmed \in BOOLEAN /\ writes \in 0..2
ConfirmedRetention == confirmed => live
====

13
docs/spec/mirrored-image/TaggedLifecycle.cfg Normal file
View file

@ -0,0 +1,13 @@
SPECIFICATION FairSpec
CONSTANTS
MaxEpoch = 2
MaxRetries = 1
Tagged = TRUE
Safe = TRUE
AllowRemoval = TRUE
AllowFailure = FALSE
AllowRetry = TRUE
RetryStageSet = {"prepare", "promotion", "cleanup"}
INVARIANTS TypeOK PromotionRequiresAcceptance DeclaredPromotionRetained DeclaredCandidateRetained CleanupObservedMeansGone DeclaredAliasOnlyWhenRequested RetryPreservesTagIdentity RetryAcknowledgedAfterCreation CleanupExcludesCopy
PROPERTIES EventuallyPromoted EventuallyRetired
CHECK_DEADLOCK FALSE

457
docs/spec/oci-l4-relay-spec.md Normal file
View file

@ -0,0 +1,457 @@
# OCI L4 Relay Specification
## 1. Problem Statement
Public services that require direct L4/TLS ingress use a stable cloud endpoint without publishing or accepting service traffic on the residential WAN address. Tier R preserves original client addresses and terminates TLS inside the cluster while keeping the home firewall outside the public ingress data path.
## 2. Deployed Design
A single-purpose OCI VM provides Tier R L4 ingress. A service assigned to Tier R uses a Cloudflare gray-cloud A record for the OCI reserved IPv4 address. The VM forwards TCP/80 and TCP/443 over WireGuard to the dedicated in-cluster Cilium Gateway identity referred to as **Gateway R**. It forwards TCP/25, TCP/465, and TCP/993 to the mail LoadBalancer.
The OCI VM provides transparent L4 forwarding. TLS terminates at the declared cluster targets.
```text
Internet -> OCI public IP -> DNAT -> WireGuard -> Gateway R or L4 service
```
OPNsense is not in the Tier R ingress data path. For Tier R, it passes only outbound WireGuard traffic from the Kubernetes-scheduled K3s node to OCI. Its WAN address is control-plane input used to constrain OCI WireGuard ingress to one IPv4 `/32`.
| Identity | Meaning |
| --- | --- |
| Tier R | Public relay ingress through OCI over WireGuard |
| Gateway R | Dedicated Cilium Gateway used by Tier R |
| `gateway-relay` | Kubernetes object name for Gateway R |
## 3. Architecture
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart LR
subgraph internet["Internet"]
Client["External Client"]
DNS["Cloudflare DNS\nA records"]
end
subgraph oci["OCI"]
NSG["Relay NSG\nservice admission"]
VM["Relay VM\nnftables + WireGuard"]
OCISecrets["OCI Secrets\nkey seed + public key"]
OCIAPI["OCI API\nrestricted NSG update"]
end
subgraph home["Home LAN"]
OPN["OPNsense\nWAN status API"]
Controller["WAN Controller\n10s poll + 2 observations"]
Node["K3s Nodes\nactive owner + standby"]
Cilium["Cilium\nLB + Gateway API"]
GW["Gateway R\ngateway-relay"]
Mail["Mail\nLoadBalancer"]
SVC["Public Services"]
Vault["Vault\npeer, PKI, API material"]
end
DNS --> Client
Client -->|"TCP 25/80/443/465/993"| NSG --> VM
VM -->|"DNAT, no SNAT"| Node
VM <-->|"WireGuard"| Node
Node -->|"routed to Gateway R"| GW
Node -->|"routed to mail address"| Mail
GW --> SVC
OCISecrets --> VM
Vault --> Node
Vault --> Controller
Controller -->|"PKI-validated HTTPS"| OPN
Controller -->|"replace exact UDP source rule"| OCIAPI --> NSG
Node -. "outbound UDP only" .-> OPN
```
### 3.1 Data Path
| Step | Source | Destination | Notes |
| --- | --- | --- | --- |
| 1 | External client | OCI public service port | Cloudflare DNS returns the reserved OCI address. |
| 2a | OCI relay | Gateway R:80/443 | OCI applies DNAT and forwards web traffic over WireGuard. |
| 2b | OCI relay | Mail target:25/465/993 | OCI applies DNAT and forwards mail traffic over WireGuard. |
| 3 | K3s node | Selected LoadBalancer | Cilium delivers the packet to Gateway R or the mail target without another DNAT layer. |
| 4a | Cilium Gateway | Web listener | Relay forwarding ends at the declared Gateway listener. |
| 4b | Mail target | Mail protocol listener | Relay forwarding ends at the declared mail listener. |
| 5 | Cluster service | External client | Replies route back through the OCI WireGuard tunnel. |
The relay must preserve the original client source address. Therefore the OCI VM performs DNAT only and does not SNAT traffic forwarded to the cluster.
### 3.2 Control Plane Ownership
| Component | Owner | Management model |
| --- | --- | --- |
| OCI infrastructure, IAM, monitoring, budget | OpenTofu | Declarative OCI provider resources |
| Cloudflare service DNS | OpenTofu | Gray-cloud records consume the reserved address from OCI remote state |
| Sealed relay image | Packer and pinned OCI SDK | Digest-identified build from a pinned Oracle Linux platform image |
| OCI relay key seed and public key | OCI Secrets | Deterministic private-key derivation; change-aware public-key publication |
| Home-side WireGuard key material | Vault | Private key consumed by Kubernetes; public key rendered into relay configuration |
| Admission source cache and publishers | Argo Workflows and Longhorn | Independent web-allow and SMTP-deny compilation, validation, and publication |
| Admission bundles | OCI Object Storage | Policy-specific validation, caching, and atomic activation |
| Cilium Gateway identity | Kubernetes manifests | Dedicated `gateway-relay` / Gateway R |
| Home-side tunnel and policy routing | Kubernetes host-level workload | Two anti-affined `hostNetwork` replicas; an exact-name Lease admits one active owner |
| WireGuard source restriction | Kubernetes controller | Polls OPNsense WAN over Vault-PKI TLS and updates one OCI NSG rule |
| OPNsense API identity and trust root | Vault and Vault PKI | VSO projects a read-only API credential and the dedicated OPNsense root CA |
| Guest logs | VictoriaLogs | Authenticated bounded shipping outside the relay availability path |
## 4. Network Design
### 4.1 Identities and Addressing
Operational design refers to Gateway identities while the implementation renders their assigned addresses into OCI `nftables`, WireGuard, and home-side routing.
| Identity or Network | Purpose | Requirement |
| --- | --- | --- |
| OCI public address | Public DNS target | Reserved IPv4; IPv6 is out of scope for v1 |
| OPNsense WAN address | WireGuard NSG source | Current public IPv4 represented as one exact `/32` |
| WireGuard transfer CIDR | Relay tunnel | `10.151.47.238/30`; does not overlap pod, service, LAN, or LB ranges |
| Gateway R / `gateway-relay` | Relay-specific public Gateway | Dedicated Cilium LoadBalancer address `10.104.40.127` |
| Mail LoadBalancer | Direct mail protocol target | Dedicated Cilium LoadBalancer address `10.66.153.202` |
The WireGuard transfer CIDR must not overlap:
| Existing Range | Purpose |
| --- | --- |
| `10.90.141.119/16` | Cilium LoadBalancer IP pool |
| `172.19.28.14/12` | Cilium pod CIDR |
| Home LAN ranges | Internal routing and OPNsense policy |
### 4.2 OCI Relay Rules
The OCI VM owns public DNAT.
| Rule | Requirement |
| --- | --- |
| Public TCP/80 | DNAT to Gateway R TCP/80 over WireGuard |
| Public TCP/443 | DNAT to Gateway R TCP/443 over WireGuard |
| Public TCP/25 | DNAT to the mail target TCP/25 over WireGuard |
| Public TCP/465 | DNAT to the mail target TCP/465 over WireGuard |
| Public TCP/993 | DNAT to the mail target TCP/993 over WireGuard |
| SNAT | Disabled for forwarded client traffic |
| Forwarding | Enabled only between public interface and `wg-oci0` for approved ports |
| Rate limiting | Enforced before traffic enters WireGuard to protect the relay path |
| SSH | No OCI network security group ingress or authorized keys; `sshd.service` and `sshd.socket` are masked in the sealed image |
| WireGuard UDP | Open only on the relay port from the current OPNsense WAN IPv4 `/32` |
The relay uses an atomically loaded declarative `nftables` ruleset and does not mutate `iptables`.
### 4.3 Relay Rate Limiting
The OCI VM enforces L4 ingress limits before conntrack, DNAT, and WireGuard forwarding. The controls preserve service availability under hostile ingress; their numeric ceilings are containment settings, not throughput objectives.
Rate limiting is not a substitute for application-level quotas or authentication. It is a transport guardrail for the relay path.
| Control | Deployed contract |
| --- | --- |
| Traffic-control shaping | Shape traffic toward home at 150 Mbit/s |
| Web SYN controls | 120/minute with burst 120 per source; 1,000/second with burst 1,000 aggregate |
| SMTP SYN controls | 60/minute with burst 20 per source; 500/second with burst 500 aggregate |
| Authenticated-mail controls | 30/minute with burst 15 per source; 200/second with burst 200 aggregate |
| Aggregate packet ceilings | 20,000/second web, 10,000/second SMTP, and 5,000/second authenticated mail, each with an equal burst |
| HTTP-only quarantine | After 30 unmatched TCP/80 attempts/hour with a five-packet burst, deny TCP/80 for 30 minutes |
| HTTPS invariant | A TCP/443 attempt records a ten-minute exception, clears HTTP-only quarantine, and remains outside it |
| State bounds | Each dynamic per-source tracker holds at most 65,536 entries and has an explicit timeout |
| Protocol boundary | WireGuard handshakes are outside web, SMTP, authenticated-mail, quarantine, and shaping admission |
| Observability | Named aggregate counters and per-element SMTP quarantine counters expose relay drops |
| Configuration | OpenTofu renders every limit into the sealed guest configuration |
### 4.4 Address Admission
```mermaid
flowchart LR
Geography[MaxMind and IPLocate\nJP KR SG US Europe] --> Candidate[Geographic consensus]
VPNGeo[MaxMind or IPLocate\nJP KR] --> VPNException[Configured VPN ASN exception]
VPNASN[Explicit operator ASNs] --> VPNException
Infrastructure[Cloud CDN and VPS ranges] --> Deny[Selective deny union]
Abuse[Tor Spamhaus selective IP2Proxy] --> Deny
Candidate --> Difference[Admitted IPv4 set]
VPNException --> Difference
Deny --> Difference
Difference --> Version["web-allow/v1\nimmutable version"]
Version --> Current[Atomic current promotion]
Current --> Guest[Guest refresh]
Guest --> Rules[Atomic web admission table]
Guest --> LastGood[Validated last-known-good cache]
```
| Boundary | Contract |
| --- | --- |
| Purpose | Reduce hosting, anonymizer, and known-abuse exposure as defense in depth; never treat source classification as human identity |
| Geographic base | Intersect MaxMind and IPLocate country data for Japan, Korea, Singapore, the United States, and Europe |
| VPN operator exception | Intersect explicitly configured VPN ASNs with the union of MaxMind and IPLocate Japan/Korea ranges, exclude the geographic base, and subtract the selective deny union |
| Selective deny | Subtract published cloud, CDN, VPS, Tor, Spamhaus DROP, and fresh selective IP2Proxy ranges |
| Publisher | Run the multi-source fetch and compile DAG in-cluster with provider cadence, validators, bounded retry, and a durable private cache |
| Publication | Publish immutable versions and current state under `web-allow/v1`; retain legacy `admission/v1` only during consumer migration |
| Guest refresh | Read the current private object through the instance principal every six hours, validate it independently, and install it atomically |
| Failure behavior | Retain the validated last-known-good bundle; a sealed guest without a valid fresh or cached bundle fails closed |
| Monitoring exceptions | Admit only the exact external health-check source addresses rendered by OpenTofu ahead of geographic filtering |
| IPv6 | Outside the Tier R v1 admission boundary |
TCP/25 uses a separate `smtp-deny/v1` transport policy.
| Boundary | Contract |
| --- | --- |
| Evidence | Relay-observed IPv4 source, transport reason, observation window, count, and expiry only |
| Excluded evidence | Message content, recipients, DNSBL, SPF, DKIM, DMARC, authentication, and spam classification |
| Bounds | At most 4,096 covered addresses, 1,024 intervals, a 10-minute observation window, 20 observations, and a 24-hour lifetime |
| Failure behavior | Invalid, absent, expired, oversized, or unreachable current and cached state installs an empty deny set |
| Decay | Each element expires no later than its declared expiry; omission from the next authorized publication removes it promptly |
| Isolation | Separate prefix, schema, publisher identity, cache, helper, timer, nftables table, counters, and alarms from `web-allow` |
`syn-rate` evidence is a 60-second local snapshot of canonical IPv4, cumulative drop count, and expiry from per-element counters on the existing ten-minute `smtp_syn_quarantine_v4`; the reader is unprivileged with only `CAP_NET_ADMIN`, and the 65,536-source enforcement capacity remains independent from bundle caps.
### 4.5 Home-Side Routing
Because client source addresses are preserved, the home-side node must return relay ingress responses through the OCI tunnel. Replies must not use the normal LAN default route through OPNsense.
The active home-side workload routes configured LAN and WireGuard client CIDRs through the main table before selecting table `51820` for other replies sourced from Gateway R at `10.104.40.127` or the mail target at `10.66.153.202`. It reconciles stale local state before competing for ownership and removes the link, routes, rules, and relay-owned nftables table when ownership ends. This keeps preserved public client destinations on the OCI return path without capturing internal connections to the same Gateway identity.
### 4.6 WireGuard Source Reconciliation
```mermaid
stateDiagram-v2
[*] --> Poll
Poll --> Retain: API, TLS, or address validation failure
Poll --> Observe: public WAN IPv4 differs
Observe --> Poll: candidate changes
Observe --> Update: same candidate observed twice
Update --> Retain: OCI update or response validation failure
Update --> Probe: exact NSG rule replaced
Probe --> Poll: relay TCP/443 recovers within 90s
Probe --> Retain: recovery probe fails
```
| Control | Contract |
| --- | --- |
| Runtime | One digest-pinned, non-root controller on a control-plane node with no service-account token |
| OPNsense polling | Read the interface overview every ten seconds and select the interface named `WAN` |
| Trust | Validate HTTPS against the dedicated `pki_opnsense` root; never disable certificate or hostname verification |
| Credential scope | Use the read-only OPNsense interface-watcher API identity projected by Vault Secrets Operator |
| Address validation | Accept only a public, unicast IPv4 address |
| Debounce | Require two consecutive observations of the same new address |
| OCI mutation | Replace only the identified stateful UDP WireGuard NSG rule while preserving its ID, port, protocol, and description |
| OCI response check | Require OCI to return exactly that rule with the requested `/32` and expected immutable fields |
| Recovery check | Require TCP/443 on the relay address to recover within 90 seconds |
| Failure behavior | Retain the current rule on polling or validation failure; retry a failed candidate without widening the source boundary |
| OpenTofu ownership | Declare the rule and ignore only controller-owned source drift |
## 5. Cilium Interaction Requirements
The home-side tunnel must avoid conflicting with Cilium-managed networking.
| Requirement | Rationale |
| --- | --- |
| Do not modify Cilium-owned iptables or nftables chains | Cilium owns service, LB, host firewall, and proxy packet paths. |
| Do not flush host firewall rulesets | Flushing can remove Cilium-managed policy state. |
| Use the dedicated WireGuard interface `wg-oci0` | Avoids collision with Cilium WireGuard encryption. |
| Use a dedicated routing table and priority | Keeps relay return routing separate from node default routing. |
| Decouple `AllowedIPs` from route installation | Preserved client sources require `0.0.0.0/0` peer association without a tunnel default route. |
| Do not perform cluster-side DNAT | OCI owns the only DNAT layer and leaves Cilium service handling unchanged. |
| Preserve packet-capture access for diagnostics | Cilium Gateway traffic can traverse host, TPROXY, and Envoy paths in non-obvious ways. |
```mermaid
flowchart LR
Lease["Lease\n30s duration"] --> Active["Active replica\nwg-oci0 owner"]
Lease -.-> Standby["Prepared standby\nno network state"]
Active --> State["wg-oci0 + table 51820\nlocal return rules + nftables"]
Active -->|"refresh every 5s"| Fence["Outer UDP fence\n15s timeout"]
Active -->|"loss"| Expiry["Fence expires"]
Expiry --> Blocked["Stale peer blocked"]
Blocked -->|"Lease release or expiry"| Standby
Standby --> Active
```
| Control | Contract |
| --- | --- |
| Runtime | Two anti-affined `hostNetwork` replicas; digest-pinned static scratch image |
| Privilege | Root + `NET_ADMIN`; read-only root filesystem; private key mode `0400` |
| Ownership | Exact-name Lease; RBAC permits only `get`, `update`, and `patch` on that Lease |
| Fencing | Outer UDP requires a 15-second nftables set element refreshed only by the owner |
| Reconciliation | Startup removes stale local relay state before leader election |
| Unhealthy-term backoff | Consecutive unhealthy ownership terms retry exponentially from five seconds to a 60-second cap |
| Reverse-path filtering | All K3s hosts persist `net.ipv4.conf.all.rp_filter=0` and `net.ipv4.conf.default.rp_filter=2` before the relay starts |
| Inner ingress | `wg-oci0` admits Gateway R TCP/80 and TCP/443 plus the mail target TCP/25, TCP/465, and TCP/993; all else drops |
| Peer identity | One shared identity and one active endpoint; active-active is prohibited |
| `AllowedIPs` | `0.0.0.0/0` on both peers; automatic route installation disabled |
| Routes | Configured internal destinations use the main table; other replies from Gateway R or the mail target select table `51820` |
## 6. Bootstrap and Secret Management
### 6.1 OCI VM Bootstrap
The live relay uses a digest-identified custom image built from a pinned Oracle Linux 10.1 AArch64 platform image. Packer owns the disposable builder and provisioning lifecycle. A lockfile-pinned OCI Python SDK seals the provisioned instance. The relay runs on `VM.Standard.A1.Flex` with 1 OCPU, 4 GB RAM, and a 50 GB boot volume. OpenTofu-rendered cloud-init writes only the runtime configuration and starts the preinstalled bootstrap unit.
| Task | Requirement |
| --- | --- |
| Build runtime packages | Packer upgrades the base and installs pinned WireGuard, nftables, traffic-control, and journal-upload packages |
| Build relay binaries | Packer installs digest-recorded guest, web admission, SMTP deny, SMTP evidence reader, and `vlagent` binaries |
| Enable IP forwarding | Required for L4 relay behavior |
| Initialize relay key | Read the OCI-generated seed and derive the same private key on every boot |
| Publish public key | Update its OCI secret only when the current value differs |
| Render WireGuard config | Use OCI private key and embedded home-side public key |
| Render nftables config | DNAT web ports to Gateway R and mail ports to the mail target over WireGuard |
| Start services | Enable WireGuard and nftables through systemd |
| Package provenance | Record exact installed package NEVRAs, binary SHA-256 digests, base-image OCID, and image-input revision digest |
| Image sealing | Record a pending image identity before waiting; publish the final manifest only after OCI reports `AVAILABLE` |
| Firewall switch | Validate nftables before stopping `firewalld`; load the ruleset atomically |
| Shaping lifecycle | WireGuard `PostUp` and `PreDown` own traffic-control restoration |
| Remote diagnostics | OCI Run Command remains self-scoped; SSH is not a recovery dependency |
| Admission refresh | Independent helpers refetch, validate, cache, and atomically install `web-allow` and `smtp-deny` |
| Failure behavior | Bootstrap and web admission fail closed; invalid SMTP-deny state installs an empty deny set |
### 6.2 Secret Boundaries
| Material | Authoritative storage | OpenTofu state contract |
| --- | --- | --- |
| OCI relay WireGuard seed | OCI Secrets | Must not be stored in state |
| OCI relay WireGuard public key | OCI Secrets publication bundle | Permitted in state |
| Home-side WireGuard private key | Vault | Must not be stored in state |
| Home-side WireGuard public key | Vault relay configuration rendered into cloud-init | Permitted in state |
| OPNsense interface API key | Vault KV | Must not be stored in state |
| OPNsense HTTPS root CA | Vault `pki_opnsense` | Public trust material |
| WAN controller OCI private key | Vault KV | Must not be stored in state |
| WAN controller OCI public key | Vault relay configuration | Permitted in state |
OpenTofu creates the seed, public-key secret, and IAM policy. The relay derives the private key in memory and publishes only its public key. Private key bytes do not enter Terraform state or OCI secret versions.
### 6.3 OCI IAM
The relay instance-principal dynamic groups require both the relay compartment and their exact `quorate_relay` authorization tags.
| Principal | Permission | Scope |
| --- | --- | --- |
| Relay instance | Read secret bundles | Exact WireGuard seed, published-key, and log-ingest secrets |
| Relay instance | Update secret | `SECRET_UPDATE` on the published-key secret only |
| Relay instance | Read objects | Validated admission bucket only |
| Relay instance | Execute Run Command | Self-scoped instance-agent command execution only |
| Web-allow publisher | Create and read immutable objects | `web-allow/v1/versions/*` and migration-only `admission/v1/versions/*` |
| Web-allow publisher | Create, overwrite, and read current objects | Exact web-allow and migration-only legacy current objects |
| SMTP-deny publisher | Create and read immutable objects | `smtp-deny/v1/versions/*` only |
| SMTP-deny publisher | Create, overwrite, and read current object | Exact SMTP-deny current object only |
| WAN controller | Update network security group security rules | Exact operation in the relay compartment |
| Object Storage service | Delete retained object versions and incomplete uploads | Admission bucket lifecycle only |
| All relay principals | No secret listing or KMS key management | Prevent lateral access to unrelated secrets and keys |
### 6.4 Observability and Diagnostics
```mermaid
flowchart LR
Guest[Sealed guest units] --> Journal[Dedicated systemd journal namespace]
Journal --> Upload[systemd-journal-upload]
Upload --> Vlagent[vlagent bounded buffer]
Vlagent --> VictoriaLogs[Authenticated VictoriaLogs ingestion]
VictoriaLogs --> Dashboard[Grafana Tier R dashboard]
Guest --> Metrics[WireGuard-only Prometheus endpoint]
Metrics --> VictoriaMetrics
VictoriaMetrics --> Dashboard
Health[OCI external health checks] --> Alarms[OCI alarms and notifications]
Console[OCI serial console capture] -. manual break glass .-> Guest
Run[OCI Run Command] -. control-plane break glass .-> Guest
```
| Control | Contract |
| --- | --- |
| Guest logs | Send the dedicated relay journal through bounded `vlagent` to VictoriaLogs |
| Guest metrics | Expose relay health, freshness, capacity, drop, evidence, and observer metrics to VictoriaMetrics over WireGuard |
| Conntrack records | Log sampled transport-only TCP completions matching a declared port and post-DNAT target; cap emission rate and burst |
| Rejected traffic | Keep nftables drop counters; do not retain OCI NSG rejects |
| Buffer bounds | Limit `vlagent` to 64 MiB memory, 512 MiB disk, and 1 MiB/s upload |
| Availability boundary | Observability failure must not affect relay forwarding |
| External monitoring | Use OCI Health Checks, Monitoring alarms, and Notifications for public web and enabled mail reachability |
| Break glass | Keep OCI serial-console capture and self-scoped Run Command without SSH |
## 7. Cloud Requirements
| Resource | Requirement |
| --- | --- |
| Compute | `VM.Standard.A1.Flex`, 1 OCPU, 4 GB RAM, and 50 GB boot volume |
| OS | Sealed custom image built from a pinned Oracle Linux 10.1 AArch64 platform image |
| Public IP | Static reserved public IPv4 |
| Firewall | TCP/80 and TCP/443 are global; mail ports use explicit source admissions; WireGuard UDP uses the current OPNsense WAN `/32` |
| Rate limit | Pre-conntrack guards and traffic-control shaping protect the relay-to-home path |
| Admission storage | Private versioned Object Storage prefixes with independent web and SMTP-deny ownership and lifecycle |
| Monitoring | OCI monitors external reachability; VictoriaLogs and VictoriaMetrics receive guest telemetry; OCI tracks cost |
| DNS | Services assigned to Tier R use Cloudflare gray-cloud A records for the OCI reserved address |
The relay VM is single-purpose and must not host application logic, terminate TLS, or provide general-purpose bastion access.
## 8. Kubernetes Requirements
| Resource | Requirement |
| --- | --- |
| Gateway | Dedicated `gateway-relay` / Gateway R at `10.104.40.127` |
| Mail forwarding target | Source-preserving LoadBalancer at `10.66.153.202` for TCP/25, TCP/465, and TCP/993 |
| TLS transparency | Preserve the TLS byte stream through the declared target |
| Network policy | Gateway ingress policy must allow preserved client source traffic |
| Host policy | Every schedulable node permits the relay tunnel and Gateway R packet path |
| Scheduling | Two tunnel replicas use node anti-affinity |
| Ownership | Exact-name Lease selects one active owner; the standby holds no relay network state |
| Confinement | Relay-owned nftables admits only declared Gateway R and mail destination/port pairs from `wg-oci0` |
| WAN controller | One control-plane replica, read-only root filesystem, no ingress, and egress to cluster DNS, exact OPNsense HTTPS, and world TCP/443 for OCI API |
| Secret sync | Vault Secrets Operator projects OPNsense credentials and CA, controller OCI credentials, and the relay health endpoint |
| Observability | Lease state, nftables counters, Hubble, relay journals, VictoriaLogs, controller logs, and packet captures support diagnostics |
## 9. Acceptance Checks
| Test | Expected Result |
| --- | --- |
| Public DNS resolves service to OCI IP | Client connects to OCI, not residential WAN |
| TCP/80 redirect | Gateway HTTP listener returns the expected HTTPS redirect |
| TCP/443 TLS | Certificate is served by Cilium Gateway, not OCI |
| Mail TLS pass-through | TCP/25, TCP/465, and TCP/993 TLS reaches the mail target unchanged |
| Port isolation | Undeclared destination and port combinations drop before forwarding |
| Source IP preservation | Application or Gateway logs observe real client IP |
| Return path | Node replies traverse `wg-oci0` to OCI |
| Rate limiting | Sustained ingress above the configured cap is shaped or dropped before saturating WireGuard |
| OPNsense visibility | OPNsense observes only outbound WireGuard traffic, not public inbound service traffic |
| Cilium stability | Cilium services, Gateway, host firewall, and pod networking remain healthy |
| Relay reboot | VM re-derives its key, restores services and shaping, and resumes forwarding |
| Node reboot | Home-side tunnel and policy routing are restored declaratively |
| Abrupt owner loss | Fence precedes ownership transfer; public service resumes through the new owner |
| Old-owner rejoin | Startup reconciliation removes stale state before leader election |
| Tunnel negative traffic | Undeclared Gateway R and mail destination/port pairs are dropped on `wg-oci0` |
| Admission failure | Web surfaces fail closed while TCP/25 installs an empty fail-open SMTP deny set |
| Pressure isolation | Web, SMTP, authenticated mail, and WireGuard control retain independent budgets |
| OPNsense API failure | The controller retains the existing WireGuard source rule and retries without widening it |
| WAN address change | Two matching observations replace the NSG source with only the new public IPv4 `/32` |
| OCI response mismatch | The controller rejects the reconciliation and does not mark the candidate applied |
| WAN recovery | Relay TCP/443 becomes reachable after the exact NSG rule replacement |
## 10. Non-Goals
| Non-Goal | Reason |
| --- | --- |
| L7 proxying on OCI | The relay forwards L4 traffic without owning service TLS or HTTP routing. |
| Cloud WAF | This design is an L4 relay, not a Cloudflare Tunnel replacement for WAF-managed services. |
| General VPN access into the LAN | The relay tunnel reaches only the ingress target required for public services. |
| Replacing Tier O | Cloudflare Tunnel remains the path for services that terminate at Cloudflare. |
| Replacing internal Tier I or Direct Tier D | Internal LAN and WireGuard access remain separate paths. |
## 11. Implementation Decisions
| Area | Decision |
| --- | --- |
| Gateway identity | A dedicated `gateway-relay` / Gateway R owns `10.104.40.127`. |
| Home-side ownership | Two anti-affined replicas use one exact-name Lease and an expiring outer-UDP fence. |
| Node selection | Kubernetes schedules the replicas; the Lease selects one active owner. |
| Return routing | Configured LAN and WireGuard destinations use the main table; other replies from `10.104.40.127/32` or `10.66.153.202/32` select table `51820`. |
| Cluster-side NAT | OCI performs DNAT; the cluster does not add another DNAT layer. |
| WireGuard exposure | The in-cluster WAN controller keeps OCI ingress restricted to the current home WAN `/32`. |
| DNS ownership | Cloudflare records consume the OCI reserved address from relay remote state. |
| IPv6 | IPv6 is outside the v1 relay boundary. |
## 12. Provider and Cost Boundary
| Control | Contract |
| --- | --- |
| Provider and region | OCI \*\*\*\*\*\*\*\* (`********`) |
| Compute allocation | `VM.Standard.A1.Flex`, 1 OCPU, 4 GB RAM |
| Storage allocation | One 50 GB boot volume, one final sealed custom image, and the private admission bucket |
| Public address | One reserved IPv4 address |
| Cost scope | All relay OCI resources remain in the dedicated relay compartment or are explicitly linked tenancy-level IAM resources |
| Budget | A declarative monthly 35-unit compartment budget alerts at actual 50%, 80%, and 100%, and forecast 100% |
| Budget semantics | Alerts provide visibility and do not enforce a hard spending limit |
| Storage lifecycle | Web-allow and SMTP-deny versions expire after 30 days; incomplete multipart uploads abort after one day |
| Excluded resources | No managed load balancer, block volume, volume backup, or boot-volume backup |

201
docs/spec/project-delivery-spec.md Normal file
View file

@ -0,0 +1,201 @@
# Project Delivery Specification
Project delivery reconciles Forgejo source into published artifacts. A `ProjectDelivery` declares the desired result, a controller resolves continuous refs and accepts declared point revisions, and a cluster-owned Argo `WorkflowTemplate` defines all execution.
## Delivery declaration
```mermaid
flowchart LR
D[ProjectDelivery]
C[Delivery controller]
T[Cluster-owned WorkflowTemplate]
W[Workflow]
V[Vault and VSO]
A[Build cache]
P[Declared publication targets]
D --> C
C -->|template reference and immutable input| W
T --> W
V -->|stage-scoped Secrets| W
W <-->|cache pull, export, and cleanup| A
W --> P
```
```yaml
apiVersion: delivery.quorate.dev/v1alpha1
kind: ProjectDelivery
metadata:
name: example
spec:
source:
repository: https://git.quorate.dev/example/project
continuous:
ref: refs/heads/main
paths: [packages/example]
targets:
- kind: QuayOCI
repository: quorate/project
- kind: ExternalOCI
registry: docker.io
repository: example/project
- kind: ForgejoRelease
repository: https://git.quorate.dev/example/project
workflowTemplateRef:
name: example-delivery
```
`source` contains exactly one delivery mode:
| Mode | Selector | Trigger identity |
| ------------ | --------------------------------------------------------------------- | ------------------------------- |
| `continuous` | One complete ref; optional exact paths and positive-duration debounce | Commit SHA or selected Git tree |
| `point` | A non-empty set of full commit SHAs | Full commit SHA |
`continuous.paths` contains exact repository-relative file or directory paths. When present, the trigger identity is the selected Git tree state rather than the commit SHA.
Point delivery declares immutable revisions:
```yaml
source:
repository: https://git.quorate.dev/example/project
point:
revisions:
- 0123456789abcdef0123456789abcdef01234567
- 89abcdef0123456789abcdef0123456789abcdef
```
`status.points` records point revisions, Workflow identities and phases, and delivery results.
Point revisions are consumed directly without ref enumeration or resolution.
`source.repository` identifies the Forgejo repository containing the selected source. Its authority is declaration data rather than an admission boundary.
`targets` is a non-empty set of workflow-defined publication declarations. The controller treats each declaration as opaque JSON and passes it unchanged. The referenced `WorkflowTemplate` defines and validates the target kinds and fields it supports.
`workflowTemplateRef` names a `WorkflowTemplate` in the `ProjectDelivery` namespace. Admission rejects cross-namespace references and malformed source selectors.
`delivery.quorate.dev/retry` names an owned terminal Workflow.
## Qualification interface
| Field | Type | Default | Model binding |
| ----------------------------------------- | ------------------------------------ | ------------ | -------------------------------------- |
| `spec.qualification` | Optional object | Absent | Configured resources form `Deliveries` |
| `workflowTemplateRef.name` | Same-namespace WorkflowTemplate name | Required | Workflow template |
| `mode` | `Qualification!Policies` | `ManualOnly` | `policy[delivery]` |
| `interval` | Duration | `0s` | `Interval[delivery]` |
| `delivery.quorate.dev/qualify` annotation | `<build-workflow-name>/<request-id>` | Absent | `request[delivery]` |
| Workflow interface | Value |
| ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Input: `source-repository`, `source-ref`, `source-revision`, `targets` | Selected successful build's parameters |
| Input: `delivery-result` | Selected build's opaque publication result |
| Model `Input(delivery, build)` | Tuple of those parameters and publication result |
| Output: `qualification-result` | Workflow-defined result identifying tested source, artifacts, and retained reports |
| Result interpretation | Qualification WorkflowTemplate |
## Reconciliation and execution
### Authority
| Artifact | Authority |
| -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| Project Delivery specification | Declaration and status schema, Workflow interface, objectives, and security |
| [Project Delivery state model](project-delivery/ProjectDelivery.tla) | Attempt eligibility and history, controller submission, Argo execution synchronization, persistence, lifecycle, and build concurrency |
| [Qualification state model](project-delivery/Qualification.tla) | Qualification eligibility, annotation consumption, input capture, interval, and global execution slot |
### Build workflow interface
The diagram is the standard `ProjectDelivery` build workflow DAG shape; deviation from it requires evidence that substantiates the deviation.
```mermaid
flowchart LR
T[Toolchain pull] --> B[Build]
C[Cache pull] --> B
S[Source pull] --> B
B --> E[Cache export and cleanup]
B --> P[Publish to declared targets]
```
| Parameter | Value |
| ------------------- | ------------------------------------------------------------- |
| `source-repository` | Declared repository URL |
| `source-ref` | Complete selected continuous ref, or empty for point delivery |
| `source-revision` | Resolved full commit SHA |
| `targets` | Declaration-provided JSON target set |
### Delivery objectives
Measure delivery elapsed time from Workflow `startedAt` to terminal `finishedAt`, excluding intervals when waiting on WAN is the sole active work.
| Cache state | Requirement |
| ----------- | -------------------------------------------- |
| Cold | No reusable build state; within 900 seconds. |
| Warm | Reusable build state; within 300 seconds. |
The `WorkflowTemplate` owns the DAG:
| Concern | Owner |
| ----------------------------------------------------- | -------------------------- |
| Stages, actions, dependencies, retries | Argo `WorkflowTemplate` |
| Build and assembly image selection | Argo `WorkflowTemplate` |
| Action and builder image preparation | Cluster |
| Stage Secret references | Argo `WorkflowTemplate` |
| Secret material and projection | Vault and VSO |
| Stage egress | Kubernetes `NetworkPolicy` |
| Continuous source selection and duplicate suppression | Delivery controller |
| Cache creation, invalidation, cleanup, and layout | Argo `WorkflowTemplate` |
| Cache contract | Value |
| -------------- | ------------------------------------------------ |
| Prefix | `projects/<project>/<workflow>/cache/<cache>/` |
| Manifest | `_latest.json` |
| Pull | `cacheVersion` and `inputHashes` match |
| Export | Replace `_latest.json` |
| Cleanup | Remove unreferenced payloads during cache export |
| Controller | No cache inspection or enforcement |
| `_latest.json` field | Type | Value |
| -------------------- | ------- | --------------------------------------- |
| `schemaVersion` | Integer | `1` |
| `cacheVersion` | String | Workflow cache version |
| `inputHashes` | Object | Repository-relative path to SHA-256 map |
| `payloadKey` | String | `objects/<payload-sha256>.tar.zst` |
### Workflow security requirements
Every delivery `WorkflowTemplate` and its supporting namespace resources meet these requirements:
| Boundary | Requirement |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Ownership | Commands, images, target interpretation, Secret references, and pod security are cluster-owned |
| Supply chain | Action and executor images use immutable digests of latest stable releases; deprecated APIs and components are forbidden; source uses the selected full SHA |
| Runtime | `gvisor` where non-cluster-owned code executes; restricted Pod Security; non-root; `RuntimeDefault`; no privilege escalation or capabilities; read-only root filesystem |
| Host | No privileged container, host namespace, `hostPath`, device, or runtime socket |
| Kubernetes | Action containers receive no Kubernetes credential; executor tokens are bound, short-lived, and limited to Workflow task results and artifact GC |
| Delivery stages | A stage is a Pod boundary only when its network, credential, input, or output access differs. Actions with identical access execute in the same Pod. |
| Network | Default deny; each stage permits only executor traffic, artifact handoff, and its declared endpoints |
| Credentials | Vault to VSO to one read-only stage mount; never parameters, environment, commands, logs, or artifacts |
| Handoff | Stages exchange state through the Workflow PVC; publishers transfer immutable output |
| Cache | Cache reads and writes use stage-scoped credentials |
| Ephemeral state | One ephemeral PVC is the sole working data plane for the build Workflow lifetime |
| Build interface | Repository supplies source only; OCI delivery uses cluster-owned apko configuration |
| Resources | CPU requests, memory requests, and memory limits on every pod |
Publication credentials are destination-native, restricted to the declared destination, and mounted only in the publication stage.
Project-delivery credentials are never shared with Smeltery.
## Scope
Delivery succeeds when every declared target reports successful publication of the Workflow's immutable output.
| Included | Excluded |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| Source-ref observation and immutable revision selection | Repository visibility |
| Duplicate suppression and Workflow creation | Vulnerability policy and scan gating |
| Project-specific build, test, assembly, and publication DAGs | Candidate repositories and promotion |
| Manual and interval-controlled qualification of published artifacts | Deployment approval and rollout gating |
| Publication digests, checksums, URLs, and Workflow status | Deployment and Flux manifest changes |
| Stage isolation, credentials, and network policy | Per-repository publisher identities and grants; published artifact retention and registry garbage collection |

16
docs/spec/project-delivery/Liveness.cfg Normal file
View file

@ -0,0 +1,16 @@
SPECIFICATION LiveSpec
CONSTANTS
Deliveries = {delivery1}
Templates = {template1}
Identities = {identity1, identity2}
Revisions = {revision1, revision2}
Others = {other1}
InitialTemplateRef <- SharedTemplateRef
InitialMode <- ContinuousMode
INVARIANTS TypeOK NoConcurrentRuns TemplateAvailable PreparedAvailable CollectedSaved
PROPERTIES TemplateStable AdmissionReady PublishDrained PublishLatest RecordOnce PublishProgress DebounceProgress DrainProgress AttemptProgress ReleaseProgress RecordProgress DeliveryProgress
CHECK_DEADLOCK FALSE
CONSTRAINT LiveDomain

426
docs/spec/project-delivery/ProjectDelivery.tla Normal file
View file

@ -0,0 +1,426 @@
---- MODULE ProjectDelivery ----
EXTENDS FiniteSets, Naturals
CONSTANTS Deliveries, Templates, Identities, Revisions, Others, InitialTemplateRef, InitialMode
Continuous == "Continuous"
Point == "Point"
Modes == {Continuous, Point}
Absent == "Absent"
Queued == "Queued"
Running == "Running"
Terminal == "Terminal"
Collected == "Collected"
\* Retry successors use fresh identities without consuming a source identity again.
Attempts == Deliveries \X Modes \X Identities
Slots == Attempts \cup Others
Phases == {Absent, Queued, Running, Terminal, Collected}
VARIABLES mode, templateRef, eligible, persisted, recorded, phase, attemptTemplate
VARIABLES wantedRev, templateRev, settled, attemptRev, availableRev
VARIABLES prepared, publishRev
vars == <<mode, templateRef, eligible, persisted, recorded, phase, attemptTemplate,
wantedRev, templateRev, settled, attemptRev, availableRev, prepared, publishRev>>
ASSUME /\ Deliveries /= {}
/\ Templates /= {}
/\ Identities /= {}
/\ Revisions /= {}
/\ Attempts \cap Others = {}
/\ InitialTemplateRef \in [Deliveries -> Templates]
/\ InitialMode \in [Deliveries -> Modes]
DeliveryOf(attempt) == attempt[1]
ModeOf(attempt) == attempt[2]
RetryPairs ==
{pair \in Attempts \X Attempts :
/\ pair[1] /= pair[2]
/\ DeliveryOf(pair[1]) = DeliveryOf(pair[2])
/\ ModeOf(pair[1]) = ModeOf(pair[2])}
SelectedAttempt(delivery) ==
<<delivery, mode[delivery], eligible[delivery][mode[delivery]]>>
RunningFor(template) ==
{attempt \in Slots :
/\ phase[attempt] = Running
/\ attemptTemplate[attempt] = {template}}
ActiveFor(template) ==
{attempt \in Slots :
/\ phase[attempt] \in {Queued, Running}
/\ attemptTemplate[attempt] = {template}}
TypeOK ==
/\ mode \in [Deliveries -> Modes]
/\ templateRef \in [Deliveries -> Templates]
/\ eligible \in [Deliveries -> [Modes -> Identities]]
/\ persisted \in SUBSET Attempts
/\ recorded \in [Deliveries -> [Modes -> SUBSET Attempts]]
/\ \A delivery \in Deliveries, sourceMode \in Modes :
/\ Cardinality(recorded[delivery][sourceMode]) <= 1
/\ \A attempt \in recorded[delivery][sourceMode] :
/\ DeliveryOf(attempt) = delivery
/\ ModeOf(attempt) = sourceMode
/\ attempt \in persisted
/\ phase \in [Slots -> Phases]
/\ availableRev \in [Templates -> SUBSET Revisions]
/\ wantedRev \in [Templates -> Revisions]
/\ templateRev \in [Templates -> Revisions]
/\ settled \in [Templates -> BOOLEAN]
/\ attemptRev \in [Slots -> SUBSET Revisions]
/\ attemptTemplate \in [Slots -> SUBSET Templates]
/\ prepared \in [Slots -> SUBSET (Templates \X Revisions)]
/\ publishRev \in [Templates -> SUBSET Revisions]
/\ \A template \in Templates : Cardinality(publishRev[template]) <= 1
/\ \A attempt \in Slots :
/\ Cardinality(prepared[attempt]) <= 1
/\ (phase[attempt] = Absent) = (attemptTemplate[attempt] = {})
/\ phase[attempt] /= Absent => Cardinality(attemptTemplate[attempt]) = 1
/\ (phase[attempt] = Absent) = (attemptRev[attempt] = {})
/\ phase[attempt] /= Absent => Cardinality(attemptRev[attempt]) = 1
Init ==
/\ mode = InitialMode
/\ templateRef = InitialTemplateRef
/\ eligible =
[delivery \in Deliveries |->
[sourceMode \in Modes |-> CHOOSE identity \in Identities : TRUE]]
/\ persisted = {}
/\ recorded =
[delivery \in Deliveries |-> [sourceMode \in Modes |-> {}]]
/\ phase = [attempt \in Slots |-> Absent]
/\ attemptTemplate = [attempt \in Slots |-> {}]
/\ templateRev = [template \in Templates |-> CHOOSE revision \in Revisions : TRUE]
/\ availableRev = [template \in Templates |-> {templateRev[template]}]
/\ wantedRev = templateRev
/\ settled = [template \in Templates |-> TRUE]
/\ attemptRev = [attempt \in Slots |-> {}]
/\ prepared = [attempt \in Slots |-> {}]
/\ publishRev = [template \in Templates |-> {}]
ChangeEligibility(delivery, sourceMode, identity) ==
/\ identity /= eligible[delivery][sourceMode]
/\ eligible' =
[eligible EXCEPT ![delivery][sourceMode] = identity]
/\ UNCHANGED <<mode, templateRef, persisted, recorded, phase, attemptTemplate,
wantedRev, templateRev, settled, attemptRev, availableRev, prepared, publishRev>>
ChangeMode(delivery, sourceMode) ==
/\ sourceMode /= mode[delivery]
/\ mode' = [mode EXCEPT ![delivery] = sourceMode]
/\ UNCHANGED <<templateRef, eligible, persisted, recorded, phase, attemptTemplate,
wantedRev, templateRev, settled, attemptRev, availableRev, prepared, publishRev>>
ChangeTemplateRef(delivery, template) ==
/\ template /= templateRef[delivery]
/\ templateRef' = [templateRef EXCEPT ![delivery] = template]
/\ UNCHANGED <<mode, eligible, persisted, recorded, phase, attemptTemplate,
wantedRev, templateRev, settled, attemptRev, availableRev, prepared, publishRev>>
CreationEnabled(delivery) ==
LET attempt == SelectedAttempt(delivery)
IN /\ recorded[delivery][mode[delivery]] = {}
/\ attempt \notin persisted
/\ phase[attempt] \in {Absent, Collected}
/\ prepared[attempt] = {}
/\ wantedRev[templateRef[delivery]] = templateRev[templateRef[delivery]]
/\ settled[templateRef[delivery]]
Prepare(delivery) ==
LET attempt == SelectedAttempt(delivery)
template == templateRef[delivery]
IN /\ CreationEnabled(delivery)
/\ prepared' =
[prepared EXCEPT ![attempt] = {<<template, templateRev[template]>>}]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, recorded, phase,
attemptTemplate, wantedRev, templateRev, settled,
attemptRev, availableRev, publishRev>>
PrepareRetry(delivery, prior, attempt) ==
LET template == templateRef[delivery]
IN /\ prior \in Attempts
/\ attempt \in Attempts
/\ DeliveryOf(prior) = delivery
/\ DeliveryOf(attempt) = delivery
/\ ModeOf(prior) = ModeOf(attempt)
/\ prior \in persisted
/\ phase[prior] \in {Terminal, Collected}
/\ attempt \notin persisted
/\ phase[attempt] \in {Absent, Collected}
/\ prepared[attempt] = {}
/\ \A sourceMode \in Modes : recorded[delivery][sourceMode] = {}
/\ wantedRev[template] = templateRev[template]
/\ settled[template]
/\ prepared' =
[prepared EXCEPT ![attempt] = {<<template, templateRev[template]>>}]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, recorded, phase,
attemptTemplate, wantedRev, templateRev, settled,
attemptRev, availableRev, publishRev>>
\* An API call can finish after a later observation without rereading admission gates.
Create(attempt) ==
LET binding == CHOOSE value \in prepared[attempt] : TRUE
IN /\ attempt \in Slots
/\ prepared[attempt] /= {}
/\ IF attempt \in Attempts
THEN /\ phase[attempt] \in {Absent, Collected}
/\ attempt \notin persisted
ELSE phase[attempt] \in {Absent, Terminal, Collected}
/\ phase' = [phase EXCEPT ![attempt] = Queued]
/\ attemptTemplate' = [attemptTemplate EXCEPT ![attempt] = {binding[1]}]
/\ attemptRev' = [attemptRev EXCEPT ![attempt] = {binding[2]}]
/\ prepared' = [prepared EXCEPT ![attempt] = {}]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, recorded,
wantedRev, templateRev, settled, availableRev, publishRev>>
Start(attempt) ==
/\ attempt \in Slots
/\ phase[attempt] = Queued
/\ \A template \in attemptTemplate[attempt] : RunningFor(template) = {}
/\ phase' = [phase EXCEPT ![attempt] = Running]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, recorded, attemptTemplate,
wantedRev, templateRev, settled, attemptRev, availableRev, prepared, publishRev>>
Persist(delivery, attempt) ==
/\ attempt \in Attempts
/\ DeliveryOf(attempt) = delivery
/\ attempt \notin persisted
/\ phase[attempt] /= Absent
/\ recorded[delivery][ModeOf(attempt)] = {}
/\ persisted' = persisted \cup {attempt}
/\ recorded' =
[recorded EXCEPT ![delivery][ModeOf(attempt)] = {attempt}]
/\ UNCHANGED <<mode, templateRef, eligible, phase, attemptTemplate,
wantedRev, templateRev, settled, attemptRev, availableRev, prepared, publishRev>>
Complete(attempt) ==
/\ attempt \in Slots
/\ phase[attempt] = Running
/\ phase' = [phase EXCEPT ![attempt] = Terminal]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, recorded, attemptTemplate,
wantedRev, templateRev, settled, attemptRev, availableRev, prepared, publishRev>>
Reject(attempt) ==
/\ attempt \in Slots
/\ phase[attempt] = Queued
/\ phase' = [phase EXCEPT ![attempt] = Terminal]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, recorded, attemptTemplate,
wantedRev, templateRev, settled, attemptRev, availableRev, prepared, publishRev>>
Release(delivery, sourceMode) ==
/\ recorded[delivery][sourceMode] /= {}
/\ \A attempt \in recorded[delivery][sourceMode] :
phase[attempt] \in {Terminal, Collected}
/\ recorded' = [recorded EXCEPT ![delivery][sourceMode] = {}]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, phase, attemptTemplate,
wantedRev, templateRev, settled, attemptRev, availableRev, prepared, publishRev>>
Collect(attempt) ==
/\ attempt \in Attempts
/\ phase[attempt] = Terminal
/\ attempt \in persisted
/\ phase' = [phase EXCEPT ![attempt] = Collected]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, recorded, attemptTemplate,
wantedRev, templateRev, settled, attemptRev, availableRev, prepared, publishRev>>
\* Other slots represent fresh workflow executions and can be reused after completion.
PrepareOther(attempt, template) ==
/\ attempt \in Others
/\ phase[attempt] \in {Absent, Terminal, Collected}
/\ prepared[attempt] = {}
/\ wantedRev[template] = templateRev[template]
/\ settled[template]
/\ prepared' = [prepared EXCEPT ![attempt] = {<<template, templateRev[template]>>}]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, recorded, phase,
attemptTemplate, wantedRev, templateRev, settled,
attemptRev, availableRev, publishRev>>
\* Requests record observed bundles; writes preceding observation leave wantedRev unchanged.
RequestTemplate(template, revision) ==
/\ revision /= wantedRev[template]
/\ wantedRev' = [wantedRev EXCEPT ![template] = revision]
/\ settled' = [settled EXCEPT ![template] = FALSE]
/\ publishRev' = [publishRev EXCEPT ![template] = {}]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, recorded, phase,
attemptTemplate, templateRev, attemptRev, availableRev, prepared>>
\* Settle abstracts expiration of the configured quiet interval after the latest change.
Settle(template) ==
/\ ~settled[template]
/\ settled' = [settled EXCEPT ![template] = TRUE]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, recorded, phase,
attemptTemplate, wantedRev, templateRev, attemptRev, availableRev, prepared, publishRev>>
PreparePublish(template) ==
/\ wantedRev[template] /= templateRev[template]
/\ settled[template]
/\ ActiveFor(template) = {}
/\ publishRev[template] = {}
/\ publishRev' = [publishRev EXCEPT ![template] = {wantedRev[template]}]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, recorded, phase,
attemptTemplate, wantedRev, templateRev, settled,
attemptRev, availableRev, prepared>>
Publish(template) ==
/\ wantedRev[template] /= templateRev[template]
/\ settled[template]
/\ publishRev[template] = {wantedRev[template]}
/\ templateRev' = [templateRev EXCEPT ![template] = wantedRev[template]]
/\ availableRev' =
[availableRev EXCEPT ![template] = @ \cup {wantedRev[template]}]
/\ publishRev' = [publishRev EXCEPT ![template] = {}]
/\ UNCHANGED <<mode, templateRef, eligible, persisted, recorded, phase,
attemptTemplate, wantedRev, settled, attemptRev, prepared>>
Next ==
\/ \E template \in Templates, revision \in Revisions :
RequestTemplate(template, revision)
\/ \E template \in Templates : Settle(template)
\/ \E template \in Templates : PreparePublish(template)
\/ \E template \in Templates : Publish(template)
\/ \E delivery \in Deliveries, sourceMode \in Modes, identity \in Identities :
ChangeEligibility(delivery, sourceMode, identity)
\/ \E delivery \in Deliveries, sourceMode \in Modes :
ChangeMode(delivery, sourceMode)
\/ \E delivery \in Deliveries, template \in Templates :
ChangeTemplateRef(delivery, template)
\/ \E delivery \in Deliveries : Prepare(delivery)
\/ \E pair \in RetryPairs :
PrepareRetry(DeliveryOf(pair[1]), pair[1], pair[2])
\/ \E attempt \in Others, template \in Templates : PrepareOther(attempt, template)
\/ \E attempt \in Slots : Create(attempt)
\/ \E attempt \in Slots : Start(attempt)
\/ \E delivery \in Deliveries, attempt \in Attempts : Persist(delivery, attempt)
\/ \E attempt \in Slots : Complete(attempt)
\/ \E attempt \in Slots : Reject(attempt)
\/ \E delivery \in Deliveries, sourceMode \in Modes : Release(delivery, sourceMode)
\/ \E attempt \in Attempts : Collect(attempt)
Spec == Init /\ [][Next]_vars
\* Argo Secure mode also reads the captured specification during artifact cleanup.
TemplateAvailable ==
\A attempt \in Slots : phase[attempt] \notin {Absent, Collected} =>
\A template \in attemptTemplate[attempt] :
attemptRev[attempt] \subseteq availableRev[template]
CollectedSaved ==
\A attempt \in Attempts : phase[attempt] = Collected => attempt \in persisted
NoConcurrentRuns ==
\A template \in Templates : Cardinality(RunningFor(template)) <= 1
TemplateStable ==
[][\A attempt \in Slots : phase[attempt] \in {Queued, Running} =>
UNCHANGED <<attemptTemplate[attempt], attemptRev[attempt]>>]_vars
PreparedAvailable ==
\A attempt \in Slots : \A binding \in prepared[attempt] :
binding[2] \in availableRev[binding[1]]
AdmissionReady ==
[][\A attempt \in Slots : \A binding \in prepared'[attempt] \ prepared[attempt] :
/\ settled[binding[1]]
/\ binding[2] = wantedRev[binding[1]]
/\ binding[2] = templateRev[binding[1]]]_vars
PublishDrained ==
[][\A template \in Templates : publishRev'[template] \ publishRev[template] /= {} =>
/\ settled[template]
/\ ActiveFor(template) = {}
/\ publishRev'[template] = {wantedRev[template]}]_vars
PublishLatest ==
[][\A template \in Templates : templateRev'[template] /= templateRev[template] =>
/\ settled[template]
/\ publishRev[template] = {wantedRev[template]}
/\ templateRev'[template] = wantedRev[template]]_vars
ProgressFair ==
/\ Spec
/\ \A template \in Templates :
/\ WF_vars(Settle(template))
/\ WF_vars(PreparePublish(template))
/\ WF_vars(Publish(template))
/\ \A attempt \in Slots :
/\ WF_vars(Create(attempt))
/\ WF_vars(Complete(attempt))
FairSpec ==
/\ ProgressFair
/\ \A attempt \in Slots : WF_vars(Start(attempt))
PublishProgress ==
\A template \in Templates, revision \in Revisions :
(<>[](wantedRev[template] = revision)) =>
<>[](templateRev[template] = revision)
\* Strong fairness abstracts Argo's equal-priority mutex queue, ordered by creation time.
LiveSpec ==
/\ ProgressFair
/\ \A attempt \in Slots : SF_vars(Start(attempt))
/\ \A delivery \in Deliveries : WF_vars(Prepare(delivery))
/\ \A delivery \in Deliveries, attempt \in Attempts :
WF_vars(Persist(delivery, attempt))
/\ \A delivery \in Deliveries, sourceMode \in Modes :
WF_vars(Release(delivery, sourceMode))
DebounceProgress ==
\A template \in Templates, revision \in Revisions :
(<>[](wantedRev[template] = revision)) => <>[]settled[template]
DrainProgress ==
\A template \in Templates :
(wantedRev[template] /= templateRev[template]) ~>
(ActiveFor(template) = {} \/ wantedRev[template] = templateRev[template])
AttemptProgress ==
\A attempt \in Slots :
(phase[attempt] \in {Queued, Running}) ~>
(phase[attempt] \in {Terminal, Collected})
ReleaseProgress ==
\A delivery \in Deliveries, sourceMode \in Modes :
(/\ recorded[delivery][sourceMode] /= {}
/\ \A attempt \in recorded[delivery][sourceMode] :
phase[attempt] \in {Terminal, Collected}) ~>
(recorded[delivery][sourceMode] = {})
RecordOnce ==
[][\A delivery \in Deliveries, sourceMode \in Modes :
(recorded'[delivery][sourceMode] \ recorded[delivery][sourceMode])
\subseteq (persisted' \ persisted)]_vars
RecordProgress ==
\A attempt \in Attempts :
(phase[attempt] /= Absent) ~> (attempt \in persisted)
SelectedFor(delivery, attempt, template, revision) ==
/\ SelectedAttempt(delivery) = attempt
/\ templateRef[delivery] = template
/\ wantedRev[template] = revision
DeliveryProgress ==
\A delivery \in Deliveries, attempt \in Attempts,
template \in Templates, revision \in Revisions :
(<>[]SelectedFor(delivery, attempt, template, revision)) =>
<>(attempt \in persisted /\ phase[attempt] \in {Terminal, Collected})
ContinuousOnly == \A delivery \in Deliveries : mode[delivery] = Continuous
LiveDomain ==
/\ ContinuousOnly
/\ \A delivery \in Deliveries :
eligible[delivery][Point] = CHOOSE identity \in Identities : TRUE
SharedTemplateRef ==
[delivery \in Deliveries |-> CHOOSE template \in Templates : TRUE]
ContinuousMode == [delivery \in Deliveries |-> Continuous]
====

14
docs/spec/project-delivery/QualConcurrency.cfg Normal file
View file

@ -0,0 +1,14 @@
SPECIFICATION FairSpec
CONSTANTS
Deliveries = {delivery1, delivery2}
MaxBuilds = 1
MaxRequests = 1
MaxTime = 0
Interval <- TwoTicks
InitialPolicy <- OnBuildPolicy
INVARIANTS TypeOK GlobalCap SuccessfulInputs AutoInterval
PROPERTIES InputsStable AutoSelection NoRestart ManualProgress AutoProgress
CHECK_DEADLOCK FALSE

14
docs/spec/project-delivery/Qualification.cfg Normal file
View file

@ -0,0 +1,14 @@
SPECIFICATION FairSpec
CONSTANTS
Deliveries = {delivery1}
MaxBuilds = 2
MaxRequests = 2
MaxTime = 2
Interval <- TwoTicks
InitialPolicy <- OnBuildPolicy
INVARIANTS TypeOK GlobalCap SuccessfulInputs AutoInterval
PROPERTIES InputsStable AutoSelection NoRestart ManualProgress AutoProgress
CHECK_DEADLOCK FALSE

Some files were not shown because too many files have changed in this diff Show more