remora/docs/spec/testing.md

Implementation Testing

Backend integration and fault tests execute as Kubernetes Job resources inside the kind dependency fixture. Cluster-local execution gives test code the same DNS, service routing, and failure boundaries as the deployed services without exposing fixture dependencies on the host.

Fixture lifecycle

make check-dependencies remains the dependency compatibility gate: it creates the full fixture, runs the released-image probes, and tears the fixture down. make check-kind-sample builds and loads a Remora-owned cargo-nextest runner, then exercises the full provider matrix around fixture-consumer-sample. Its init containers access the lane database, coordination namespace, OIDC discovery and signing keys, target bucket, and Vault path before cargo-nextest runs. That Job is the reference structure for a TestComponent test Job.

For interactive work, tests/kind/fixture/setup.sh creates a ready fixture and leaves it running. tests/kind/fixture/destroy.sh removes it. Setup replaces any fixture left by an earlier run, teardown is idempotent, and a host-runtime lock shared across worktrees serializes lifecycle mutations around the single remora-testbed cluster and remora-dependency-testbed containerd namespace. REMORA_TESTBED_RETAIN=1 retains a failed all-in-one run for inspection.

The fixture owns an isolated kubeconfig at tests/kind/.runtime/kubeconfig and the kind-remora-testbed context. Every Kubernetes operation uses both explicitly. Package runners use the kube helper from tests/kind/fixture/lib.sh; they do not read or change the user's kubeconfig or current context.

Package test Jobs

Each TestComponent owns a directory under tests/kind/ containing its Job manifests and acceptance runners. It also owns its test code, test-runner image, and required compatibility families. Its all-in-one runner follows sample/run.sh: acquire the fixture lock, install a cleanup trap, call setup_fixture <families...>, load any component-owned image with load_local_image <image> [containerd-namespace], and call run_job_matrix <job-name> <manifest> <image> <families...>. The image namespace defaults to default. A retained failure prints the isolated kubeconfig and context.

The selectable families are base, durable, coordination, oidc, target, secret, and all. base provisions only the cluster and common namespace for suites whose controlled fixtures run inside their test pod. Individual implementations are not selectors. A suite that requests coordination, oidc, and target runs the unchanged cargo-nextest archive against the full Redis/DragonflyDB × Keycloak/ZITADEL × Garage/VersityGW Cartesian product. Lanes run sequentially and stop on the first failure.

Before each lane, the runner recreates a lane database, allocates unique coordination keys, provisions an OIDC tenant/client/user/role grant, creates a target bucket and prefix, and allocates a Vault path as required by the selected families. The resulting environment is exposed both through environment variables and a ConfigMap-mounted /run/remora-fixture/profile.env. Package manifests use @@JOB_NAME@@, @@PROFILE_NAME@@, and @@TEST_IMAGE@@ placeholders so only the profile and lane identity vary between Jobs.

Released third-party dependencies are always the digest-pinned images loaded by the fixture. Test components may build Remora test-runner images locally; they do not rebuild or wrap dependency images. The Job manifest names every loaded image exactly and sets imagePullPolicy: Never.

A test Job:

• runs in remora-testbed with restartPolicy: Never, backoffLimit: 0, and a finite activeDeadlineSeconds;
• disables service-account token mounting and Kubernetes service-link environment variables unless the test explicitly requires them;
• declares CPU and memory requests and limits;
• exits nonzero for every failed assertion and writes diagnostic output to its container logs; and
• uses ClusterIP service names for dependencies and synthetic fixture credentials only.

Package runners may create additional namespaced fixtures and fault injectors needed by their acceptance matrix. Process-wide coordination fault tests run exclusively. A failed lane is retained when REMORA_TESTBED_RETAIN=1; successful lanes delete their Job and profile before the next lane. Tests never depend on state left by an earlier lane or package.

Network access

Automated backend tests use cluster-local service discovery. Tests of Keycloak or ZITADEL issuer behavior use the dnsPolicy and dnsConfig pattern in fixture-consumer-sample so discovery URLs and token issuers have one identity from every participating pod.

Port forwarding is for interactive diagnostics only and is not an acceptance-test transport. System qualification owns an external routing fixture around the chart Services; substrate test components do not install a routing controller merely to reach fixture services.