remora/docs/spec/testing.md

Implementation Testing

Backend integration and fault tests execute as Kubernetes Job resources inside the kind dependency fixture. Cluster-local execution gives test code the same DNS, service routing, and failure boundaries as the deployed services without exposing fixture dependencies on the host.

Fixture lifecycle

make check-dependencies remains the dependency compatibility gate: it creates the full fixture, runs the released-image probes, and tears the fixture down. make check-kind-sample builds and loads a Remora-owned cargo-nextest runner, then exercises the full provider matrix around fixture-consumer-sample. Its init containers access the lane database, coordination namespace, OIDC discovery and signing keys, target bucket, and Vault path before cargo-nextest runs. That Job is the reference structure for a work-package test Job.

For interactive work, tests/kind/setup.sh creates a ready fixture and leaves it running. tests/kind/destroy.sh removes it. Setup replaces any fixture left by an earlier run, teardown is idempotent, and the harness serializes lifecycle mutations around the single remora-testbed cluster. REMORA_TESTBED_RETAIN=1 retains a failed all-in-one run for inspection.

The fixture owns an isolated kubeconfig at tests/kind/.runtime/kubeconfig and the kind-remora-testbed context. Every Kubernetes operation uses both explicitly. Package runners use the kube helper from tests/kind/lib.sh; they do not read or change the user's kubeconfig or current context.

Package test Jobs

Each work package owns its test code, test-runner image, Job manifest, acceptance command, and required compatibility families. Its all-in-one runner follows run-sample-job.sh: acquire the fixture lock, install a cleanup trap, call setup_fixture <families...>, load any package-owned image with load_local_image <image> [containerd-namespace], and call run_job_matrix <job-name> <manifest> <image> <families...>. The image namespace defaults to default. A retained failure prints the isolated kubeconfig and context.

The selectable families are durable, coordination, oidc, target, secret, and all. Individual implementations are not selectors. A suite that requests coordination, oidc, and target runs the unchanged cargo-nextest archive against the full Redis/DragonflyDB × Keycloak/ZITADEL × Garage/VersityGW Cartesian product. Lanes run sequentially and stop on the first failure.

Before each lane, the runner recreates a lane database, allocates unique coordination keys, provisions an OIDC tenant/client/user/role grant, creates a target bucket and prefix, and allocates a Vault path as required by the selected families. The resulting environment is exposed both through environment variables and a ConfigMap-mounted /run/remora-fixture/profile.env. Package manifests use @@JOB_NAME@@, @@PROFILE_NAME@@, and @@TEST_IMAGE@@ placeholders so only the profile and lane identity vary between Jobs.

Released third-party dependencies are always the digest-pinned images loaded by the fixture. Work packages may build Remora test-runner images locally; they do not rebuild or wrap dependency images. The Job manifest names every loaded image exactly and sets imagePullPolicy: Never.

A test Job:

• runs in remora-testbed with restartPolicy: Never, backoffLimit: 0, and a finite activeDeadlineSeconds;
• disables service-account token mounting and Kubernetes service-link environment variables unless the test explicitly requires them;
• declares CPU and memory requests and limits;
• exits nonzero for every failed assertion and writes diagnostic output to its container logs; and
• uses ClusterIP service names for dependencies and synthetic fixture credentials only.

Package runners may create additional namespaced fixtures and fault injectors needed by their acceptance matrix. Process-wide coordination fault tests run exclusively. A failed lane is retained when REMORA_TESTBED_RETAIN=1; successful lanes delete their Job and profile before the next lane. Tests never depend on state left by an earlier lane or package.

Network access

Automated backend tests use cluster-local service discovery. Tests of Keycloak or ZITADEL issuer behavior use the dnsPolicy and dnsConfig pattern in fixture-consumer-sample so discovery URLs and token issuers have one identity from every participating pod.

Port forwarding is for interactive diagnostics only and is not an acceptance-test transport. Ingress is introduced by deployment and system qualification, where routing is itself under test; substrate work packages do not install an ingress controller merely to reach fixture services.