remora/docs/spec/testing.md

Implementation Testing

Backend integration and fault tests execute as Kubernetes Job resources inside the kind dependency fixture. Cluster-local execution gives test code the same DNS, service routing, and failure boundaries as the deployed services without exposing fixture dependencies on the host.

Fixture lifecycle

make check-dependencies remains the dependency compatibility gate: it creates a fresh fixture, runs the released-image probes, and tears the fixture down. make check-kind-sample builds and loads a Remora-owned cargo-nextest runner, then runs the same lifecycle around fixture-consumer-sample. That Job is the reference structure for a work-package test Job.

For interactive work, tests/kind/setup.sh creates a ready fixture and leaves it running. tests/kind/destroy.sh removes it. Setup replaces any fixture left by an earlier run, teardown is idempotent, and the harness serializes lifecycle mutations around the single remora-testbed cluster. REMORA_TESTBED_RETAIN=1 retains a failed all-in-one run for inspection.

The fixture owns an isolated kubeconfig at tests/kind/.runtime/kubeconfig and the kind-remora-testbed context. Every Kubernetes operation uses both explicitly. Package runners use the kube helper from tests/kind/lib.sh; they do not read or change the user's kubeconfig or current context.

Package test Jobs

Each work package owns its test code, test-runner image, Job manifest, and acceptance command. Its all-in-one runner follows run-sample-job.sh: acquire the fixture lock, install a cleanup trap, call setup_fixture, load any package-owned image with load_local_image <image> [containerd-namespace], apply the Job with run_job_manifest <namespace> <job-name> <manifest>, and propagate the Job result before teardown. The image namespace defaults to default. A retained failure prints the isolated kubeconfig and context.

Released third-party dependencies are always the digest-pinned images loaded by the fixture. Work packages may build Remora test-runner images locally; they do not rebuild or wrap dependency images. The Job manifest names every loaded image exactly and sets imagePullPolicy: Never.

A test Job:

• runs in remora-testbed with restartPolicy: Never, backoffLimit: 0, and a finite activeDeadlineSeconds;
• disables service-account token mounting and Kubernetes service-link environment variables unless the test explicitly requires them;
• declares CPU and memory requests and limits;
• exits nonzero for every failed assertion and writes diagnostic output to its container logs; and
• uses ClusterIP service names for dependencies and synthetic fixture credentials only.

Package runners may create additional namespaced fixtures and fault injectors needed by their acceptance matrix. They clean those resources or rely on whole-cluster teardown; they never depend on state left by another package.

Network access

Automated backend tests use cluster-local service discovery. Tests of Keycloak or ZITADEL issuer behavior use the dnsPolicy and dnsConfig pattern in oidc-probe so discovery URLs and token issuers have one identity from every participating pod.

Port forwarding is for interactive diagnostics only and is not an acceptance-test transport. Ingress is introduced by deployment and system qualification, where routing is itself under test; substrate work packages do not install an ingress controller merely to reach fixture services.