nuthatch/docs/spec/testing.md

Testing

Host unit tests

The recorded host toolchain is GHC 9.10.3, cabal-install 3.16.1.0, and libpq 18beta1 or a later compatible libpq release. make test runs the nuthatch-unit Tasty suite entirely on the host. It requires no network access and no PostgreSQL, Redis-compatible, OIDC, S3-compatible, container, or Kubernetes service.

The suite uses Tasty as its single explicit test tree, Tasty HUnit for named examples, and Hedgehog through Tasty Hedgehog for generated properties. Hedgehog failures include the seed needed to replay the shrunk counterexample. Generators represent valid and invalid domains with distinct types.

Model correspondence

make check-model-correspondence runs before the kind protocol-conformance gate. It uses the typed Haskell protocol evidence executable, Node 24.18.0, Java 25.0.3, and the TLA+ Tools 2.19 jar whose checksum is pinned in tests/model-correspondence/tools.env. The jar defaults to /usr/local/lib/tla2tools.jar; TLA2TOOLS_JAR may name an equivalent file with the pinned checksum.

The checker discovers executable leaf actions below Next from SANY's semantic graph, executes a minimal TLC configuration, resolves checks named by every behavioral configuration, and compares them with typed implementation inventories. Concrete snapshot fixtures are projected into the model vocabulary without implementing an abstract transition relation. Generated graphs and TLC state are created in a temporary directory and are not retained.

Kind dependency fixture

Backend integration tests execute inside an isolated kind cluster so they use the same DNS and service routing boundaries as deployed Nuthatch components. The fixture provides PostgreSQL, Redis and DragonflyDB, Keycloak and ZITADEL, and Garage and VersityGW. Images and fixture tools are checksum or digest-pinned.

make check-dependencies creates the complete fixture, runs the Haskell dependency probe against each released product interface, and destroys the fixture. make check-kind-matrix runs the Haskell fixture probe sequentially across the Redis/DragonflyDB × Keycloak/ZITADEL × Garage/VersityGW matrix without deploying Nuthatch. make check-app deploys Nuthatch and runs its smoke Job across the Redis/DragonflyDB × Garage/VersityGW matrix. make app-up retains the first app lane for interactive work, and make app-down removes it. tests/kind/setup.sh creates the dependency fixture without Nuthatch for interactive test work; tests/kind/destroy.sh removes either form. NUTHATCH_TESTBED_RETAIN=1 retains a failed all-in-one or matrix run for inspection.

The fixture uses tests/kind/.runtime/kubeconfig and the kind-nuthatch-testbed context. Every Kubernetes operation supplies both explicitly. Fixture lifecycle operations are serialized by a host-runtime lock shared across worktrees, and teardown is idempotent.

All fixture credentials and data are synthetic. Test workloads use ClusterIP service names and do not require host port forwarding. The fixture does not read or change the user's Kubernetes context.

Test runners select the base, durable, coordination, oidc, and target fixture families; all selects every dependency family. The selected families expand into provider lanes, which run sequentially and stop at the first failure. Each lane receives an isolated PostgreSQL database, Redis-compatible logical database, and target bucket as applicable.

Lane configuration is available both as environment variables from a lane-specific ConfigMap and as /run/nuthatch-fixture/profile.env. Successful lanes delete their Job, profile, database, coordination state, and bucket before the next lane. Failed lanes remain intact when the enclosing runner is retained. Locally built application or test images are loaded with load_local_image; released dependencies always use the digest-pinned images from tests/kind/versions.env.

tests/kind/lib.sh is a compatibility facade over the fixture modules. The shared core owns only paths, the explicit Kubernetes wrapper, locking, and Job observation. Runtime-provider setup, dependency deployment, lane selection, PostgreSQL, Redis-compatible coordination, OIDC, target provisioning, profiles, and lane cleanup have separate modules under tests/kind/fixture/. Job manifests configure and launch one component-owned Haskell probe; they do not contain shell test programs.

The host provides rootless nerdctl and containerd at the versions recorded in tests/kind/versions.env, together with ctr, curl, flock, jq, and sha256sum. The fixture downloads its pinned kind and kubectl binaries.