nuthatch/docs/spec/testing.md

Testing

Host unit tests

The recorded host toolchain is GHC 9.10.3, cabal-install 3.16.1.0, and libpq 18beta1 or a later compatible libpq release. make test runs the nuthatch-unit Tasty suite entirely on the host. It requires no network access and no PostgreSQL, Redis-compatible, OIDC, S3-compatible, container, or Kubernetes service.

The suite uses Tasty as its single explicit test tree, Tasty HUnit for named examples, and Hedgehog through Tasty Hedgehog for generated properties. Hedgehog failures include the seed needed to replay the shrunk counterexample. Generators represent valid and invalid domains with distinct types.

Browser product contract

make check-web-contract regenerates the TypeScript product contract and shared JSON fixtures into a temporary directory and requires byte identity with the checked artifacts. Host tests round-trip every generated wire DTO, validate explicit TypeScript mappings against representative Aeson shapes, and derive complete operation error sets from their Haskell constructors.

The same gate runs every handwritten browser product method against an in-process Servant ProductApi. Browser unit tests separately cover same-origin credentials, encoding, binary bodies, response headers, declared and undeclared failures, network failures, and the no-automatic-retry policy. make check-product runs this gate before its kind product lane.

Browser behavior and accessibility

make check-browser runs the browser unit and component tests, strict TypeScript and Svelte diagnostics, the production Vite build, and one Playwright journey against compiled assets served by the real Haskell application. The reference lane uses Redis, Keycloak, and Garage from the isolated kind fixture.

Vitest 4.1.10 with Testing Library owns component behavior. Playwright 1.61.1 with Chromium and axe-core 4.12.1 owns rendered accessibility and critical product journeys. The browser runner installs the Chromium revision selected by the pinned Playwright package before creating the fixture; the host provides Playwright's supported Linux shared libraries. The browser gate covers authenticated startup, active-root state, keyboard operation, browsing, bounded partial search, upload failure and retry, worker-backed upload, download, single-delete focus recovery, persisted recursive-delete recovery, logout, and production asset loading. Accessibility checks cover programmatic names, roles, states, focus, keyboard operation, and serious or critical axe findings, including rendered color contrast.

Playwright retains traces, screenshots, and video for failures under web/test-results/; its HTML report is written under web/playwright-report/. Both directories are disposable and ignored. The fixture authentication file contains only synthetic credentials, is mode-restricted, and is removed by the runner.

Model correspondence

make check-model-correspondence runs before the kind protocol-conformance gate. It uses the typed Haskell protocol evidence executable, Node 24.18.0, Java 25.0.4, and the TLA+ Tools 2.19 jar whose checksum is pinned in tests/model-correspondence/tools.env. The jar defaults to /usr/local/lib/tla2tools.jar; TLA2TOOLS_JAR may name an equivalent file with the pinned checksum.

The checker discovers executable leaf actions below Next from SANY's semantic graph, executes a minimal TLC configuration, resolves checks named by every behavioral configuration, and compares them with typed implementation inventories. Concrete snapshot fixtures are projected into the model vocabulary without implementing an abstract transition relation. Generated graphs and TLC state are created in a temporary directory and are not retained.

Kind dependency fixture

Backend integration tests execute inside an isolated kind cluster so they use the same DNS and service routing boundaries as deployed Nuthatch components. The fixture provides PostgreSQL, Redis and DragonflyDB, Keycloak and ZITADEL, and Garage and VersityGW. Images and fixture tools are checksum or digest-pinned.

make check-dependencies creates the complete fixture, runs the Haskell dependency probe against each released product interface, and destroys the fixture. make check-kind-matrix runs the Haskell fixture probe sequentially across the Redis/DragonflyDB × Keycloak/ZITADEL × Garage/VersityGW matrix without deploying Nuthatch. make check-app deploys Nuthatch and runs its smoke Job across the Redis/DragonflyDB × Garage/VersityGW matrix. make app-up retains the first app lane for interactive work, and make app-down removes it. tests/kind/setup.sh creates the dependency fixture without Nuthatch for interactive test work; tests/kind/destroy.sh removes either form. NUTHATCH_TESTBED_RETAIN=1 retains a failed all-in-one or matrix run for inspection.

The fixture uses tests/kind/.runtime/kubeconfig and the kind-nuthatch-testbed context. Every Kubernetes operation supplies both explicitly. Fixture lifecycle operations are serialized by a host-runtime lock shared across worktrees, and teardown is idempotent.

All fixture credentials and data are synthetic. Test workloads use ClusterIP service names and do not require host port forwarding. The fixture does not read or change the user's Kubernetes context.

Test runners select the base, durable, coordination, oidc, and target fixture families; all selects every dependency family. The selected families expand into provider lanes, which run sequentially and stop at the first failure. Each lane receives an isolated PostgreSQL database, Redis-compatible logical database, and target bucket as applicable.

Lane configuration is available both as environment variables from a lane-specific ConfigMap and as /run/nuthatch-fixture/profile.env. Successful lanes delete their Job, profile, database, coordination state, and bucket before the next lane. Failed lanes remain intact when the enclosing runner is retained. Locally built application or test images are loaded with load_local_image; released dependencies always use the digest-pinned images from tests/kind/versions.env.

tests/kind/lib.sh is a compatibility facade over the fixture modules. The shared core owns only paths, the explicit Kubernetes wrapper, locking, and Job observation. Runtime-provider setup, dependency deployment, lane selection, PostgreSQL, Redis-compatible coordination, OIDC, target provisioning, profiles, and lane cleanup have separate modules under tests/kind/fixture/. Job manifests configure and launch one component-owned Haskell probe; they do not contain shell test programs.

The host provides rootless nerdctl and containerd at the versions recorded in tests/kind/versions.env, together with ctr, curl, flock, jq, and sha256sum. The fixture downloads its pinned kind and kubectl binaries.