nuthatch/docs/spec/testing.md

# Testing

## Host unit tests

The recorded host toolchain is GHC 9.10.3, cabal-install 3.16.1.0, and libpq
18beta1 or a later compatible libpq release. `make test` runs the
`nuthatch-unit` Tasty suite entirely on the host. It requires no network access
and no PostgreSQL, Redis-compatible, OIDC, S3-compatible, container, or
Kubernetes service.

The suite uses Tasty as its single explicit test tree, Tasty HUnit for named
examples, and Hedgehog through Tasty Hedgehog for generated properties.
Hedgehog failures include the seed needed to replay the shrunk counterexample.
Generators represent valid and invalid domains with distinct types.

## Browser product contract

`make check-web-contract` regenerates the TypeScript product contract and
shared JSON fixtures into a temporary directory and requires byte identity
with the checked artifacts. Host tests round-trip every generated wire DTO,
validate explicit TypeScript mappings against representative Aeson shapes,
and derive complete operation error sets from their Haskell constructors.

The same gate runs every handwritten browser product method against an
in-process Servant `ProductApi`. Browser unit tests separately cover
same-origin credentials, encoding, binary bodies, response headers, declared
and undeclared failures, network failures, and the no-automatic-retry policy.
`make check-product` runs this gate before its kind product lane.

## Browser behavior and accessibility

`make check-browser` runs the browser unit and component tests, strict
TypeScript and Svelte diagnostics, the production Vite build, and one
Playwright journey against compiled assets served by the real Haskell
application. The reference lane uses Redis, Keycloak, and Garage from the
isolated kind fixture.

Vitest 4.1.10 with Testing Library owns component behavior. Playwright 1.61.1
with Chromium and axe-core 4.12.1 owns rendered accessibility and critical
product journeys. The browser runner installs the Chromium revision selected
by the pinned Playwright package before creating the fixture; the host
provides Playwright's supported Linux shared libraries. The browser gate
covers authenticated startup, active-root state, keyboard operation,
browsing, bounded partial search, upload failure and retry, worker-backed
upload, download, single-delete focus recovery, persisted recursive-delete
recovery, logout, and production asset loading. Accessibility checks cover
programmatic names, roles, states, focus, keyboard operation, and serious or
critical axe findings, including rendered color contrast.

Playwright retains traces, screenshots, and video for failures under
`web/test-results/`; its HTML report is written under `web/playwright-report/`.
Both directories are disposable and ignored. The fixture authentication file
contains only synthetic credentials, is mode-restricted, and is removed by
the runner.

## Model correspondence

`make check-model-correspondence` runs before the kind protocol-conformance
gate. It uses the typed Haskell protocol evidence executable, Node 24.18.0,
Java 25.0.3, and the TLA+ Tools 2.19 jar whose checksum is pinned in
`tests/model-correspondence/tools.env`. The jar defaults to
`/usr/local/lib/tla2tools.jar`; `TLA2TOOLS_JAR` may name an equivalent file
with the pinned checksum.

The checker discovers executable leaf actions below `Next` from SANY's
semantic graph, executes a minimal TLC configuration, resolves checks named by
every behavioral configuration, and compares them with typed implementation
inventories. Concrete snapshot fixtures are projected into the model
vocabulary without implementing an abstract transition relation. Generated
graphs and TLC state are created in a temporary directory and are not retained.

## Kind dependency fixture

Backend integration tests execute inside an isolated kind cluster so they use
the same DNS and service routing boundaries as deployed Nuthatch components.
The fixture provides PostgreSQL, Redis and DragonflyDB, Keycloak and ZITADEL,
and Garage and VersityGW. Images and fixture tools are checksum or
digest-pinned.

`make check-dependencies` creates the complete fixture, runs the Haskell
dependency probe against each released product interface, and destroys the
fixture. `make check-kind-matrix` runs the Haskell fixture probe sequentially across the
Redis/DragonflyDB × Keycloak/ZITADEL × Garage/VersityGW matrix without
deploying Nuthatch. `make check-app` deploys Nuthatch and runs its smoke Job
across the Redis/DragonflyDB × Garage/VersityGW matrix. `make app-up` retains
the first app lane for interactive work, and `make app-down` removes it.
`tests/kind/setup.sh` creates the dependency fixture without Nuthatch for
interactive test work; `tests/kind/destroy.sh` removes either form.
`NUTHATCH_TESTBED_RETAIN=1` retains a failed all-in-one or matrix run for
inspection.

The fixture uses `tests/kind/.runtime/kubeconfig` and the
`kind-nuthatch-testbed` context. Every Kubernetes operation supplies both
explicitly. Fixture lifecycle operations are serialized by a host-runtime lock
shared across worktrees, and teardown is idempotent.

All fixture credentials and data are synthetic. Test workloads use ClusterIP
service names and do not require host port forwarding. The fixture does not
read or change the user's Kubernetes context.

Test runners select the `base`, `durable`, `coordination`, `oidc`, and
`target` fixture families; `all` selects every dependency family. The selected
families expand into provider lanes, which run sequentially and stop at the
first failure. Each lane receives an isolated PostgreSQL database,
Redis-compatible logical database, and target bucket as applicable.

Lane configuration is available both as environment variables from a
lane-specific ConfigMap and as `/run/nuthatch-fixture/profile.env`. Successful
lanes delete their Job, profile, database, coordination state, and bucket
before the next lane. Failed lanes remain intact when the enclosing runner is
retained. Locally built application or test images are loaded with
`load_local_image`; released dependencies always use the digest-pinned images
from `tests/kind/versions.env`.

`tests/kind/lib.sh` is a compatibility facade over the fixture modules.
The shared core owns only paths, the explicit Kubernetes wrapper, locking, and
Job observation. Runtime-provider setup, dependency deployment, lane
selection, PostgreSQL, Redis-compatible coordination, OIDC, target
provisioning, profiles, and lane cleanup have separate modules under
`tests/kind/fixture/`. Job manifests configure and launch one component-owned
Haskell probe; they do not contain shell test programs.

The host provides rootless nerdctl and containerd at the versions recorded in
`tests/kind/versions.env`, together with `ctr`, `curl`, `flock`, `jq`, and
`sha256sum`. The fixture downloads its pinned kind and kubectl binaries.