125 lines
6.7 KiB
Markdown
125 lines
6.7 KiB
Markdown
# Testing
|
||
|
||
## Host unit tests
|
||
|
||
The recorded host toolchain is GHC 9.10.3, cabal-install 3.16.1.0, and libpq
|
||
18beta1 or a later compatible libpq release. `make test` runs the
|
||
`nuthatch-unit` Tasty suite entirely on the host. It requires no network access
|
||
and no PostgreSQL, Redis-compatible, OIDC, S3-compatible, container, or
|
||
Kubernetes service.
|
||
|
||
The suite uses Tasty as its single explicit test tree, Tasty HUnit for named
|
||
examples, and Hedgehog through Tasty Hedgehog for generated properties.
|
||
Hedgehog failures include the seed needed to replay the shrunk counterexample.
|
||
Generators represent valid and invalid domains with distinct types.
|
||
|
||
## Browser product contract
|
||
|
||
`make check-web-contract` regenerates the TypeScript product contract and
|
||
shared JSON fixtures into a temporary directory and requires byte identity
|
||
with the checked artifacts. Host tests round-trip every generated wire DTO,
|
||
validate explicit TypeScript mappings against representative Aeson shapes,
|
||
and derive complete operation error sets from their Haskell constructors.
|
||
|
||
The same gate runs every handwritten browser product method against an
|
||
in-process Servant `ProductApi`. Browser unit tests separately cover
|
||
same-origin credentials, encoding, binary bodies, response headers, declared
|
||
and undeclared failures, network failures, and the no-automatic-retry policy.
|
||
`make check-product` runs this gate before its kind product lane.
|
||
|
||
## Browser behavior and accessibility
|
||
|
||
`make check-browser` runs the browser unit and component tests, strict
|
||
TypeScript and Svelte diagnostics, the production Vite build, and one
|
||
Playwright journey against compiled assets served by the real Haskell
|
||
application. The reference lane uses Redis, Keycloak, and Garage from the
|
||
isolated kind fixture.
|
||
|
||
Vitest 4.1.10 with Testing Library owns component behavior. Playwright 1.61.1
|
||
with Chromium and axe-core 4.12.1 owns rendered accessibility and critical
|
||
product journeys. The browser runner installs the Chromium revision selected
|
||
by the pinned Playwright package before creating the fixture; the host
|
||
provides Playwright's supported Linux shared libraries. The browser gate
|
||
covers authenticated startup, active-root state, keyboard operation,
|
||
browsing, bounded partial search, upload failure and retry, worker-backed
|
||
upload, download, single-delete focus recovery, persisted recursive-delete
|
||
recovery, logout, and production asset loading. Accessibility checks cover
|
||
programmatic names, roles, states, focus, keyboard operation, and serious or
|
||
critical axe findings, including rendered color contrast.
|
||
|
||
Playwright retains traces, screenshots, and video for failures under
|
||
`web/test-results/`; its HTML report is written under `web/playwright-report/`.
|
||
Both directories are disposable and ignored. The fixture authentication file
|
||
contains only synthetic credentials, is mode-restricted, and is removed by
|
||
the runner.
|
||
|
||
## Model correspondence
|
||
|
||
`make check-model-correspondence` runs before the kind protocol-conformance
|
||
gate. It uses the typed Haskell protocol evidence executable, Node 24.18.0,
|
||
Java 25.0.4, and the TLA+ Tools 2.19 jar whose checksum is pinned in
|
||
`tests/model-correspondence/tools.env`. The jar defaults to
|
||
`/usr/local/lib/tla2tools.jar`; `TLA2TOOLS_JAR` may name an equivalent file
|
||
with the pinned checksum.
|
||
|
||
The checker discovers executable leaf actions below `Next` from SANY's
|
||
semantic graph, executes a minimal TLC configuration, resolves checks named by
|
||
every behavioral configuration, and compares them with typed implementation
|
||
inventories. Concrete snapshot fixtures are projected into the model
|
||
vocabulary without implementing an abstract transition relation. Generated
|
||
graphs and TLC state are created in a temporary directory and are not retained.
|
||
|
||
## Kind dependency fixture
|
||
|
||
Backend integration tests execute inside an isolated kind cluster so they use
|
||
the same DNS and service routing boundaries as deployed Nuthatch components.
|
||
The fixture provides PostgreSQL, Redis and DragonflyDB, Keycloak and ZITADEL,
|
||
and Garage and VersityGW. Images and fixture tools are checksum or
|
||
digest-pinned.
|
||
|
||
`make check-dependencies` creates the complete fixture, runs the Haskell
|
||
dependency probe against each released product interface, and destroys the
|
||
fixture. `make check-kind-matrix` runs the Haskell fixture probe sequentially across the
|
||
Redis/DragonflyDB × Keycloak/ZITADEL × Garage/VersityGW matrix without
|
||
deploying Nuthatch. `make check-app` deploys Nuthatch and runs its smoke Job
|
||
across the Redis/DragonflyDB × Garage/VersityGW matrix. `make app-up` retains
|
||
the first app lane for interactive work, and `make app-down` removes it.
|
||
`tests/kind/setup.sh` creates the dependency fixture without Nuthatch for
|
||
interactive test work; `tests/kind/destroy.sh` removes either form.
|
||
`NUTHATCH_TESTBED_RETAIN=1` retains a failed all-in-one or matrix run for
|
||
inspection.
|
||
|
||
The fixture uses `tests/kind/.runtime/kubeconfig` and the
|
||
`kind-nuthatch-testbed` context. Every Kubernetes operation supplies both
|
||
explicitly. Fixture lifecycle operations are serialized by a host-runtime lock
|
||
shared across worktrees, and teardown is idempotent.
|
||
|
||
All fixture credentials and data are synthetic. Test workloads use ClusterIP
|
||
service names and do not require host port forwarding. The fixture does not
|
||
read or change the user's Kubernetes context.
|
||
|
||
Test runners select the `base`, `durable`, `coordination`, `oidc`, and
|
||
`target` fixture families; `all` selects every dependency family. The selected
|
||
families expand into provider lanes, which run sequentially and stop at the
|
||
first failure. Each lane receives an isolated PostgreSQL database,
|
||
Redis-compatible logical database, and target bucket as applicable.
|
||
|
||
Lane configuration is available both as environment variables from a
|
||
lane-specific ConfigMap and as `/run/nuthatch-fixture/profile.env`. Successful
|
||
lanes delete their Job, profile, database, coordination state, and bucket
|
||
before the next lane. Failed lanes remain intact when the enclosing runner is
|
||
retained. Locally built application or test images are loaded with
|
||
`load_local_image`; released dependencies always use the digest-pinned images
|
||
from `tests/kind/versions.env`.
|
||
|
||
`tests/kind/lib.sh` is a compatibility facade over the fixture modules.
|
||
The shared core owns only paths, the explicit Kubernetes wrapper, locking, and
|
||
Job observation. Runtime-provider setup, dependency deployment, lane
|
||
selection, PostgreSQL, Redis-compatible coordination, OIDC, target
|
||
provisioning, profiles, and lane cleanup have separate modules under
|
||
`tests/kind/fixture/`. Job manifests configure and launch one component-owned
|
||
Haskell probe; they do not contain shell test programs.
|
||
|
||
The host provides rootless nerdctl and containerd at the versions recorded in
|
||
`tests/kind/versions.env`, together with `ctr`, `curl`, `flock`, `jq`, and
|
||
`sha256sum`. The fixture downloads its pinned kind and kubectl binaries.
|