501 lines
30 KiB
Markdown
501 lines
30 KiB
Markdown
|
|
# Nuthatch — user-facing S3 BFF
|
|||
|
|
|
|||
|
|
## Status and terminology
|
|||
|
|
|
|||
|
|
This document defines the version-one product behavior, architecture, trust
|
|||
|
|
boundaries, dependency roles, and externally observable requirements for
|
|||
|
|
Nuthatch. It does not define mutable HTTP schemas, concrete database schemas,
|
|||
|
|
implementation libraries, or deployment-specific numeric settings.
|
|||
|
|
|
|||
|
|
The key words **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, and **MAY** are
|
|||
|
|
to be interpreted as described by RFC 2119 and RFC 8174.
|
|||
|
|
|
|||
|
|
Requirements tables and explicitly labeled rules are normative even when they
|
|||
|
|
do not use one of these key words.
|
|||
|
|
|
|||
|
|
## Specification ownership
|
|||
|
|
|
|||
|
|
This document is authoritative for:
|
|||
|
|
|
|||
|
|
- user-visible storage behavior and access boundaries;
|
|||
|
|
- the identity and object-root model;
|
|||
|
|
- externally visible quota and mutation guarantees;
|
|||
|
|
- allocation of authority among Nuthatch, PostgreSQL, Redis, and the object
|
|||
|
|
store; and
|
|||
|
|
- failure behavior at dependency boundaries.
|
|||
|
|
|
|||
|
|
[`Nuthatch.tla`](../protocol/Nuthatch.tla) is the design of record for the
|
|||
|
|
abstract mutation protocol. It owns mutation states, transition ordering,
|
|||
|
|
reservations, exclusive same-key admission, duplicate requests, upload-body
|
|||
|
|
loss, crash recovery, object-store settlement assumptions, and the safety and
|
|||
|
|
liveness properties that establish the guarantees in this document. Concrete
|
|||
|
|
mutation code MUST refine that model.
|
|||
|
|
|
|||
|
|
The TLA+ configurations are checked finite instances. Their finite users, keys,
|
|||
|
|
sizes, contents, operation ids, quotas, and fault counts are not production
|
|||
|
|
limits.
|
|||
|
|
|
|||
|
|
HTTP routes and schemas, SQL layouts, libraries, object-store vendors, identity
|
|||
|
|
providers, cryptographic algorithms, and deployment topology are separate
|
|||
|
|
contracts or deployment choices. They may narrow behavior but MUST preserve
|
|||
|
|
this specification.
|
|||
|
|
|
|||
|
|
### Core terms
|
|||
|
|
|
|||
|
|
| Term | Definition |
|
|||
|
|
| ---- | ---------- |
|
|||
|
|
| Pool | One quota-accounting domain: a personal root or the shared root |
|
|||
|
|
| Mutation | One put, overwrite, or delete against one canonical object key |
|
|||
|
|
| Operation id | A caller-generated opaque identifier for one logical mutation |
|
|||
|
|
| Request fingerprint | A canonical identity of the mutation kind, target, and content where applicable |
|
|||
|
|
| Content identity | A collision-resistant digest binding the exact uploaded bytes; its algorithm and encoding belong to the API contract |
|
|||
|
|
| Successful mutation | A mutation whose object state and PostgreSQL accounting are durably finalized |
|
|||
|
|
| Unfinished mutation | A durably recorded mutation that has not yet reached its successful or unsuccessful final result |
|
|||
|
|
|
|||
|
|
## 1. Problem statement
|
|||
|
|
|
|||
|
|
| Item | Detail |
|
|||
|
|
| ---- | ------ |
|
|||
|
|
| Need | Authenticated end users must upload, download, browse, search, and delete files without operator privileges on the object store |
|
|||
|
|
| Problem with admin UIs | Typical S3 admin consoles are operator-privileged and must not be the user plane |
|
|||
|
|
| Scale | Fewer than ten users |
|
|||
|
|
|
|||
|
|
| Must not | Reason |
|
|||
|
|
| -------- | ------ |
|
|||
|
|
| Anonymous / public share links | Out of trust model |
|
|||
|
|
| Desktop sync or desktop app | Not required |
|
|||
|
|
| Office / collaboration suite | Not required |
|
|||
|
|
| Browser-held object-store credentials | Key leak and over-privilege |
|
|||
|
|
| User access to the object-store admin console | Wrong control plane |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 2. Proposed solution
|
|||
|
|
|
|||
|
|
**Nuthatch** is a **backend-for-frontend (BFF)**: a web UI and API that call an S3-compatible API with a single service principal and enforce identity-derived key prefixes. End users never receive object-store credentials.
|
|||
|
|
|
|||
|
|
Nuthatch is an **OIDC relying party** and performs its own authorization-code flow. An external auth proxy is not required on this path.
|
|||
|
|
|
|||
|
|
| Decision | Choice |
|
|||
|
|
| -------- | ------ |
|
|||
|
|
| Product name | **Nuthatch** |
|
|||
|
|
| Service / executable name | `nuthatch` |
|
|||
|
|
| Auth | OIDC. Login UI is the identity provider’s hosted login |
|
|||
|
|
| Session | Signed JWT cookie (HttpOnly, Secure, host-only). No server session store |
|
|||
|
|
| CSRF | Required for cookie-authenticated state-changing requests |
|
|||
|
|
| Durable state | **PostgreSQL**: quota limits, usage, and protocol-required mutation metadata |
|
|||
|
|
| Shared ephemeral state | **Redis-compatible** store: list/metadata cache and bandwidth throttle |
|
|||
|
|
| Object store | **S3-compatible** API, one bucket, one service principal |
|
|||
|
|
| Key layout | Personal `users/{user_id}/` derived from verified (`iss`, `sub`). Shared `shared/` |
|
|||
|
|
| Audit | Structured JSON on stdout |
|
|||
|
|
| Admin product surface | None. Object-store admin tooling remains operators-only |
|
|||
|
|
| Implementation language | Not fixed by this design |
|
|||
|
|
|
|||
|
|
| Principle | Meaning |
|
|||
|
|
| --------- | ------- |
|
|||
|
|
| Stateless controller | No in-process authoritative or cross-request product state. Any replica may serve any request |
|
|||
|
|
| Shared ephemeral coordination | Cache and throttle live in Redis so multi-replica works without sticky sessions |
|
|||
|
|
| Service object principal | One IAM-style user. The app enforces per-user and shared prefixes |
|
|||
|
|
| No browser to object store | All product-path object bytes stream through Nuthatch |
|
|||
|
|
| Objects for end users only via BFF | Operators may inspect objects separately but MUST NOT mutate Nuthatch-owned prefixes while product service is enabled |
|
|||
|
|
| Mutation integrity | Configured quota limits are never exceeded. A successful mutation includes durable quota accounting, and retrying one logical mutation within the advertised idempotency window is idempotent |
|
|||
|
|
|
|||
|
|
### 2.1 Runtime dependencies
|
|||
|
|
|
|||
|
|
| Dependency | Role | Interface |
|
|||
|
|
| ---------- | ---- | --------- |
|
|||
|
|
| OIDC identity provider | Authenticate users | Standard OIDC (authorization code + PKCE) |
|
|||
|
|
| PostgreSQL | Quota limits and usage | SQL |
|
|||
|
|
| Redis-compatible store | List/metadata cache, bandwidth throttle | Redis protocol |
|
|||
|
|
| S3-compatible object store | Object bytes | Ordinary object put, get, head, delete, and list operations |
|
|||
|
|
|
|||
|
|
Nuthatch MUST NOT require object versioning, conditional mutation, event
|
|||
|
|
delivery, server-side copy, or transactional object-store extensions for
|
|||
|
|
correctness.
|
|||
|
|
|
|||
|
|
No other runtime dependencies are required for the product (no message bus, no search engine, no durable server-side session store, no auth proxy). Short-lived OIDC transaction state may be held in an integrity-protected, HttpOnly browser cookie.
|
|||
|
|
|
|||
|
|
Secrets (OIDC client secret, object-store keys, JWT signing material, Redis password if any) are supplied by the deployment environment.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 3. Architecture
|
|||
|
|
|
|||
|
|
### 3.1 Context
|
|||
|
|
|
|||
|
|
```mermaid
|
|||
|
|
flowchart TB
|
|||
|
|
User[End user browser] -->|HTTPS| App[Nuthatch]
|
|||
|
|
App -->|OIDC| IdP[OIDC identity provider]
|
|||
|
|
App -->|quota| PG[(PostgreSQL)]
|
|||
|
|
App -->|cache and throttle| Redis[(Redis compatible)]
|
|||
|
|
App -->|S3 API| S3[S3 compatible store]
|
|||
|
|
App -->|stdout| Logs[Log sink]
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 3.2 Request path
|
|||
|
|
|
|||
|
|
Every authenticated request validates the Nuthatch session JWT and binds the verified OIDC identity. The server derives the active storage root; the browser never supplies a complete S3 key. Bandwidth accounting applies to **upload and download** body bytes (see §3.8). Further steps depend on the operation.
|
|||
|
|
|
|||
|
|
```mermaid
|
|||
|
|
flowchart TD
|
|||
|
|
start[Request] --> jwt{Valid JWT cookie}
|
|||
|
|
jwt -->|no| oidc[OIDC code flow]
|
|||
|
|
oidc --> mint[Mint JWT cookie]
|
|||
|
|
mint --> ready
|
|||
|
|
jwt -->|yes| ready{Operation}
|
|||
|
|
ready -->|browse| listPath[List under active root]
|
|||
|
|
ready -->|search| searchPath[Search under active root]
|
|||
|
|
ready -->|download| dlPath[Download under allowed key]
|
|||
|
|
ready -->|upload| upPath[Upload under allowed key]
|
|||
|
|
ready -->|delete| delPath[Delete key or prefix page]
|
|||
|
|
listPath --> listCache[List cache then S3 ListObjects]
|
|||
|
|
listCache --> page[HTML or JSON]
|
|||
|
|
searchPath --> searchS3[S3 list under root then filter keys]
|
|||
|
|
searchS3 --> page
|
|||
|
|
dlPath --> dlThrottle[Count download bytes toward throttle]
|
|||
|
|
dlThrottle --> streamOut[Byte stream response]
|
|||
|
|
upPath --> csrfUp[CSRF check]
|
|||
|
|
csrfUp --> quota[Admit mutation under quota invariant]
|
|||
|
|
quota --> upThrottle[Count upload bytes toward throttle]
|
|||
|
|
upThrottle --> put[S3 put]
|
|||
|
|
put --> usageUp[Finalize object and durable accounting]
|
|||
|
|
usageUp --> inv[Invalidate list cache for parent prefixes]
|
|||
|
|
inv --> page
|
|||
|
|
delPath --> csrfDel[CSRF check]
|
|||
|
|
csrfDel --> delKey[S3 delete one key]
|
|||
|
|
delKey --> usageDn[Finalize per-key mutation outcome]
|
|||
|
|
usageDn --> inv
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 3.3 Trust and data planes
|
|||
|
|
|
|||
|
|
| Plane | Store | Contents |
|
|||
|
|
| ----- | ----- | -------- |
|
|||
|
|
| Identity | OIDC IdP | Users and OIDC client registration for Nuthatch |
|
|||
|
|
| Objects | S3-compatible store | Keys under `users/*` and `shared/*` |
|
|||
|
|
| Durable app | PostgreSQL | Quota limits, usage, and any durable mutation metadata required by the integrity protocol |
|
|||
|
|
| Shared ephemeral | Redis-compatible | List/metadata cache. Per-`user_id` bandwidth throttle |
|
|||
|
|
| Secrets | Deployment secret store | OIDC client secret, S3 access keys, JWT signing key, Redis auth if required |
|
|||
|
|
| Audit | stdout | Authn, authz, and mutations |
|
|||
|
|
|
|||
|
|
| Store | Holds | Does not hold |
|
|||
|
|
| ----- | ----- | ------------- |
|
|||
|
|
| PostgreSQL | Quota limits, usage, protocol-required mutation metadata | Browser sessions, throttle, object bytes |
|
|||
|
|
| Redis-compatible | Cache, bandwidth throttle | Sessions, durable quota, object bytes |
|
|||
|
|
| JWT cookie | Session | — |
|
|||
|
|
| S3-compatible | Object bytes | Authn, quota policy |
|
|||
|
|
|
|||
|
|
### 3.4 Object key layout and roots
|
|||
|
|
|
|||
|
|
| Scope | Prefix | Access |
|
|||
|
|
| ----- | ------ | ------ |
|
|||
|
|
| Personal | `users/{user_id}/` | Owning verified OIDC identity only |
|
|||
|
|
| Shared | `shared/` | Any authenticated user may list, read, write, and delete any object |
|
|||
|
|
|
|||
|
|
| Rule | Detail |
|
|||
|
|
| ---- | ------ |
|
|||
|
|
| Identity tuple | Ownership is based on the verified pair (`iss`, `sub`), not email |
|
|||
|
|
| User id | `user_id` is a deterministic, collision-resistant, URL-safe, opaque encoding of (`iss`, `sub`), defined by a versioned implementation contract and stable across deployments using the same identity configuration |
|
|||
|
|
| Active root | UI and API operate in one root at a time: personal or shared. The client selects only the root enum |
|
|||
|
|
| Shared trust | Shared is a **single trust domain** with no per-object ACLs. Any authenticated user can wipe shared content. Accepted at this scale |
|
|||
|
|
| Shared UX | The UI must clearly distinguish the shared root and identify destructive actions as affecting all users |
|
|||
|
|
| Folders | Virtual via S3 delimiter `/` |
|
|||
|
|
| Client path | The client supplies a relative logical path, never a complete S3 key or ownership prefix |
|
|||
|
|
| Path representation | A logical object path is an NFC-normalized sequence of non-empty Unicode segments separated by `/` |
|
|||
|
|
| Invalid paths | Reject invalid UTF-8, leading or trailing separators, empty segments, `.` or `..` segments, backslashes, Unicode control characters, and encoded forms that introduce any rejected value |
|
|||
|
|
| Decoding | Percent decoding occurs exactly once at the HTTP boundary. Decoded input is validated rather than rewritten into an alias |
|
|||
|
|
| Authorization check | After construction, the canonical key must remain beneath the server-derived active root |
|
|||
|
|
| Bucket | Single configured bucket (name is deployment configuration) |
|
|||
|
|
| Principal | Single configured access key id for the BFF (deployment configuration) |
|
|||
|
|
|
|||
|
|
The exact `user_id` encoding is an implementation contract and must avoid collisions. Raw `sub` and email values are not inserted directly into object keys.
|
|||
|
|
|
|||
|
|
### 3.5 S3 access model
|
|||
|
|
|
|||
|
|
```mermaid
|
|||
|
|
flowchart LR
|
|||
|
|
App[Nuthatch] -->|service credentials| Principal[S3 principal]
|
|||
|
|
Principal --> Bucket[configured bucket]
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
| Layer | Control |
|
|||
|
|
| ----- | ------- |
|
|||
|
|
| Network | Users never call the object store directly for product traffic |
|
|||
|
|
| IAM / policy | Principal limited to the configured bucket (and optional prefix constraints) |
|
|||
|
|
| App | Verified (`iss`, `sub`) maps to an opaque personal root. Shared root is open to any authenticated user |
|
|||
|
|
| Transport | Upload and download are proxied through Nuthatch |
|
|||
|
|
|
|||
|
|
### 3.6 OIDC, session (JWT), and CSRF
|
|||
|
|
|
|||
|
|
| Item | Value |
|
|||
|
|
| ---- | ----- |
|
|||
|
|
| OIDC flow | Authorization code with PKCE using the identity provider’s hosted login |
|
|||
|
|
| Client type | Confidential OIDC client |
|
|||
|
|
| Transaction state | Short-lived integrity-protected, HttpOnly browser state for `state`, `nonce`, PKCE verifier, and a validated local return target. Not a durable server session |
|
|||
|
|
| Callback validation | Validate `state`, issuer, signature, audience/client id, nonce, code exchange result, token lifetime, and the exact configured redirect URI |
|
|||
|
|
| Token validation policy | Trusted issuer metadata and deployment policy determine accepted keys and algorithms; an untrusted token header alone does not |
|
|||
|
|
| Mechanism | App-signed Nuthatch session JWT after successful OIDC validation |
|
|||
|
|
| Cookie name | Deployment configuration |
|
|||
|
|
| Cookie flags | `HttpOnly`, `Secure`, host-only (no `Domain`), `Path=/` |
|
|||
|
|
| SameSite | `Lax` (compatible with OIDC top-level return) |
|
|||
|
|
| Lifetime | Bound to JWT `exp`. Duration is deployment configuration |
|
|||
|
|
| Claims | Identify Nuthatch as session issuer and audience and carry the verified upstream `oidc_issuer` and `oidc_subject`; `iat` and `exp` are required. Email or display name may be copied for UI display only |
|
|||
|
|
| Signing | Nuthatch signs and validates the session under an explicitly configured policy using app key material from the deployment secret store |
|
|||
|
|
| Authorization model | Authenticated user only. No role matrix beyond personal vs shared roots |
|
|||
|
|
| Logout | Clear the Nuthatch session cookie and transient login cookies. An authenticated logout is state-changing and follows the CSRF rule |
|
|||
|
|
| Revocation | JWT expiry only. No server-side session list |
|
|||
|
|
| Not used | Redis or PostgreSQL browser-session rows |
|
|||
|
|
|
|||
|
|
| CSRF rule | Detail |
|
|||
|
|
| --------- | ------ |
|
|||
|
|
| Applies to | State-changing requests that use the session cookie: upload, delete, and any other mutating API |
|
|||
|
|
| Does not replace | OIDC transaction validation or normal authorization checks |
|
|||
|
|
| Mechanism | Synchronizer token or equivalent double-submit mechanism issued by Nuthatch and required on mutating requests |
|
|||
|
|
| Safe methods | `GET` and other safe methods for browse, search, and download do not require a CSRF token |
|
|||
|
|
| Failure | A mutating request without a valid CSRF token is rejected before object or quota mutation |
|
|||
|
|
|
|||
|
|
### 3.7 Quota and mutation integrity
|
|||
|
|
|
|||
|
|
| Pool | Scope | Accounting |
|
|||
|
|
| ---- | ----- | ---------- |
|
|||
|
|
| Personal | `users/{user_id}/` | One deployment-configured personal limit applied independently to each user, with per-user `used_bytes` in PostgreSQL |
|
|||
|
|
| Shared | `shared/` | One global limit and `used_bytes` in PostgreSQL |
|
|||
|
|
|
|||
|
|
| Rule | Detail |
|
|||
|
|
| ---- | ------ |
|
|||
|
|
| Limit values | Deployment configuration. Not fixed by this design |
|
|||
|
|
| No overshoot | No object state produced by a Nuthatch mutation may place its pool above the configured limit |
|
|||
|
|
| Successful mutation | Put, overwrite, or delete succeeds only after the intended object state and its PostgreSQL accounting are durably finalized |
|
|||
|
|
| Unfinished mutation | A mutation that cannot be finalized MUST NOT report success. Conflicting mutations against its key fail closed until recovery resolves the unfinished operation |
|
|||
|
|
| Idempotency identity | Every mutation carries a client-generated operation id scoped to the verified user identity and operation kind |
|
|||
|
|
| Replay | Within the advertised idempotency window, exact replay of an unfinished mutation reports it as unfinished without starting a second effect. Once terminal, exact replay returns the recorded terminal result. Reuse with a different request fingerprint is rejected |
|
|||
|
|
| Request fingerprint | The fingerprint binds the operation kind, active root, canonical relative path, declared size, and content identity for puts. Its wire encoding and the idempotency-window duration belong to the API contract |
|
|||
|
|
| Same-key contention | At most one mutation may be admitted for a canonical object key. A new mutation for a locked key is rejected rather than queued; retry after resolution uses a new operation id |
|
|||
|
|
| Accounting | When no mutation in a pool is unfinished, `used_bytes` equals the sum of object sizes in that Nuthatch-owned pool |
|
|||
|
|
| Shared writes | Charge only the shared pool |
|
|||
|
|
| Personal writes | Charge only that user’s personal pool |
|
|||
|
|
| Downloads | Do not change usage |
|
|||
|
|
| Size known | Uploads must declare total size through the API contract before body acceptance |
|
|||
|
|
| Request settlement | An object-store request is settled only when it can no longer mutate its key. The reservation and key lock remain held until settlement |
|
|||
|
|
| Object observation | After settlement, Nuthatch identifies the stored bytes through their size and content identity using ordinary head or streaming get behavior |
|
|||
|
|
| Required object semantics | Put and delete settle within a configured maximum interval, replace one key atomically, and cannot take effect after settlement. After settlement, reads observe either the complete prior object or the complete requested object |
|
|||
|
|
| Unsupported store | A deployment that cannot establish the required object semantics is unsupported |
|
|||
|
|
| Upload interruption | Nuthatch does not retain or resume a lost request body. After request settlement, recovery finalizes success if the intended content is present; otherwise it records failure and releases the reservation and key lock |
|
|||
|
|
| Upload retry | The UI identifies the failed upload and allows the user to start a complete replacement upload with a new operation id |
|
|||
|
|
| Overwrite | Admission and final accounting use the size delta between the prior committed object and the replacement, not the replacement size alone |
|
|||
|
|
| Authority | Usage is app-maintained for mutations through Nuthatch |
|
|||
|
|
| Dependency loss | PostgreSQL or object-store loss fails new mutations closed. Unfinished mutations remain durably resolvable after transient dependency loss and process restart |
|
|||
|
|
| Object-store assumption | The object store supplies ordinary blob operations but no cross-system transaction, compare-and-swap, versioning, or event guarantee |
|
|||
|
|
| Deployment writer contract | Nuthatch is the exclusive writer to `users/` and `shared/` while product service is enabled. Out-of-band mutation violates the deployment contract and may invalidate accounting |
|
|||
|
|
|
|||
|
|
Standing “list the bucket and rewrite counters” is not the primary correctness mechanism. A repair or verification job may exist as supplementary defense, but normal mutation success and crash recovery must not depend on a full-bucket reconciliation.
|
|||
|
|
|
|||
|
|
### 3.8 Throttle (Redis-compatible)
|
|||
|
|
|
|||
|
|
| Item | Value |
|
|||
|
|
| ---- | ----- |
|
|||
|
|
| Dimension | Bytes transferred per time unit per verified user identity |
|
|||
|
|
| Directions | **Upload and download** body bytes both count against one user bucket |
|
|||
|
|
| Algorithm | Shared token bucket with configurable sustained rate and burst allowance |
|
|||
|
|
| Store | Redis-compatible, shared across replicas |
|
|||
|
|
| When checked | Before accepting or sending further body bytes |
|
|||
|
|
| Mid-stream exhaustion | Stop transferring and fail the request. Bytes already moved still count |
|
|||
|
|
| Over limit before start | HTTP 429 |
|
|||
|
|
| Parallel streams | All replicas and simultaneous streams consume from the same logical user bucket |
|
|||
|
|
| Aborted transfers | Count bytes actually proxied before termination |
|
|||
|
|
| Range requests | Not supported in version one |
|
|||
|
|
| Redis unavailable | Throttle enforcement fails closed for new uploads and downloads. Browse and search may continue without cache |
|
|||
|
|
| Not used | Request-count limits, concurrent-upload caps, in-process counters |
|
|||
|
|
|
|||
|
|
### 3.9 Browse, search, upload, download, and delete
|
|||
|
|
|
|||
|
|
| Feature | Behavior |
|
|||
|
|
| ------- | -------- |
|
|||
|
|
| Browse | `ListObjectsV2` with delimiter `/`, configured page size, and continuation token under the active root |
|
|||
|
|
| Search | Case-insensitive substring on object keys under the active root only. List and filter in process; no content index |
|
|||
|
|
| Search bounds | Configurable maximum keys/pages scanned and request timeout. A bounded partial result must be identified as partial |
|
|||
|
|
| Upload | Stream request bodies to the object store. Require declared total size and enforce maximum object size before accepting the body. Interrupted bodies are not resumed |
|
|||
|
|
| Overwrite | Replacing an existing key is allowed and follows the mutation-integrity protocol and delta accounting in §3.7 |
|
|||
|
|
| Download | Stream from the object store through Nuthatch. Default to `Content-Disposition: attachment` with a safely encoded filename |
|
|||
|
|
| Download safety | Set `X-Content-Type-Options: nosniff`; do not render arbitrary stored HTML or SVG inline; apply a restrictive application Content Security Policy |
|
|||
|
|
| Delete object | Delete one canonical key through the mutation-integrity protocol, then invalidate affected list-cache entries |
|
|||
|
|
| Delete folder | After UI confirmation, enumerate keys under the prefix in configured-size pages and execute one idempotent mutation per key |
|
|||
|
|
| Folder delete consistency | Recursive deletion is not atomic or snapshot-isolated. Each observed key has an independent result, and keys created concurrently may remain |
|
|||
|
|
| Partial folder delete | Return per-key progress sufficient for a client to retry unfinished pages; do not report a key as deleted until its mutation succeeds |
|
|||
|
|
| Missing key | Deleting an already-absent key succeeds without changing usage |
|
|||
|
|
| Max object size | Configurable deployment limit |
|
|||
|
|
| Previews | None |
|
|||
|
|
| Versioning UI | None |
|
|||
|
|
|
|||
|
|
Per-key recursive deletion is deliberate: it bounds each mutation and avoids one
|
|||
|
|
large deferred accounting adjustment. The version-one mutation protocol model
|
|||
|
|
covers individual keys. A durable server-side batch deletion protocol is
|
|||
|
|
outside version one.
|
|||
|
|
|
|||
|
|
### 3.10 Caching (Redis-compatible)
|
|||
|
|
|
|||
|
|
| Item | Value |
|
|||
|
|
| ---- | ----- |
|
|||
|
|
| What | Browse list and light metadata for a cache key of active root plus list parameters (prefix, delimiter, continuation, page size) |
|
|||
|
|
| Search | May reuse browse list entries for the same root listing parameters, then filter in process. Do not cache search-filter results as a separate long-lived view unless keyed by the query string as well |
|
|||
|
|
| Where | Redis-compatible store (same instance as throttle is allowed, distinct key namespace from throttle keys) |
|
|||
|
|
| TTL | Configurable |
|
|||
|
|
| Invalidate | After a put or delete mutation succeeds, invalidate list-cache keys for the active root whose listed prefix is the object’s parent prefix or an ancestor prefix under that root (including the root listing). TTL remains the backstop |
|
|||
|
|
| Redis unavailable | Cache fails open: bypass cache and read from S3. Cache failure alone does not make browse/search unavailable |
|
|||
|
|
| Required | Yes |
|
|||
|
|
|
|||
|
|
### 3.11 Observability
|
|||
|
|
|
|||
|
|
| Signal | Path |
|
|||
|
|
| ------ | ---- |
|
|||
|
|
| Audit | Structured JSON on stdout for authentication events, authorization failures, and object mutations |
|
|||
|
|
| Common fields | `ts`, `action`, `result`, and request or mutation correlation id |
|
|||
|
|
| Failure fields | Stable `reason` value when the result is not successful |
|
|||
|
|
| Identity fields | Stable opaque user id when a verified identity is available |
|
|||
|
|
| Object fields | `root` and `bytes` when applicable to the recorded action |
|
|||
|
|
| Object key policy | Full key, relative path, or keyed hash is configurable. Deployments must treat filenames as potentially sensitive |
|
|||
|
|
| Prohibited log data | Session JWTs, OIDC tokens/codes, CSRF tokens, object bodies, secrets, and raw authorization headers |
|
|||
|
|
| Read logging | Download and browse/search key detail may be reduced or hashed to limit privacy exposure |
|
|||
|
|
| Metrics | Not required by this design |
|
|||
|
|
|
|||
|
|
### 3.12 Process and dependency failure model
|
|||
|
|
|
|||
|
|
| Item | Value |
|
|||
|
|
| ---- | ----- |
|
|||
|
|
| App processes | Stateless with respect to browser sessions and in-process product authority. Multi-replica safe when PostgreSQL and Redis-compatible backends are shared |
|
|||
|
|
| Liveness | Reports whether the process can serve requests; does not fail solely because a dependency is unavailable |
|
|||
|
|
| Readiness | Reports whether the replica can safely serve its configured traffic class |
|
|||
|
|
| PostgreSQL unavailable | Reject mutations that require quota or protocol state. Read-only object operations may continue only if their authorization and audit requirements can still be met |
|
|||
|
|
| Redis unavailable | Bypass cache; fail closed for new uploads/downloads because bandwidth enforcement is unavailable |
|
|||
|
|
| S3 unavailable | Reject object operations with a dependency error; authentication and non-object endpoints may remain available |
|
|||
|
|
| Recovery | After transient dependency loss or process restart, unfinished mutations are recoverable without sticky sessions or leader election |
|
|||
|
|
| Leader election | None required by this design |
|
|||
|
|
|
|||
|
|
|
|||
|
|
### 3.13 Mutation protocol boundary
|
|||
|
|
|
|||
|
|
This document defines the externally visible mutation guarantees and the
|
|||
|
|
authority assigned to PostgreSQL and the object store. `Nuthatch.tla` defines
|
|||
|
|
the abstract state machine that implements those guarantees.
|
|||
|
|
|
|||
|
|
The protocol model defines:
|
|||
|
|
|
|||
|
|
- admissible states and transitions for put, overwrite, delete, retry, and recovery;
|
|||
|
|
- quota reservation and release semantics;
|
|||
|
|
- exclusive same-key admission and contention rejection;
|
|||
|
|
- duplicate operation-id and request-fingerprint behavior;
|
|||
|
|
- crash and dependency-failure recovery;
|
|||
|
|
- the relationship between object state and PostgreSQL accounting; and
|
|||
|
|
- safety invariants sufficient to establish the no-overshoot, idempotency, and
|
|||
|
|
accounting requirements in §3.7.
|
|||
|
|
|
|||
|
|
The model may assume that transient dependency failures eventually recover and
|
|||
|
|
that recovery processing continues. Permanent dependency loss is outside its
|
|||
|
|
liveness boundary and causes Nuthatch to remain failed closed.
|
|||
|
|
|
|||
|
|
The version-one model covers individual object keys. Recursive prefix deletion
|
|||
|
|
is a sequence of per-key operations and is not part of that model. A concrete
|
|||
|
|
implementation is conformant only when its mutation behavior refines the model.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 4. Requirements
|
|||
|
|
|
|||
|
|
### 4.1 Functional
|
|||
|
|
|
|||
|
|
| ID | Requirement |
|
|||
|
|
| -- | ----------- |
|
|||
|
|
| F1 | Serve a browser UI and API over HTTPS |
|
|||
|
|
| F2 | Authenticate with OIDC using the IdP-hosted login as Nuthatch’s own OIDC client |
|
|||
|
|
| F3 | Issue and validate a JWT session cookie after OIDC |
|
|||
|
|
| F4 | Protect state-changing cookie-authenticated requests with CSRF tokens |
|
|||
|
|
| F5 | Personal root `users/{user_id}/` derived from verified (`iss`, `sub`) |
|
|||
|
|
| F6 | Shared root `shared/` for any authenticated user (single trust domain) |
|
|||
|
|
| F7 | Client-selectable active root (personal or shared) |
|
|||
|
|
| F8 | Paginated browse under the active root |
|
|||
|
|
| F9 | Drag-and-drop and form upload under the active root; identify failed uploads and allow complete user-initiated retry without server-side body resumption |
|
|||
|
|
| F10 | Download under the active root |
|
|||
|
|
| F11 | Delete an object or recursively enumerate a prefix under the active root using paged, independently idempotent per-key mutations |
|
|||
|
|
| F12 | Search by key substring under the active root |
|
|||
|
|
| F13 | Enforce configurable personal and shared storage quotas in PostgreSQL under the mutation-integrity invariants in §3.7 |
|
|||
|
|
| F14 | Enforce a configurable shared token-bucket byte throttle per user via Redis-compatible store for **upload and download** |
|
|||
|
|
| F15 | List and metadata cache in Redis-compatible store with configurable TTL |
|
|||
|
|
| F16 | Structured audit logs on stdout |
|
|||
|
|
| F17 | Require operation ids for put, overwrite, and delete; within the advertised idempotency window, replay exact requests without starting a second effect, preserve terminal results, and reject conflicting reuse; reject new mutations while their canonical key is locked |
|
|||
|
|
| F18 | Bound search work by configurable scan and timeout limits |
|
|||
|
|
| F19 | Use safe attachment-oriented download headers and prohibit inline rendering of active stored content |
|
|||
|
|
|
|||
|
|
### 4.2 Non-functional
|
|||
|
|
|
|||
|
|
| ID | Requirement |
|
|||
|
|
| -- | ----------- |
|
|||
|
|
| N1 | Stateless app processes. No sticky sessions. No in-process cache or throttle authority |
|
|||
|
|
| N2 | No object-store credentials in the browser |
|
|||
|
|
| N3 | No object-store admin console dependency for user workflows |
|
|||
|
|
| N4 | Stream bodies. Do not buffer whole objects in memory |
|
|||
|
|
| N5 | Design is language-agnostic. Language is chosen at implementation start |
|
|||
|
|
| N6 | Preserve mutation integrity under concurrent replicas and recover unfinished mutations after process restart or transient dependency loss |
|
|||
|
|
| N7 | Cache failure degrades to direct S3 access; throttle failure does not silently disable bandwidth enforcement |
|
|||
|
|
|
|||
|
|
### 4.3 Security
|
|||
|
|
|
|||
|
|
| ID | Requirement |
|
|||
|
|
| -- | ----------- |
|
|||
|
|
| S1 | Confidential OIDC client using authorization code with PKCE. Secrets from the deployment secret store |
|
|||
|
|
| S2 | JWT signed with an app key. Cookie: Secure, HttpOnly, host-only, `Path=/`, SameSite=Lax |
|
|||
|
|
| S3 | CSRF token required on mutating cookie-authenticated requests |
|
|||
|
|
| S4 | Construct keys server-side from a root enum and canonical relative path; reject ambiguous, absolute, traversal, and cross-root forms |
|
|||
|
|
| S5 | User A cannot access another identity’s `users/{user_id}/` root |
|
|||
|
|
| S6 | Shared root requires authentication |
|
|||
|
|
| S7 | No anonymous access to object data. Unauthenticated callers may only hit login, callback, logout, health, and static entrypoints explicitly intended for them |
|
|||
|
|
| S8 | Validate OIDC `state`, `nonce`, issuer, audience, signature, redirect URI, and token lifetime |
|
|||
|
|
| S9 | Do not log tokens, secrets, CSRF values, object bodies, or raw authorization headers |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 5. Scope
|
|||
|
|
|
|||
|
|
### 5.1 In scope
|
|||
|
|
|
|||
|
|
| Area |
|
|||
|
|
| ---- |
|
|||
|
|
| OIDC (Nuthatch as RP), transient login transaction state, and JWT session cookie |
|
|||
|
|
| CSRF on mutating cookie-authenticated requests |
|
|||
|
|
| Active root selection (personal or shared) |
|
|||
|
|
| Paginated browse, upload, download |
|
|||
|
|
| Search by key substring under active root |
|
|||
|
|
| Delete object and recursive prefix with per-key usage updates |
|
|||
|
|
| PostgreSQL quotas and durable mutation metadata required by §3.7 and the protocol boundary in §3.13 |
|
|||
|
|
| Redis-compatible shared token-bucket throttle for upload and download bytes |
|
|||
|
|
| Redis-compatible list and metadata cache (configurable TTL) |
|
|||
|
|
| S3-compatible object storage via a service principal |
|
|||
|
|
| stdout audit logs |
|
|||
|
|
|
|||
|
|
### 5.2 Out of scope
|
|||
|
|
|
|||
|
|
| Item |
|
|||
|
|
| ---- |
|
|||
|
|
| Anonymous or time-limited public links |
|
|||
|
|
| Desktop client or sync |
|
|||
|
|
| Office or WOPI |
|
|||
|
|
| Object versioning UI |
|
|||
|
|
| In-browser previews |
|
|||
|
|
| Range downloads |
|
|||
|
|
| Server-side upload resumption |
|
|||
|
|
| WebDAV or SFTP |
|
|||
|
|
| Per-user object-store IAM users |
|
|||
|
|
| Browser presigned object-store URLs |
|
|||
|
|
| App-local durable disk for product data |
|
|||
|
|
| In-process throttle or cache as authority |
|
|||
|
|
| Redis or PostgreSQL as session store |
|
|||
|
|
| Standing bucket-list reconciler as the primary mutation-correctness mechanism |
|
|||
|
|
| Request-rate or concurrency throttle dimensions |
|
|||
|
|
| Fixed quota or throttle numeric defaults in this document |
|
|||
|
|
| Per-object ACLs on the shared root |
|
|||
|
|
| External auth proxy in front of Nuthatch |
|
|||
|
|
| Virus scan or DLP |
|
|||
|
|
| Teams or orgs beyond the single shared prefix |
|
|||
|
|
| Cluster ingress, DNS, network policy, and GitOps wiring |
|
|||
|
|
| Mutation state machine, transition ordering, and proof obligations (owned by the protocol model) |
|
|||
|
|
| Test harness and environment design |
|