ms2k/remora
SHA256
Watch
0
Code Releases Activity

Model checkpoint staging ownership transfer

Browse source
This commit is contained in:
ms2k 2026-07-17 15:01:15 +09:00
commit 16cba3c317
1 changed files with 111 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

7
docs/spec/architecture.md
View file

@ -107,7 +107,7 @@ flowchart TD
- Activation and inactive-attempt recovery serialize against the same durable authority. Recovery revalidates coordination after entering that boundary and reaps an attempt that remains authorized; compensation removes any pre-commit lease. Activation removes any lease that cannot be paired with the durable executing tuple.
- Reservations have a bounded lifetime. Expiry makes unallocated work eligible for redispatch and makes a durably authorized attempt without its exact live lease eligible for reaping; it never reverses durable authority from coordination state alone.
- A lost successful activation response replays success only while the exact live lease and complete durable executing tuple still agree. A changed attempt, fence, or worker incarnation is a precondition failure.
- The envelope projects the accepted non-secret source metadata and a sensitive-input reference, any durable `ResolvedSource`, the immutable execution grant and concrete target, the attempt-scoped staging identity, pinned credential-version references, and the validated checkpoint and destination-resume evidence available to that attempt.
- The envelope projects the accepted non-secret source metadata and a sensitive-input reference, any durable `ResolvedSource`, the immutable execution grant and concrete target, the attempt's exact staging identity, pinned credential-version references, and the validated checkpoint and destination-resume evidence available to that attempt. A replacement may inherit the predecessor's physical staging identity only through the durable checkpoint-ownership transfer defined below.
- A source identity or inventory that is initially unresolved is durably pinned once before concrete checkpoint validation or reuse relies on it. Pinning is a fenced durable mutation and a conflicting value is rejected.
- Large source inventories, completed-part sets, and other unbounded collections are represented by durable streamable descriptors rather than materialized in the envelope.
- Envelope transport is an implementation choice. It may accompany queue delivery or claim, or be read separately, but a worker never infers it from reconstructible coordination state.
@ -136,12 +136,15 @@ flowchart TD
- Concrete work for `CleanupStage` is materialized from exact attempt ownership and pending cleanup data in `DurableStore`, independently of `CoordinationStore`.
- Ambient cleanup processes are stateless and idempotent. Worker-local cleanup may reduce latency but is not the only cleanup path.
- Cleanup addresses exact attempt object or multipart identities. Eligibility comes only from `CleanupStage`; missing coordination state, target-object age, and raw prefix membership are not substitutes.
- Every reusable checkpoint has one durable logical staging owner. Its exact retained objects, multipart state, inventory rows, and pending cleanup identities are ineligible for cleanup while the job is nonterminal.
- A replacement is authorized only after the checkpoint owner is durably stopped. Authorization atomically transfers logical ownership and all retained exact identities to the replacement before it can perform I/O. Physical keys may retain their original globally unique attempt namespace; key prefixes do not determine current logical ownership.
- State beyond the checkpoint's validated boundary remains owned by the predecessor and becomes independently cleanup-eligible after transfer. A claimed or completed cleanup identity cannot be transferred or used as resume evidence.
- Target lifecycle rules are storage hygiene only. They do not substitute for `JobProtocol` publication or cleanup eligibility and must not expire objects retained by a committed aggregate.
- Independent writers and lifecycle mechanisms must be unable to replace or expire live or committed identities in Remora's reserved namespace. A deployment that cannot establish this property is unsupported.
## Source Identity and Resumption
- `checkpointValid` abstracts the concrete proof defined here. A reusable checkpoint binds an immutable `Source` identity, an exact completed boundary, and validated destination state; absent or changed proof makes the checkpoint unavailable to a replacement attempt.
- `checkpointValid` abstracts the concrete proof defined here. A reusable checkpoint binds an immutable `Source` identity, an exact completed boundary, validated destination state, and its current durable staging owner; absent or changed proof makes the checkpoint unavailable to a replacement attempt.
- HTTP identity includes the resolved final URL, a strong representation validator as defined by `RFC 9110`, representation length, and identity content encoding. Resume requires a conditional byte-range response whose validator, URL, encoding, and range exactly match the checkpoint. Weak validators, length, or modification time alone do not qualify.
- Object-store identity includes endpoint, bucket, key, size, and an immutable version ID when available. Without a version ID, resume requires a matching conditional range read. Destination multipart reuse additionally requires the same endpoint, bucket, key, upload ID, and exact re-listed completed parts.
- An HTTP or object-store single-object source supplies one explicit `result_path` validated under the manifest member-path rules. The adapter does not derive the result member path from a URL or object key.