ms2k/remora
SHA256
Watch
0
Code Releases Activity

Define manifest-rooted result architecture

Browse source
This commit is contained in:
ms2k 2026-07-13 12:43:55 +09:00
commit a9f611e4f0
2 changed files with 24 additions and 47 deletions
Download patch file Download diff file Expand all files Collapse all files

5
docs/protocol/Remora.tla
View file

@ -7,8 +7,9 @@ EXTENDS Naturals, FiniteSets, TLC
\* allocation, fencing tokens, and committed results. CoordinationStore variables
\* are reconstructible: queues, reservations, leases, cancel notices, live progress.
\* Attempts remain observable after losing authority so delayed outcomes and zombie
\* target writes are checked. Each attempt uses an attempt-specific staging identity;
\* only the durable AcceptSuccess compare-and-swap publishes a CommittedResult.
\* target writes are checked. An attempt staging identity abstracts its immutable
\* member set and result manifest; only the durable AcceptSuccess compare-and-swap
\* selects that manifest descriptor as the CommittedResult.
\*
\* Abstractions and bounds:
\* - Jobs in Init are already durably accepted (submission outbox precedes queueing).

66
docs/spec/architecture.md
View file

@ -13,8 +13,9 @@ Terms in this glossary are stable identifiers for use across Remora documentatio
| `WorkerIncarnation` | One lifetime of a `Worker` process. A restarted process is a new incarnation even when it occupies the same replica slot. |
| `Claim` | Durable authority for one `Attempt`, identified by its attempt ID, worker incarnation, and monotonically increasing fencing token. |
| `Lease` | Reconstructible coordination state granting a live attempt a deployment-wide concurrency permit for a bounded interval. |
| `StagingObject` | An immutable, attempt-specific target identity that is not externally visible as the result of a job. |
| `CommittedResult` | The immutable object identity selected by the authoritative durable commit for a successful job. |
| `StagingObject` | An immutable, attempt-specific result member or manifest object that has not been selected by durable publication. |
| `ResultManifest` | The immutable exact inventory and root identity for one attempt's result aggregate. |
| `CommittedResult` | The durable descriptor that selects one `ResultManifest` for a successful job. |
| `WebClient` | The browser-based user interface for submitting and managing jobs. |
| `API` | The authenticated service interface for job submission, status, cancellation, history, and deletion. |
| `Worker` | A stateless process that executes attempts and reports progress and outcomes. |
@ -28,9 +29,9 @@ Terms in this glossary are stable identifiers for use across Remora documentatio
## Scope
This document defines Remora's product requirements, system boundaries, major components, and allocation of responsibilities. It intentionally does not define the distributed job protocol or implementation-level technology choices.
This document defines Remora's product requirements, system boundaries, major components, result representation and access contract, and allocation of responsibilities. It intentionally does not define the distributed job protocol or implementation-level technology choices.
`JobProtocol` defines job state transitions, at-least-once delivery, claims and fencing, retry scheduling, cancellation races, recovery, and the associated safety and liveness properties. Concrete frameworks, libraries, and deployment choices are outside the scope of this specification.
`JobProtocol` defines job state transitions, at-least-once delivery, claims and fencing, retry scheduling, cancellation races, recovery, and the associated safety and liveness properties. Concrete frameworks, libraries, wire formats, and deployment choices are outside the scope of this specification.
## Problem Statement
@ -40,7 +41,7 @@ Pulling large objects from remote sources into S3-compatible storage is commonly
Remora exposes a browser UI and API for submitting, monitoring, and managing download jobs into a target object store. Supported source modes are HTTP URLs, compatible object stores, and Hugging Face repository snapshots.
Each download is an independent background job with streaming I/O, automatic retry, live progress, and cancellation. A failed attempt resumes when source and destination support safe resumption; otherwise its replacement starts from the beginning. Jobs default to a configured target bucket and allow per-request bucket and prefix overrides subject to authorization and validation.
Each download is an independent background job with streaming I/O, automatic retry, live progress, and cancellation. A failed attempt resumes when source and destination support safe resumption; otherwise its replacement starts from the beginning. Jobs default to a configured target bucket and allow per-request bucket and prefix overrides subject to authorization and validation. Every successful job publishes one immutable `ResultManifest`, including single-object jobs.
Identity, coordination, and durable state are external services. Multiple API and worker replicas share them for authenticated multi-user operation.
@ -48,9 +49,9 @@ Identity, coordination, and durable state are external services. Multiple API an
`WebClient` communicates with `API`, which authorizes requests through `IdentityProvider`, records jobs in `DurableStore`, and submits work through `CoordinationStore`. `Worker` replicas consume jobs and stream data from `Source` through `StorageAdapter` into `TargetStore`.
`API` and `Worker` replicas are stateless. `DurableStore` is authoritative for accepted jobs, cancellation intent, attempt allocation, fencing tokens, retry consumption, terminal outcomes, and the identity of a committed result. `CoordinationStore` contains reconstructible queue entries, leases, deployment-wide permits, cancellation notifications, and live progress. Loss of coordination state must not change durable authority or expose an uncommitted result.
`API` and `Worker` replicas are stateless. `DurableStore` contains authoritative job records and committed result descriptors. `CoordinationStore` contains reconstructible operational state such as queue entries, leases, deployment-wide permits, cancellation notifications, and live progress.
At-least-once delivery may leave multiple attempt processes alive. Every attempt writes only to its own `StagingObject`. A successful attempt becomes externally visible only when a compare-and-swap transition in `DurableStore` selects its attempt ID, fencing token, and immutable object identity as the `CommittedResult`. A stale, expired, or cancelled attempt may finish writing staging data, but cannot publish it.
`TargetStore` contains immutable attempt-scoped objects. A `ResultManifest` roots each attempt-scoped aggregate; `DurableStore` contains the manifest descriptor exposed as the `CommittedResult`. `JobProtocol` is the sole specification of attempt authority and publication behavior.
Workers enforce a configurable outbound-source policy for user-supplied locations on every connection, including redirects. Deployments also restrict worker egress as a defense-in-depth boundary. Private or otherwise sensitive destinations are denied by default unless explicitly allowed by deployment policy.
@ -95,40 +96,15 @@ flowchart TD
StorageAdapter --> TargetStore
```
## Architectural Semantics
## Result Aggregate and Direct Access
### Authority and Recovery
- `DurableStore` is the sole authority for job phase, cancellation intent, attempt count, current fencing token, current authoritative attempt, and terminal result.
- Every authority-changing request carries a job ID, attempt ID, worker incarnation, fencing token, and idempotency identity. Repeating a request has the same authoritative effect as processing it once.
- Submission creates a durable dispatch record before work is placed in `CoordinationStore`. A reconciler repeatedly projects eligible durable jobs, cancellation intent, and expired work into coordination state.
- `CoordinationStore` loss may discard queues, leases, permits, progress, and notifications. Those records are rebuilt from durable state; they never override a durable terminal outcome.
- A replacement attempt receives a new attempt ID and a strictly greater fencing token. Creating that authoritative replacement consumes retry budget, including when the previous attempt was lost without reporting an outcome.
### Publication and Cancellation
- Attempts never write directly to the authoritative result identity. Each attempt streams into an immutable attempt-specific staging or multipart identity.
- Publication linearizes at one durable compare-and-swap that verifies the current attempt ID and fencing token, verifies that cancellation has not won, and records the immutable committed object identity.
- Readers resolve only the `CommittedResult`. Uncommitted staging data is never presented as a job result and is deleted asynchronously after it can no longer be committed.
- `Cancelled` means that durable publication is permanently forbidden. It does not assert that every partitioned process has stopped or that staging cleanup has completed.
- Cancellation is propagated cooperatively to reachable workers. Lease loss or cancellation causes a connected worker to stop promptly, but publication safety does not depend on observing that stop.
### Progress and Resumption
- Live attempt progress is ephemeral and scoped to the current fencing token.
- Durable resume progress represents only a validated checkpoint, completed multipart range, or equivalent reusable state. A replacement may reuse it only when source identity and destination state validators still match.
- User-visible progress combines validated durable progress with current-attempt progress. If validation fails, reusable progress resets and the UI identifies that a new attempt restarted rather than presenting stale progress as completed work.
- Durable progress and attempt history may be coalesced; correctness must not depend on persisting every live progress event.
### Concurrency Semantics
- A per-incarnation local limiter bounds tasks executing in one live `Worker` process.
- The deployment-wide limit bounds live authoritative leases and reservations in `CoordinationStore`.
- A partitioned or paused attempt can temporarily continue physical I/O after its lease expires. Such zombie work is isolated in staging and does not count as an authoritative lease. A strict bound on all physical I/O during partitions requires infrastructure-level enforcement outside `JobProtocol`.
### Availability Assumptions
Safety requirements hold during worker failure, delayed requests, coordination loss, and temporary service partitions. Eventual completion is required only when durable storage and coordination recover, a worker incarnation remains available long enough to run an attempt, required source and target operations eventually return, lease time advances, and retry policy permits another attempt.
- Every successful job is represented by one immutable `ResultManifest`. A single-object transfer uses a one-member manifest.
- A manifest is the exact inventory of immutable member identities and carries sufficient integrity metadata to validate the aggregate.
- Manifest construction uses memory bounded independently of member count and total result size.
- Remora resolves a result only from its `CommittedResult`; it never derives one by listing a target bucket or prefix.
- An authorized reader may access committed members directly in `TargetStore` using exact manifest identities or time-limited URLs.
- Raw target listing is operational storage visibility, not result resolution. Clients with independent target credentials may observe uncommitted attempt objects.
- A stable user-chosen object key or atomically visible raw target prefix is not part of a job result. Providing one requires a separate export contract.
## Requirements
@ -141,15 +117,15 @@ Safety requirements hold during worker failure, delayed requests, coordination l
| FR-3 | Support `Job` sources using HTTP URLs, compatible object stores, and Hugging Face repositories | High |
| FR-4 | Allow per-`Job` target bucket and prefix overrides subject to authorization and validation | High |
| FR-5 | Provide attempt-aware live progress, validated resume progress, and cancellation for in-flight jobs | High |
| FR-6 | Retry failed or lost attempts according to `JobProtocol`, using a new attempt identity and fencing token | High |
| FR-6 | Retry failed or lost work according to `JobProtocol` | High |
| FR-7 | Maintain durable job and attempt history, including terminal outcomes and structured failure information | Medium |
| FR-8 | Support soft deletion by tombstoning job metadata without deleting stored objects | Medium |
| FR-9 | Support hard deletion by deleting stored objects while retaining a metadata tombstone | Medium |
| FR-10 | Stream data without buffering complete objects in `Worker` memory | High |
| FR-10 | Stream data and construct result inventories with memory bounded independently of complete object and aggregate size | High |
| FR-11 | Enforce independently configurable per-incarnation task limits and deployment-wide authoritative lease limits | Medium |
| FR-12 | Isolate every attempt's target writes and expose at most one durably committed immutable result per job | High |
| FR-13 | Make submission, dispatch, claiming, renewal, cancellation, progress, and finalization safe under duplicate or ambiguous requests | High |
| FR-14 | Reconstruct operational coordination state from durable state after coordination loss or failover | High |
| FR-12 | Isolate attempt target writes and expose exactly one immutable `ResultManifest` for each successful job | High |
| FR-13 | Ensure distributed job operations refine `JobProtocol` under duplicate or ambiguous requests | High |
| FR-14 | Reconstruct operational coordination state after loss or failover as required by `JobProtocol` | High |
| FR-15 | Enforce a configurable outbound-source policy for user-supplied locations and redirects | High |
### Non-Functional Requirements