148 lines
12 KiB
Markdown
148 lines
12 KiB
Markdown
# Remora Architecture Specification
|
|
|
|
**Remora** is a multi-user service that runs durable download jobs from remote sources into S3-compatible object storage.
|
|
|
|
## Glossary
|
|
|
|
Terms in this glossary are stable identifiers for use across Remora documentation and protocol models.
|
|
|
|
| Identifier | Definition |
|
|
| ---------- | ---------- |
|
|
| `Job` | A durable request to transfer one object or multi-object source snapshot from a `Source` to a `TargetStore`. |
|
|
| `Attempt` | One execution of a `Job`. At-least-once delivery may produce more than one `Attempt` for a `Job`. |
|
|
| `WorkerIncarnation` | One lifetime of a `Worker` process. A restarted process is a new incarnation even when it occupies the same replica slot. |
|
|
| `Claim` | Durable authority for one `Attempt`, identified by its attempt ID, worker incarnation, and monotonically increasing fencing token. |
|
|
| `Lease` | Reconstructible coordination state granting a live attempt a deployment-wide concurrency permit for a bounded interval. |
|
|
| `StagingObject` | An immutable, attempt-specific result member or manifest object that has not been selected by durable publication. |
|
|
| `ResultManifest` | The immutable exact inventory and root identity for one attempt's result aggregate. |
|
|
| `CommittedResult` | The durable descriptor that selects one `ResultManifest` for a successful job. |
|
|
| `WebClient` | The browser-based user interface for submitting and managing jobs. |
|
|
| `API` | The authenticated service interface for job submission, status, cancellation, history, and deletion. |
|
|
| `Worker` | A stateless process that executes attempts and reports progress and outcomes. |
|
|
| `IdentityProvider` | The external authority for user identity, administrative roles, and storage authorization scopes. |
|
|
| `CoordinationStore` | Shared operational state for queued work, in-flight coordination, live progress, cancellation, and deployment-wide concurrency. |
|
|
| `DurableStore` | The source of truth for job identity, ownership, configuration, history, and terminal outcomes. |
|
|
| `Source` | An HTTP endpoint, compatible object store, or Hugging Face repository from which a job reads. |
|
|
| `StorageAdapter` | The service boundary through which workers write to the target object store. |
|
|
| `TargetStore` | The configured S3-compatible object store receiving downloaded objects. |
|
|
| `JobProtocol` | The TLA+ specification defining distributed job state, coordination semantics, and safety and liveness properties. |
|
|
|
|
## Scope
|
|
|
|
This document defines Remora's product requirements, system boundaries, major components, result representation and access contract, and allocation of responsibilities. It intentionally does not define the distributed job protocol or implementation-level technology choices.
|
|
|
|
`JobProtocol` defines job state transitions, at-least-once delivery, claims and fencing, retry scheduling, cancellation races, recovery, and the associated safety and liveness properties. Concrete frameworks, libraries, wire formats, and deployment choices are outside the scope of this specification.
|
|
|
|
## Problem Statement
|
|
|
|
Pulling large objects from remote sources into S3-compatible storage is commonly done with one-off CLI tools or scripts. Those approaches are hard to monitor for long transfers, lack shared retry and resumption, offer little cross-user visibility, and do not enforce authorization over target placement.
|
|
|
|
## Proposed Solution
|
|
|
|
Remora exposes a browser UI and API for submitting, monitoring, and managing download jobs into a target object store. Supported source modes are HTTP URLs, compatible object stores, and Hugging Face repository snapshots.
|
|
|
|
Each download is an independent background job with streaming I/O, automatic retry, live progress, and cancellation. A failed attempt resumes when source and destination support safe resumption; otherwise its replacement starts from the beginning. Jobs default to a configured target bucket and allow per-request bucket and prefix overrides subject to authorization and validation. Every successful job publishes one immutable `ResultManifest`, including single-object jobs.
|
|
|
|
Identity, coordination, and durable state are external services. Multiple API and worker replicas share them for authenticated multi-user operation.
|
|
|
|
## Architecture
|
|
|
|
`WebClient` communicates with `API`, which authorizes requests through `IdentityProvider`, records jobs in `DurableStore`, and submits work through `CoordinationStore`. `Worker` replicas consume jobs and stream data from `Source` through `StorageAdapter` into `TargetStore`.
|
|
|
|
`API` and `Worker` replicas are stateless. `DurableStore` contains authoritative job records and committed result descriptors. `CoordinationStore` contains reconstructible operational state such as queue entries, leases, deployment-wide permits, cancellation notifications, and live progress.
|
|
|
|
`TargetStore` contains immutable attempt-scoped objects. A `ResultManifest` roots each attempt-scoped aggregate; `DurableStore` contains the manifest descriptor exposed as the `CommittedResult`. `JobProtocol` is the sole specification of attempt authority and publication behavior.
|
|
|
|
Workers enforce a configurable outbound-source policy for user-supplied locations on every connection, including redirects. Deployments also restrict worker egress as a defense-in-depth boundary. Private or otherwise sensitive destinations are denied by default unless explicitly allowed by deployment policy.
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
subgraph Client
|
|
WebClient[WebClient]
|
|
end
|
|
|
|
subgraph Identity
|
|
IdentityProvider[IdentityProvider]
|
|
end
|
|
|
|
subgraph Service
|
|
API[API]
|
|
Worker[Worker]
|
|
end
|
|
|
|
subgraph State
|
|
CoordinationStore[CoordinationStore]
|
|
DurableStore[DurableStore]
|
|
end
|
|
|
|
subgraph Sources
|
|
Source[Source]
|
|
end
|
|
|
|
subgraph Target
|
|
StorageAdapter[StorageAdapter]
|
|
TargetStore[TargetStore]
|
|
end
|
|
|
|
WebClient -- OIDC --> IdentityProvider
|
|
WebClient -- API + live updates --> API
|
|
IdentityProvider -- Identity and roles --> API
|
|
API -- Submit and coordinate --> CoordinationStore
|
|
API -- Record and query --> DurableStore
|
|
Worker -- Consume and coordinate --> CoordinationStore
|
|
Worker -- Report durable outcomes --> DurableStore
|
|
Worker -- Stream from --> Source
|
|
Worker -- Stream to --> StorageAdapter
|
|
StorageAdapter --> TargetStore
|
|
```
|
|
|
|
## Result Aggregate and Direct Access
|
|
|
|
- Every successful job is represented by one immutable `ResultManifest`. A single-object transfer uses a one-member manifest.
|
|
- A manifest is the exact inventory of immutable member identities and carries sufficient integrity metadata to validate the aggregate.
|
|
- Manifest construction uses memory bounded independently of member count and total result size.
|
|
- Remora resolves a result only from its `CommittedResult`; it never derives one by listing a target bucket or prefix.
|
|
- An authorized reader may access committed members directly in `TargetStore` using exact manifest identities or time-limited URLs.
|
|
- Raw target listing is operational storage visibility, not result resolution. Clients with independent target credentials may observe uncommitted attempt objects.
|
|
- A stable user-chosen object key or atomically visible raw target prefix is not part of a job result. Providing one requires a separate export contract.
|
|
|
|
## Requirements
|
|
|
|
### Functional Requirements
|
|
|
|
| ID | Requirement | Priority |
|
|
| ---- | ------------------------------------------------------------------------------------------------------------------------------------- | -------- |
|
|
| FR-1 | Authenticate users through `IdentityProvider` and derive administrative and storage scopes from its roles | High |
|
|
| FR-2 | Enforce bucket and prefix scopes for non-administrative users | High |
|
|
| FR-3 | Support `Job` sources using HTTP URLs, compatible object stores, and Hugging Face repositories | High |
|
|
| FR-4 | Allow per-`Job` target bucket and prefix overrides subject to authorization and validation | High |
|
|
| FR-5 | Provide attempt-aware live progress, validated resume progress, and cancellation for in-flight jobs | High |
|
|
| FR-6 | Retry failed or lost work according to `JobProtocol` | High |
|
|
| FR-7 | Maintain durable job and attempt history, including terminal outcomes and structured failure information | Medium |
|
|
| FR-8 | Support soft deletion by tombstoning job metadata without deleting stored objects | Medium |
|
|
| FR-9 | Support hard deletion by deleting stored objects while retaining a metadata tombstone | Medium |
|
|
| FR-10 | Stream data and construct result inventories with memory bounded independently of complete object and aggregate size | High |
|
|
| FR-11 | Enforce independently configurable per-incarnation task limits and deployment-wide authoritative lease limits | Medium |
|
|
| FR-12 | Isolate attempt target writes and expose exactly one immutable `ResultManifest` for each successful job | High |
|
|
| FR-13 | Ensure distributed job operations refine `JobProtocol` under duplicate or ambiguous requests | High |
|
|
| FR-14 | Reconstruct operational coordination state after loss or failover as required by `JobProtocol` | High |
|
|
| FR-15 | Enforce a configurable outbound-source policy for user-supplied locations and redirects | High |
|
|
|
|
### Non-Functional Requirements
|
|
|
|
| Category | Requirement |
|
|
| ------------- | ----------------------------------------------------------------------------------------------------------- |
|
|
| Reliability | Preserve durable authority and committed results across worker or coordination failures; reconcile in-flight work by protocol |
|
|
| Observability | Expose job status, progress, and errors through the UI and API |
|
|
| Scalability | Support multiple stateless API and worker replicas using shared state services |
|
|
| Performance | Stream multi-gigabyte objects without buffering complete objects in worker memory |
|
|
| Security | Authenticate and authorize all operations; do not expose long-lived source or target credentials to the UI |
|
|
| Deployment | Deploy as a standard Kubernetes service using externally managed identity, state, and object storage |
|
|
|
|
## Responsibility Boundaries
|
|
|
|
- This architecture specification defines components, trust boundaries, externally visible behavior, and responsibility allocation.
|
|
- `JobProtocol` defines distributed state and coordination semantics, including safety and liveness invariants.
|
|
- `JobProtocol` is the normative implementation contract for durable transitions, fencing, leases, retries, cancellation, publication, and reconciliation. Implementations may optimize storage layout but must refine its observable behavior and preserve its linearization points.
|
|
- Implementation documentation selects technologies and maps architectural components onto them.
|
|
- Deployment configuration supplies service credentials and endpoints, configures allowed source reachability, and provides infrastructure-level egress enforcement.
|