remora/docs/spec/architecture.md

Remora Architecture Specification

Remora is a multi-user service that runs durable download jobs from remote sources into S3-compatible object storage.

Glossary

Terms in this glossary are stable identifiers for use across Remora documentation and protocol models.

Identifier	Definition
Job	A durable request to transfer one object or multi-object source snapshot from a Source to a TargetStore.
Attempt	One execution of a Job. At-least-once delivery may produce more than one Attempt for a Job.
WorkerIncarnation	One lifetime of a Worker process. A restarted process is a new incarnation even when it occupies the same replica slot.
Claim	Durable authority for one Attempt, identified by its attempt ID, worker incarnation, and monotonically increasing fencing token.
Lease	Reconstructible coordination state granting a live attempt a deployment-wide concurrency permit for a bounded interval.
StagingObject	An immutable, attempt-specific target identity that is not externally visible as the result of a job.
CommittedResult	The immutable object identity selected by the authoritative durable commit for a successful job.
WebClient	The browser-based user interface for submitting and managing jobs.
API	The authenticated service interface for job submission, status, cancellation, history, and deletion.
Worker	A stateless process that executes attempts and reports progress and outcomes.
IdentityProvider	The external authority for user identity, administrative roles, and storage authorization scopes.
CoordinationStore	Shared operational state for queued work, in-flight coordination, live progress, cancellation, and deployment-wide concurrency.
DurableStore	The source of truth for job identity, ownership, configuration, history, and terminal outcomes.
Source	An HTTP endpoint, compatible object store, or Hugging Face repository from which a job reads.
StorageAdapter	The service boundary through which workers write to the target object store.
TargetStore	The configured S3-compatible object store receiving downloaded objects.
JobProtocol	The TLA+ specification defining distributed job state, coordination semantics, and safety and liveness properties.

Scope

This document defines Remora's product requirements, system boundaries, major components, and allocation of responsibilities. It intentionally does not define the distributed job protocol or implementation-level technology choices.

JobProtocol defines job state transitions, at-least-once delivery, claims and fencing, retry scheduling, cancellation races, recovery, and the associated safety and liveness properties. Concrete frameworks, libraries, and deployment choices are outside the scope of this specification.

Problem Statement

Pulling large objects from remote sources into S3-compatible storage is commonly done with one-off CLI tools or scripts. Those approaches are hard to monitor for long transfers, lack shared retry and resumption, offer little cross-user visibility, and do not enforce authorization over target placement.

Proposed Solution

Remora exposes a browser UI and API for submitting, monitoring, and managing download jobs into a target object store. Supported source modes are HTTP URLs, compatible object stores, and Hugging Face repository snapshots.

Each download is an independent background job with streaming I/O, automatic retry, live progress, and cancellation. A failed attempt resumes when source and destination support safe resumption; otherwise its replacement starts from the beginning. Jobs default to a configured target bucket and allow per-request bucket and prefix overrides subject to authorization and validation.

Identity, coordination, and durable state are external services. Multiple API and worker replicas share them for authenticated multi-user operation.

Architecture

WebClient communicates with API, which authorizes requests through IdentityProvider, records jobs in DurableStore, and submits work through CoordinationStore. Worker replicas consume jobs and stream data from Source through StorageAdapter into TargetStore.

API and Worker replicas are stateless. DurableStore is authoritative for accepted jobs, cancellation intent, attempt allocation, fencing tokens, retry consumption, terminal outcomes, and the identity of a committed result. CoordinationStore contains reconstructible queue entries, leases, deployment-wide permits, cancellation notifications, and live progress. Loss of coordination state must not change durable authority or expose an uncommitted result.

At-least-once delivery may leave multiple attempt processes alive. Every attempt writes only to its own StagingObject. A successful attempt becomes externally visible only when a compare-and-swap transition in DurableStore selects its attempt ID, fencing token, and immutable object identity as the CommittedResult. A stale, expired, or cancelled attempt may finish writing staging data, but cannot publish it.

Workers enforce a configurable outbound-source policy for user-supplied locations on every connection, including redirects. Deployments also restrict worker egress as a defense-in-depth boundary. Private or otherwise sensitive destinations are denied by default unless explicitly allowed by deployment policy.

flowchart TD
    subgraph Client
        WebClient[WebClient]
    end

    subgraph Identity
        IdentityProvider[IdentityProvider]
    end

    subgraph Service
        API[API]
        Worker[Worker]
    end

    subgraph State
        CoordinationStore[CoordinationStore]
        DurableStore[DurableStore]
    end

    subgraph Sources
        Source[Source]
    end

    subgraph Target
        StorageAdapter[StorageAdapter]
        TargetStore[TargetStore]
    end

    WebClient -- OIDC --> IdentityProvider
    WebClient -- API + live updates --> API
    IdentityProvider -- Identity and roles --> API
    API -- Submit and coordinate --> CoordinationStore
    API -- Record and query --> DurableStore
    Worker -- Consume and coordinate --> CoordinationStore
    Worker -- Report durable outcomes --> DurableStore
    Worker -- Stream from --> Source
    Worker -- Stream to --> StorageAdapter
    StorageAdapter --> TargetStore

Architectural Semantics

Authority and Recovery

• DurableStore is the sole authority for job phase, cancellation intent, attempt count, current fencing token, current authoritative attempt, and terminal result.
• Every authority-changing request carries a job ID, attempt ID, worker incarnation, fencing token, and idempotency identity. Repeating a request has the same authoritative effect as processing it once.
• Submission creates a durable dispatch record before work is placed in CoordinationStore. A reconciler repeatedly projects eligible durable jobs, cancellation intent, and expired work into coordination state.
• CoordinationStore loss may discard queues, leases, permits, progress, and notifications. Those records are rebuilt from durable state; they never override a durable terminal outcome.
• A replacement attempt receives a new attempt ID and a strictly greater fencing token. Creating that authoritative replacement consumes retry budget, including when the previous attempt was lost without reporting an outcome.

Publication and Cancellation

• Attempts never write directly to the authoritative result identity. Each attempt streams into an immutable attempt-specific staging or multipart identity.
• Publication linearizes at one durable compare-and-swap that verifies the current attempt ID and fencing token, verifies that cancellation has not won, and records the immutable committed object identity.
• Readers resolve only the CommittedResult. Uncommitted staging data is never presented as a job result and is deleted asynchronously after it can no longer be committed.
• Cancelled means that durable publication is permanently forbidden. It does not assert that every partitioned process has stopped or that staging cleanup has completed.
• Cancellation is propagated cooperatively to reachable workers. Lease loss or cancellation causes a connected worker to stop promptly, but publication safety does not depend on observing that stop.

Progress and Resumption

• Live attempt progress is ephemeral and scoped to the current fencing token.
• Durable resume progress represents only a validated checkpoint, completed multipart range, or equivalent reusable state. A replacement may reuse it only when source identity and destination state validators still match.
• User-visible progress combines validated durable progress with current-attempt progress. If validation fails, reusable progress resets and the UI identifies that a new attempt restarted rather than presenting stale progress as completed work.
• Durable progress and attempt history may be coalesced; correctness must not depend on persisting every live progress event.

Concurrency Semantics

• A per-incarnation local limiter bounds tasks executing in one live Worker process.
• The deployment-wide limit bounds live authoritative leases and reservations in CoordinationStore.
• A partitioned or paused attempt can temporarily continue physical I/O after its lease expires. Such zombie work is isolated in staging and does not count as an authoritative lease. A strict bound on all physical I/O during partitions requires infrastructure-level enforcement outside JobProtocol.

Availability Assumptions

Safety requirements hold during worker failure, delayed requests, coordination loss, and temporary service partitions. Eventual completion is required only when durable storage and coordination recover, a worker incarnation remains available long enough to run an attempt, required source and target operations eventually return, lease time advances, and retry policy permits another attempt.

Requirements

Functional Requirements

ID	Requirement	Priority
FR-1	Authenticate users through IdentityProvider and derive administrative and storage scopes from its roles	High
FR-2	Enforce bucket and prefix scopes for non-administrative users	High
FR-3	Support Job sources using HTTP URLs, compatible object stores, and Hugging Face repositories	High
FR-4	Allow per-Job target bucket and prefix overrides subject to authorization and validation	High
FR-5	Provide attempt-aware live progress, validated resume progress, and cancellation for in-flight jobs	High
FR-6	Retry failed or lost attempts according to JobProtocol, using a new attempt identity and fencing token	High
FR-7	Maintain durable job and attempt history, including terminal outcomes and structured failure information	Medium
FR-8	Support soft deletion by tombstoning job metadata without deleting stored objects	Medium
FR-9	Support hard deletion by deleting stored objects while retaining a metadata tombstone	Medium
FR-10	Stream data without buffering complete objects in Worker memory	High
FR-11	Enforce independently configurable per-incarnation task limits and deployment-wide authoritative lease limits	Medium
FR-12	Isolate every attempt's target writes and expose at most one durably committed immutable result per job	High
FR-13	Make submission, dispatch, claiming, renewal, cancellation, progress, and finalization safe under duplicate or ambiguous requests	High
FR-14	Reconstruct operational coordination state from durable state after coordination loss or failover	High
FR-15	Enforce a configurable outbound-source policy for user-supplied locations and redirects	High

Non-Functional Requirements

Category	Requirement
Reliability	Preserve durable authority and committed results across worker or coordination failures; reconcile in-flight work by protocol
Observability	Expose job status, progress, and errors through the UI and API
Scalability	Support multiple stateless API and worker replicas using shared state services
Performance	Stream multi-gigabyte objects without buffering complete objects in worker memory
Security	Authenticate and authorize all operations; do not expose long-lived source or target credentials to the UI
Deployment	Deploy as a standard Kubernetes service using externally managed identity, state, and object storage

Responsibility Boundaries

• This architecture specification defines components, trust boundaries, externally visible behavior, and responsibility allocation.
• JobProtocol defines distributed state and coordination semantics, including safety and liveness invariants.
• JobProtocol is the normative implementation contract for durable transitions, fencing, leases, retries, cancellation, publication, and reconciliation. Implementations may optimize storage layout but must refine its observable behavior and preserve its linearization points.
• Implementation documentation selects technologies and maps architectural components onto them.
• Deployment configuration supplies service credentials and endpoints, configures allowed source reachability, and provides infrastructure-level egress enforcement.