remora/docs/spec/architecture.md

Remora Architecture Specification

Remora is a multi-user service that runs durable download jobs from remote sources into S3-compatible object storage.

Glossary

Terms in this glossary are stable identifiers for use across Remora documentation and protocol models.

Identifier	Definition
Job	A durable request to transfer one object or multi-object source snapshot from a Source to a TargetStore.
Attempt	One execution of a Job. At-least-once delivery may produce more than one Attempt for a Job.
WorkerIncarnation	One lifetime of a Worker process. A restarted process is a new incarnation even when it occupies the same replica slot.
Claim	Durable authority for one Attempt, identified by its attempt ID, worker incarnation, and monotonically increasing fencing token.
Lease	Reconstructible coordination state granting a live attempt a deployment-wide concurrency permit for a bounded interval.
StagingObject	An immutable, attempt-specific result member or manifest object that has not been selected by durable publication.
ResultManifest	The immutable exact inventory and root identity for one attempt's result aggregate.
CommittedResult	The durable descriptor that selects one ResultManifest for a successful job.
WebClient	The browser-based user interface for submitting and managing jobs.
API	The authenticated service interface for job submission, status, cancellation, history, and deletion.
Worker	A stateless process that executes attempts and reports progress and outcomes.
IdentityProvider	The external authority for user identity, administrative roles, and storage authorization scopes.
CoordinationStore	Shared operational state for queued work, in-flight coordination, live progress, cancellation, and deployment-wide concurrency.
DurableStore	The source of truth for job identity, ownership, configuration, history, and terminal outcomes.
Source	An HTTP endpoint, compatible object store, or Hugging Face repository from which a job reads.
StorageAdapter	The service boundary through which workers write to the target object store.
TargetStore	The configured S3-compatible object store receiving downloaded objects.
JobProtocol	The TLA+ specification defining distributed job state, coordination semantics, and safety and liveness properties.

Scope

This document defines Remora's product requirements, system boundaries, major components, result representation and access contract, and allocation of responsibilities. It intentionally does not define the distributed job protocol or implementation-level technology choices.

JobProtocol defines job state transitions, at-least-once delivery, claims and fencing, retry scheduling, cancellation races, recovery, and the associated safety and liveness properties. Concrete frameworks, libraries, wire formats, and deployment choices are outside the scope of this specification.

Problem Statement

Pulling large objects from remote sources into S3-compatible storage is commonly done with one-off CLI tools or scripts. Those approaches are hard to monitor for long transfers, lack shared retry and resumption, offer little cross-user visibility, and do not enforce authorization over target placement.

Proposed Solution

Remora exposes a browser UI and API for submitting, monitoring, and managing download jobs into a target object store. Supported source modes are HTTP URLs, compatible object stores, and Hugging Face repository snapshots.

Each download is an independent background job with streaming I/O, automatic retry, live progress, and cancellation. A failed attempt resumes when source and destination support safe resumption; otherwise its replacement starts from the beginning. Jobs default to a configured target bucket and allow per-request bucket and prefix overrides subject to authorization and validation. Every successful job publishes one immutable ResultManifest, including single-object jobs.

Identity, coordination, and durable state are external services. Multiple API and worker replicas share them for authenticated multi-user operation.

Architecture

WebClient communicates with API, which authorizes requests through IdentityProvider, records jobs in DurableStore, and submits work through CoordinationStore. Worker replicas consume jobs and stream data from Source through StorageAdapter into TargetStore.

API and Worker replicas are stateless. DurableStore contains authoritative job records and committed result descriptors. CoordinationStore contains reconstructible operational state such as queue entries, leases, deployment-wide permits, cancellation notifications, and live progress.

TargetStore contains immutable attempt-scoped objects. A ResultManifest roots each attempt-scoped aggregate; DurableStore contains the manifest descriptor exposed as the CommittedResult. JobProtocol is the sole specification of attempt authority and publication behavior.

Workers enforce a configurable outbound-source policy for user-supplied locations on every connection, including redirects. Deployments also restrict worker egress as a defense-in-depth boundary. Private or otherwise sensitive destinations are denied by default unless explicitly allowed by deployment policy.

flowchart TD
    subgraph Client
        WebClient[WebClient]
    end

    subgraph Identity
        IdentityProvider[IdentityProvider]
    end

    subgraph Service
        API[API]
        Worker[Worker]
    end

    subgraph State
        CoordinationStore[CoordinationStore]
        DurableStore[DurableStore]
    end

    subgraph Sources
        Source[Source]
    end

    subgraph Target
        StorageAdapter[StorageAdapter]
        TargetStore[TargetStore]
    end

    WebClient -- OIDC --> IdentityProvider
    WebClient -- API + live updates --> API
    IdentityProvider -- Identity and roles --> API
    API -- Submit and coordinate --> CoordinationStore
    API -- Record and query --> DurableStore
    Worker -- Consume and coordinate --> CoordinationStore
    Worker -- Report durable outcomes --> DurableStore
    Worker -- Stream from --> Source
    Worker -- Stream to --> StorageAdapter
    StorageAdapter --> TargetStore

Result Aggregate and Direct Access

• Every successful job is represented by one immutable ResultManifest. A single-object transfer uses a one-member manifest.
• A manifest is the exact inventory of immutable member identities and carries sufficient integrity metadata to validate the aggregate.
• Manifest member paths are NFC-normalized relative POSIX segment sequences. Empty segments, dot segments, traversal, leading or trailing separators, and backslashes are invalid rather than normalized into aliases.
• Manifest construction uses memory bounded independently of member count and total result size.
• Remora resolves a result only from its CommittedResult; it never derives one by listing a target bucket or prefix.
• An authorized reader may access committed members directly in TargetStore using exact manifest identities or time-limited URLs.
• Raw target listing is operational storage visibility, not result resolution. Clients with independent target credentials may observe uncommitted attempt objects.
• A stable user-chosen object key or atomically visible raw target prefix is not part of a job result. Providing one requires a separate export contract.

Target Cleanup

• DurableStore retains exact attempt ownership and pending cleanup work so cleanup can recover independently of CoordinationStore.
• Ambient cleanup processes are stateless and idempotent. Worker-local cleanup may reduce latency but is not the only cleanup path.
• Cleanup revalidates authority in DurableStore and addresses exact attempt object or multipart identities. Missing coordination state, target-object age, and raw prefix membership do not grant cleanup authority.
• Target lifecycle rules are storage hygiene only. They do not determine result publication or cleanup eligibility and must not expire completed attempt objects that Remora may still protect.

Source Identity and Resumption

• A reusable checkpoint binds an immutable Source identity, an exact completed boundary, and validated destination state. If any proof is absent or changed, the replacement restarts that member from the beginning.
• HTTP identity includes the resolved final URL, a strong representation validator as defined by RFC 9110, representation length, and identity content encoding. Resume requires a conditional byte-range response whose validator, URL, encoding, and range exactly match the checkpoint. Weak validators, length, or modification time alone do not qualify.
• Object-store identity includes endpoint, bucket, key, size, and an immutable version ID when available. Without a version ID, resume requires a matching conditional range read. Destination multipart reuse additionally requires the same endpoint, bucket, key, upload ID, and exact re-listed completed parts.
• A Hugging Face branch or tag is resolved once to a full commit. The source identity includes repository type and ID, that commit, and the exact file inventory with Git blob or LFS identities. Retries and cache reuse remain pinned to the stored commit and member identities.
• A checkpoint never authorizes best-effort concatenation. An unavailable explicitly pinned or resolved snapshot produces a structured source-changed outcome rather than silently resolving a newer snapshot.

Retry and Structured Failures

• Structured failures use the stable classes validation, authorization, source_not_found, source_changed, transient_source, transient_target, integrity, capacity, internal, cancelled, and lost_attempt.
• Attempt-reported runtime failures carry a durable permanent or retryable disposition into JobProtocol. The model is the sole definition of how that disposition affects execution phase and attempt budget.
• Authorization, source-not-found, and source-changed runtime failures are permanent. Transient-source, transient-target, integrity, capacity, and internal failures are retryable. Unrecognized runtime errors map to a redacted internal failure.
• Validation normally rejects submission before a job exists. Cancellation and lost attempts follow their dedicated JobProtocol behavior rather than a worker-selected failure disposition.
• All retryable work uses the model-wide attempt bound. Source adapters may vary connect, response-start, no-progress, bounded full-jitter, and bounded server-hint timing without changing that bound. Invalid or excessive server hints are ignored or capped; they never change a retryable disposition into a permanent failure.
• Remora has no elapsed wall-clock job deadline. Per-operation timeouts produce a structured runtime failure and remain subject to JobProtocol.
• Public failure details are constructed from per-class allowlists. Credentials, upstream bodies, raw URLs, and internal exception text are excluded.

Result Visibility and Deletion

• JobProtocol remains the sole owner of execution phase. Deletion never rewrites a terminal execution outcome and is available only after a job is terminal; cancelling active work remains a separate protocol operation.
• Soft deletion tombstones Remora visibility immediately without deleting target objects. It is idempotent and may be restored until hard deletion begins.
• Hard deletion atomically tombstones visibility and creates one durable, idempotent deletion operation. Once that operation exists, the result cannot be restored even while physical deletion is pending or retrying.
• A hard-deletion operation uses the ResultManifest and durable attempt ownership to materialize exact deletion work with bounded memory. It removes committed members and eligible attempt data, treats an already-missing exact identity as success, and deletes the manifest only after its members.
• Physical deletion is asynchronous and recoverable across worker loss and transient target failures. It does not depend on client request lifetime or reconstruct work by listing a target prefix.
• A permanent minimal tombstone retains job identity, owner, terminal outcome, deletion identity, and audit timing. Retention of more detailed job metadata is deployment policy and never restores result visibility after hard deletion begins.

Authorization and Credentials

• Submission canonicalizes a concrete target and authorizes it against the caller's current scopes. The accepted job stores an immutable execution grant containing the owner subject, target-store identity, bucket, prefix segments, policy version, and opaque credential references. It stores neither raw roles nor access tokens.
• A bucket is 3–63 lowercase ASCII letters, digits, dots, or hyphens, begins and ends with a letter or digit, and contains no adjacent dots. Bucket and target-store equality are exact and case-sensitive.
• A target prefix is represented as NFC-normalized, case-sensitive Unicode segments separated by /. Empty prefix means the complete bucket, and one trailing separator is insignificant. Leading or doubled separators, dot segments, backslashes, Unicode category-C characters, and deployment-reserved internal segments are invalid.
• A storage scope contains one target-store identity, bucket, and prefix segment tuple. It covers a target only when store and bucket match exactly and its segments are a complete prefix of the target segments; textual prefix coincidence is insufficient.
• An owner may list, read, cancel, soft-delete, restore, or hard-delete a job only while current authorization still covers its concrete target. Administrators may manage all jobs and tombstones. List operations apply the same rule as a filter and never reveal unauthorized job existence.
• Workers execute an accepted job from its immutable grant without consulting current identity roles. Failure to establish current authorization fails new submission, reads, and management closed but does not reinterpret accepted execution authority.
• An administrator may explicitly and idempotently revoke an accepted execution grant. Revocation records durable cancellation intent through JobProtocol; it does not add an identity-provider lookup to workers or rewrite a terminal outcome.
• Source credentials are server-side opaque references resolved only by the worker component that needs them. Secret values never appear in job configuration, API data, events, logs, or structured failures. Target credentials remain internal to StorageAdapter.

Outbound Source Policy

• The default HTTP-style source policy permits HTTPS on port 443. Deployments may add explicit scheme and port pairs; userinfo, fragments, malformed hosts, and non-canonical numeric IP forms are rejected.
• Every new connection parses and canonicalizes its URL, resolves its hostname once through the policy resolver, and validates the complete returned address set. If any address is denied, the connection is denied.
• Unspecified, loopback, private, shared, link-local, documentation, multicast, reserved, and other non-global addresses are denied by default for both IPv4 and IPv6, including IPv4-mapped IPv6. Deployments may explicitly allow required private networks, but cloud metadata endpoints remain denied.
• The transport connects to one approved resolved address without a second name resolution while retaining the canonical hostname for the HTTP authority, TLS SNI, and certificate verification. Connection-pool reuse must retain the same approved address and policy identity.
• Redirects are bounded to five by default. Every hop repeats URL, scheme/port, DNS, and address validation before dialing. HTTPS-to-HTTP downgrade is denied, and credentials and cookies are removed when the origin changes.
• Ambient proxy environment variables are ignored. An outbound proxy is disabled unless configured as a trusted enforcement boundary that provides the same hostname, resolution, address, redirect, and credential controls. Infrastructure egress policy remains defense in depth.

API and Live Updates

• api/openapi.yaml is the normative HTTP contract and api/events.schema.json is the normative WebSocket frame contract. Their compatibility and replay semantics are defined in api/README.md.
• Every public mutation uses a caller- and operation-scoped idempotency key. The server atomically retains the canonical request fingerprint and original response for at least the advertised 24-hour default window; changed payload reuse is a conflict.
• Public job status projects the JobProtocol phase without redefining its transitions and separately exposes deletion lifecycle, current authoritative attempt, matching progress, validated checkpoint, committed result, and structured failure.
• Job listing applies current authorization before stable keyset pagination. Cursors are opaque, integrity-protected, and bound to the subject, filters, and membership snapshot.
• HTTP snapshots carry a live-event cursor. WebSocket replay is ordered and at least once; invalid, expired, authorization-incompatible, or unavailable history requires a new HTTP snapshot. Stale-attempt progress is suppressed before emission.
• Internal claim identifies the job, worker incarnation, and idempotent operation because no attempt exists yet. Renew, progress, and outcome additionally require the opaque attempt ID and fence defined by JobProtocol; stale authority has no effect.

Requirements

Functional Requirements

ID	Requirement	Priority
FR-1	Authenticate users through IdentityProvider and derive administrative and storage scopes from its roles	High
FR-2	Enforce bucket and prefix scopes for non-administrative users	High
FR-3	Support Job sources using HTTP URLs, compatible object stores, and Hugging Face repositories	High
FR-4	Allow per-Job target bucket and prefix overrides subject to authorization and validation	High
FR-5	Provide attempt-aware live progress, validated resume progress, and cancellation for in-flight jobs	High
FR-6	Retry failed or lost work according to JobProtocol	High
FR-7	Maintain durable job and attempt history, including terminal outcomes and structured failure information	Medium
FR-8	Support soft deletion by tombstoning job metadata without deleting stored objects	Medium
FR-9	Support hard deletion by deleting stored objects while retaining a metadata tombstone	Medium
FR-10	Stream data and construct result inventories with memory bounded independently of complete object and aggregate size	High
FR-11	Enforce independently configurable per-incarnation task limits and deployment-wide authoritative lease limits	Medium
FR-12	Isolate attempt target writes and expose exactly one immutable ResultManifest for each successful job	High
FR-13	Ensure distributed job operations refine JobProtocol under duplicate or ambiguous requests	High
FR-14	Reconstruct operational coordination state after loss or failover as required by JobProtocol	High
FR-15	Enforce a configurable outbound-source policy for user-supplied locations and redirects	High

Non-Functional Requirements

Category	Requirement
Reliability	Preserve durable authority and committed results across worker or coordination failures; reconcile in-flight work by protocol
Observability	Expose job status, progress, and errors through the UI and API
Scalability	Support multiple stateless API and worker replicas using shared state services
Performance	Stream multi-gigabyte objects without buffering complete objects in worker memory
Security	Authenticate and authorize all operations; do not expose long-lived source or target credentials to the UI
Deployment	Deploy as a standard Kubernetes service using externally managed identity, state, and object storage

Responsibility Boundaries

• This architecture specification defines components, trust boundaries, externally visible behavior, and responsibility allocation.
• JobProtocol defines distributed state and coordination semantics, including safety and liveness invariants.
• JobProtocol is the normative implementation contract for durable transitions, fencing, leases, retries, cancellation, publication, and reconciliation. Implementations may optimize storage layout but must refine its observable behavior and preserve its linearization points.
• Implementation documentation selects technologies and maps architectural components onto them.
• Deployment configuration supplies service credentials and endpoints, configures allowed source reachability, and provides infrastructure-level egress enforcement.