remora/docs/spec/architecture.md

# Remora Architecture Specification

**Remora** is a multi-user service that runs durable download jobs from remote sources into S3-compatible object storage.

## Glossary

Terms in this glossary are stable identifiers for use across Remora documentation and protocol models.

| Identifier | Definition |
| ---------- | ---------- |
| `Job` | A durable request to transfer one object or multi-object source snapshot from a `Source` to a `TargetStore`. |
| `Attempt` | One execution of a `Job`. At-least-once delivery may produce more than one `Attempt` for a `Job`. |
| `WorkerIncarnation` | One lifetime of a `Worker` process. A restarted process is a new incarnation even when it occupies the same replica slot. |
| `Claim` | Durable authority for one `Attempt`, identified by its attempt ID, worker incarnation, and monotonically increasing fencing token. |
| `Lease` | Reconstructible coordination state granting a live attempt a deployment-wide concurrency permit for a bounded interval. |
| `StagingObject` | An immutable, attempt-specific target identity that is not externally visible as the result of a job. |
| `CommittedResult` | The immutable object identity selected by the authoritative durable commit for a successful job. |
| `WebClient` | The browser-based user interface for submitting and managing jobs. |
| `API` | The authenticated service interface for job submission, status, cancellation, history, and deletion. |
| `Worker` | A stateless process that executes attempts and reports progress and outcomes. |
| `IdentityProvider` | The external authority for user identity, administrative roles, and storage authorization scopes. |
| `CoordinationStore` | Shared operational state for queued work, in-flight coordination, live progress, cancellation, and deployment-wide concurrency. |
| `DurableStore` | The source of truth for job identity, ownership, configuration, history, and terminal outcomes. |
| `Source` | An HTTP endpoint, compatible object store, or Hugging Face repository from which a job reads. |
| `StorageAdapter` | The service boundary through which workers write to the target object store. |
| `TargetStore` | The configured S3-compatible object store receiving downloaded objects. |
| `JobProtocol` | The TLA+ specification defining distributed job state, coordination semantics, and safety and liveness properties. |

## Scope

This document defines Remora's product requirements, system boundaries, major components, and allocation of responsibilities. It intentionally does not define the distributed job protocol or implementation-level technology choices.

`JobProtocol` defines job state transitions, at-least-once delivery, claims and fencing, retry scheduling, cancellation races, recovery, and the associated safety and liveness properties. Concrete frameworks, libraries, and deployment choices are outside the scope of this specification.

## Problem Statement

Pulling large objects from remote sources into S3-compatible storage is commonly done with one-off CLI tools or scripts. Those approaches are hard to monitor for long transfers, lack shared retry and resumption, offer little cross-user visibility, and do not enforce authorization over target placement.

## Proposed Solution

Remora exposes a browser UI and API for submitting, monitoring, and managing download jobs into a target object store. Supported source modes are HTTP URLs, compatible object stores, and Hugging Face repository snapshots.

Each download is an independent background job with streaming I/O, automatic retry, live progress, and cancellation. A failed attempt resumes when source and destination support safe resumption; otherwise its replacement starts from the beginning. Jobs default to a configured target bucket and allow per-request bucket and prefix overrides subject to authorization and validation.

Identity, coordination, and durable state are external services. Multiple API and worker replicas share them for authenticated multi-user operation.

## Architecture

`WebClient` communicates with `API`, which authorizes requests through `IdentityProvider`, records jobs in `DurableStore`, and submits work through `CoordinationStore`. `Worker` replicas consume jobs and stream data from `Source` through `StorageAdapter` into `TargetStore`.

`API` and `Worker` replicas are stateless. `DurableStore` is authoritative for accepted jobs, cancellation intent, attempt allocation, fencing tokens, retry consumption, terminal outcomes, and the identity of a committed result. `CoordinationStore` contains reconstructible queue entries, leases, deployment-wide permits, cancellation notifications, and live progress. Loss of coordination state must not change durable authority or expose an uncommitted result.

At-least-once delivery may leave multiple attempt processes alive. Every attempt writes only to its own `StagingObject`. A successful attempt becomes externally visible only when a compare-and-swap transition in `DurableStore` selects its attempt ID, fencing token, and immutable object identity as the `CommittedResult`. A stale, expired, or cancelled attempt may finish writing staging data, but cannot publish it.

Workers enforce a configurable outbound-source policy for user-supplied locations on every connection, including redirects. Deployments also restrict worker egress as a defense-in-depth boundary. Private or otherwise sensitive destinations are denied by default unless explicitly allowed by deployment policy.

```mermaid
flowchart TD
    subgraph Client
        WebClient[WebClient]
    end

    subgraph Identity
        IdentityProvider[IdentityProvider]
    end

    subgraph Service
        API[API]
        Worker[Worker]
    end

    subgraph State
        CoordinationStore[CoordinationStore]
        DurableStore[DurableStore]
    end

    subgraph Sources
        Source[Source]
    end

    subgraph Target
        StorageAdapter[StorageAdapter]
        TargetStore[TargetStore]
    end

    WebClient -- OIDC --> IdentityProvider
    WebClient -- API + live updates --> API
    IdentityProvider -- Identity and roles --> API
    API -- Submit and coordinate --> CoordinationStore
    API -- Record and query --> DurableStore
    Worker -- Consume and coordinate --> CoordinationStore
    Worker -- Report durable outcomes --> DurableStore
    Worker -- Stream from --> Source
    Worker -- Stream to --> StorageAdapter
    StorageAdapter --> TargetStore
```

## Architectural Semantics

### Authority and Recovery

- `DurableStore` is the sole authority for job phase, cancellation intent, attempt count, current fencing token, current authoritative attempt, and terminal result.
- Every authority-changing request carries a job ID, attempt ID, worker incarnation, fencing token, and idempotency identity. Repeating a request has the same authoritative effect as processing it once.
- Submission creates a durable dispatch record before work is placed in `CoordinationStore`. A reconciler repeatedly projects eligible durable jobs, cancellation intent, and expired work into coordination state.
- `CoordinationStore` loss may discard queues, leases, permits, progress, and notifications. Those records are rebuilt from durable state; they never override a durable terminal outcome.
- A replacement attempt receives a new attempt ID and a strictly greater fencing token. Creating that authoritative replacement consumes retry budget, including when the previous attempt was lost without reporting an outcome.

### Publication and Cancellation

- Attempts never write directly to the authoritative result identity. Each attempt streams into an immutable attempt-specific staging or multipart identity.
- Publication linearizes at one durable compare-and-swap that verifies the current attempt ID and fencing token, verifies that cancellation has not won, and records the immutable committed object identity.
- Readers resolve only the `CommittedResult`. Uncommitted staging data is never presented as a job result and is deleted asynchronously after it can no longer be committed.
- `Cancelled` means that durable publication is permanently forbidden. It does not assert that every partitioned process has stopped or that staging cleanup has completed.
- Cancellation is propagated cooperatively to reachable workers. Lease loss or cancellation causes a connected worker to stop promptly, but publication safety does not depend on observing that stop.

### Progress and Resumption

- Live attempt progress is ephemeral and scoped to the current fencing token.
- Durable resume progress represents only a validated checkpoint, completed multipart range, or equivalent reusable state. A replacement may reuse it only when source identity and destination state validators still match.
- User-visible progress combines validated durable progress with current-attempt progress. If validation fails, reusable progress resets and the UI identifies that a new attempt restarted rather than presenting stale progress as completed work.
- Durable progress and attempt history may be coalesced; correctness must not depend on persisting every live progress event.

### Concurrency Semantics

- A per-incarnation local limiter bounds tasks executing in one live `Worker` process.
- The deployment-wide limit bounds live authoritative leases and reservations in `CoordinationStore`.
- A partitioned or paused attempt can temporarily continue physical I/O after its lease expires. Such zombie work is isolated in staging and does not count as an authoritative lease. A strict bound on all physical I/O during partitions requires infrastructure-level enforcement outside `JobProtocol`.

### Availability Assumptions

Safety requirements hold during worker failure, delayed requests, coordination loss, and temporary service partitions. Eventual completion is required only when durable storage and coordination recover, a worker incarnation remains available long enough to run an attempt, required source and target operations eventually return, lease time advances, and retry policy permits another attempt.

## Requirements

### Functional Requirements

| ID   | Requirement                                                                                                                           | Priority |
| ---- | ------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| FR-1 | Authenticate users through `IdentityProvider` and derive administrative and storage scopes from its roles                                | High     |
| FR-2 | Enforce bucket and prefix scopes for non-administrative users                                                                          | High     |
| FR-3 | Support `Job` sources using HTTP URLs, compatible object stores, and Hugging Face repositories                                            | High     |
| FR-4 | Allow per-`Job` target bucket and prefix overrides subject to authorization and validation                                               | High     |
| FR-5 | Provide attempt-aware live progress, validated resume progress, and cancellation for in-flight jobs                                      | High     |
| FR-6 | Retry failed or lost attempts according to `JobProtocol`, using a new attempt identity and fencing token                                 | High     |
| FR-7 | Maintain durable job and attempt history, including terminal outcomes and structured failure information                                 | Medium   |
| FR-8 | Support soft deletion by tombstoning job metadata without deleting stored objects                                                      | Medium   |
| FR-9 | Support hard deletion by deleting stored objects while retaining a metadata tombstone                                                  | Medium   |
| FR-10 | Stream data without buffering complete objects in `Worker` memory                                                                       | High     |
| FR-11 | Enforce independently configurable per-incarnation task limits and deployment-wide authoritative lease limits                            | Medium   |
| FR-12 | Isolate every attempt's target writes and expose at most one durably committed immutable result per job                                  | High     |
| FR-13 | Make submission, dispatch, claiming, renewal, cancellation, progress, and finalization safe under duplicate or ambiguous requests         | High     |
| FR-14 | Reconstruct operational coordination state from durable state after coordination loss or failover                                         | High     |
| FR-15 | Enforce a configurable outbound-source policy for user-supplied locations and redirects                                                   | High     |

### Non-Functional Requirements

| Category      | Requirement                                                                                                 |
| ------------- | ----------------------------------------------------------------------------------------------------------- |
| Reliability   | Preserve durable authority and committed results across worker or coordination failures; reconcile in-flight work by protocol |
| Observability | Expose job status, progress, and errors through the UI and API                                              |
| Scalability   | Support multiple stateless API and worker replicas using shared state services                             |
| Performance   | Stream multi-gigabyte objects without buffering complete objects in worker memory                          |
| Security      | Authenticate and authorize all operations; do not expose long-lived source or target credentials to the UI |
| Deployment    | Deploy as a standard Kubernetes service using externally managed identity, state, and object storage       |

## Responsibility Boundaries

- This architecture specification defines components, trust boundaries, externally visible behavior, and responsibility allocation.
- `JobProtocol` defines distributed state and coordination semantics, including safety and liveness invariants.
- `JobProtocol` is the normative implementation contract for durable transitions, fencing, leases, retries, cancellation, publication, and reconciliation. Implementations may optimize storage layout but must refine its observable behavior and preserve its linearization points.
- Implementation documentation selects technologies and maps architectural components onto them.
- Deployment configuration supplies service credentials and endpoints, configures allowed source reachability, and provides infrastructure-level egress enforcement.