257 lines
31 KiB
Markdown
257 lines
31 KiB
Markdown
# Remora Architecture Specification
|
||
|
||
**Remora** is a multi-user service that runs durable download jobs from remote sources into S3-compatible object storage.
|
||
|
||
## Glossary
|
||
|
||
Terms in this glossary are stable identifiers for use across Remora documentation and protocol models.
|
||
|
||
| Identifier | Definition |
|
||
| ---------- | ---------- |
|
||
| `Job` | A durable request to transfer one object or multi-object source snapshot from a `Source` to a `TargetStore`. |
|
||
| `Attempt` | A concrete execution corresponding to an attempt allocated by `JobProtocol`. |
|
||
| `WorkerIncarnation` | One lifetime of a `Worker` process. A restarted process is a new incarnation even when it occupies the same replica slot. |
|
||
| `Claim` | The worker-facing projection of `JobProtocol` attempt authority, containing its opaque attempt ID, worker incarnation, and fencing token. |
|
||
| `Lease` | The concrete coordination record representing a `JobProtocol` lease. |
|
||
| `ExecutionEnvelope` | The attempt-bound projection of durable inputs that a worker must validate before source or target I/O. |
|
||
| `ResolvedSource` | The immutable source identity or inventory pinned from an accepted source request. |
|
||
| `SourceCredential` | An owner-authorized registry record whose immutable secret versions are referenced by execution grants. |
|
||
| `StagingObject` | An immutable, attempt-specific result member or manifest object that has not been selected by durable publication. |
|
||
| `ResultManifest` | The immutable exact inventory and root identity for one attempt's result aggregate. |
|
||
| `CommittedResult` | The concrete durable descriptor corresponding to the attempt committed by `JobProtocol`. |
|
||
| `WebClient` | The browser-based user interface for submitting and managing jobs. |
|
||
| `API` | The authenticated service interface for job submission, status, cancellation, history, and deletion. |
|
||
| `Worker` | A stateless process that executes attempts and reports progress and outcomes. |
|
||
| `IdentityProvider` | The external authority for user identity, administrative roles, and storage authorization scopes. |
|
||
| `CoordinationStore` | Shared operational state for queued work, in-flight coordination, live progress, cancellation, and deployment-wide concurrency. |
|
||
| `DurableStore` | The source of truth for job identity, ownership, configuration, history, and terminal outcomes. |
|
||
| `Source` | An HTTP endpoint, compatible object store, or Hugging Face repository from which a job reads. |
|
||
| `StorageAdapter` | The service boundary through which workers write to the target object store. |
|
||
| `TargetStore` | The configured S3-compatible object store receiving downloaded objects. |
|
||
| `JobProtocol` | The TLA+ specification defining distributed job state, coordination semantics, and safety and liveness properties. |
|
||
|
||
## Scope
|
||
|
||
This document is authoritative for Remora's product requirements, system boundaries, major components, result representation and access contract, and allocation of responsibilities below the abstraction of `JobProtocol`. It intentionally does not define implementation-level technology choices.
|
||
|
||
[`Remora.tla`](../protocol/Remora.tla) is the authoritative `JobProtocol`. This document references its symbols without restating their definitions or actions; where they overlap, the model controls. Concrete frameworks, libraries, wire formats, mutable API descriptions, and deployment choices are outside the scope of this specification.
|
||
|
||
## Problem Statement
|
||
|
||
Pulling large objects from remote sources into S3-compatible storage is commonly done with one-off CLI tools or scripts. Those approaches are hard to monitor for long transfers, lack shared retry and resumption, offer little cross-user visibility, and do not enforce authorization over target placement.
|
||
|
||
## Proposed Solution
|
||
|
||
Remora exposes a browser UI and API for submitting, monitoring, and managing download jobs into a target object store. Supported source modes are HTTP URLs, compatible object stores, and Hugging Face repository snapshots.
|
||
|
||
Each download is an independent background job with streaming I/O and the retry, progress, and cancellation behavior defined by `JobProtocol`. A replacement attempt can reuse concrete checkpoint state only when its source and destination proofs remain valid. Jobs default to a configured target bucket and allow per-request bucket and prefix overrides subject to authorization and validation. The attempt committed by `JobProtocol` is exposed through one immutable `ResultManifest`.
|
||
|
||
Identity, coordination, and durable state are external services. Multiple API and worker replicas share them for authenticated multi-user operation.
|
||
|
||
## Architecture
|
||
|
||
`WebClient` communicates with `API`, which authorizes requests through `IdentityProvider`, records jobs in `DurableStore`, and submits work through `CoordinationStore`. `Worker` replicas consume jobs and stream data from `Source` through `StorageAdapter` into `TargetStore`.
|
||
|
||
`API` and `Worker` replicas are stateless. `DurableStore` realizes `durableVars` and retains architecture data outside the model. `CoordinationStore` realizes `coordinationVars` and may be reconstructed from durable state as permitted by `JobProtocol`.
|
||
|
||
`TargetStore` contains immutable attempt-scoped objects. The model's staging identity abstracts each concrete aggregate, and its committed attempt maps to the manifest descriptor exposed as the `CommittedResult`. `JobProtocol` is the sole specification of attempt authority and publication behavior.
|
||
|
||
Workers enforce a configurable outbound-source policy for user-supplied locations on every connection, including redirects. Deployments also restrict worker egress as a defense-in-depth boundary. Private or otherwise sensitive destinations are denied by default unless explicitly allowed by deployment policy.
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
subgraph Client
|
||
WebClient[WebClient]
|
||
end
|
||
|
||
subgraph Identity
|
||
IdentityProvider[IdentityProvider]
|
||
end
|
||
|
||
subgraph Service
|
||
API[API]
|
||
Worker[Worker]
|
||
end
|
||
|
||
subgraph State
|
||
CoordinationStore[CoordinationStore]
|
||
DurableStore[DurableStore]
|
||
end
|
||
|
||
subgraph Sources
|
||
Source[Source]
|
||
end
|
||
|
||
subgraph Target
|
||
StorageAdapter[StorageAdapter]
|
||
TargetStore[TargetStore]
|
||
end
|
||
|
||
WebClient -- OIDC --> IdentityProvider
|
||
WebClient -- API + live updates --> API
|
||
IdentityProvider -- Identity and roles --> API
|
||
API -- Submit and coordinate --> CoordinationStore
|
||
API -- Record and query --> DurableStore
|
||
Worker -- Consume and coordinate --> CoordinationStore
|
||
Worker -- Report durable outcomes --> DurableStore
|
||
Worker -- Stream from --> Source
|
||
Worker -- Stream to --> StorageAdapter
|
||
StorageAdapter --> TargetStore
|
||
```
|
||
|
||
## Worker Execution Envelope
|
||
|
||
- Every Worker process creates a fresh, non-reused `WorkerIncarnation`. A restarted process cannot reuse authority issued to an earlier lifetime, including when it occupies the same replica slot.
|
||
- `ExecutionEnvelope` supplies the concrete inputs abstracted by `ActivateAttempt`. Before concrete source or target I/O, a worker verifies that its job ID, opaque attempt ID, fencing token, and worker incarnation match the activated attempt projection.
|
||
- Authorization alone is not executable. The exact authority becomes usable only after coordination activation and the durable attempt state both identify it as executing.
|
||
- Activation and inactive-attempt recovery serialize against the same durable authority. Recovery revalidates coordination after entering that boundary and reaps an attempt that remains authorized; compensation removes any pre-commit lease. Activation removes any lease that cannot be paired with the durable executing tuple.
|
||
- Reservations have a bounded lifetime. Expiry makes unallocated work eligible for redispatch and makes a durably authorized attempt without its exact live lease eligible for reaping; it never reverses durable authority from coordination state alone.
|
||
- A lost successful activation response replays success only while the exact live lease and complete durable executing tuple still agree. A changed attempt, fence, or worker incarnation is a precondition failure.
|
||
- The envelope projects the accepted non-secret source metadata and a sensitive-input reference, any durable `ResolvedSource`, the immutable execution grant and concrete target, the attempt's exact staging identity, pinned credential-version references, and the validated checkpoint and destination-resume evidence available to that attempt. A replacement may inherit the predecessor's physical staging identity only through the durable checkpoint-ownership transfer defined below.
|
||
- A source identity or inventory that is initially unresolved is durably pinned once before concrete checkpoint validation or reuse relies on it. Pinning is a fenced durable mutation and a conflicting value is rejected.
|
||
- Large source inventories, completed-part sets, and other unbounded collections are represented by durable streamable descriptors rather than materialized in the envelope.
|
||
- Envelope transport is an implementation choice. It may accompany queue delivery or claim, or be read separately, but a worker never infers it from reconstructible coordination state.
|
||
- A missing envelope, an identity mismatch, an inaccessible credential reference, or invalid resume evidence fails closed. None grants permission to use a different source, target, staging identity, or checkpoint.
|
||
- Every resolved-source, checkpoint, progress, outcome, and publication mutation validates the complete attempt, fence, and Worker-incarnation tuple against current durable authority and the required attempt state.
|
||
- An adapter validates the activated envelope and exact live lease when it enters or restarts source or target I/O. Lease loss may leave harmless attempt-scoped staging writes in progress, as permitted by `JobProtocol`; it never authorizes a durable mutation or publication.
|
||
- Coordination projection is idempotent and driven from durable projection work. Missing coordination state never reverses authority into `DurableStore`.
|
||
|
||
## Result Aggregate and Direct Access
|
||
|
||
- The attempt selected by `committedAttempt` is represented by one immutable `ResultManifest`. A single-object transfer uses a one-member manifest.
|
||
- A manifest is the exact inventory of immutable member identities and carries sufficient integrity metadata to validate the aggregate.
|
||
- This specification defines the manifest's logical invariants, not its serialization or target-object layout. An implementation that exposes the manifest object directly to clients publishes a versioned format contract; an implementation may instead keep the format private and expose member enumeration and access through the service.
|
||
- Manifest member paths are NFC-normalized relative POSIX segment sequences. Empty segments, dot segments, traversal, leading or trailing separators, and backslashes are invalid rather than normalized into aliases.
|
||
- Manifest construction uses memory bounded independently of member count and total result size.
|
||
- Remora resolves the concrete aggregate only from the selected `CommittedResult`; it never derives one by listing a target bucket or prefix.
|
||
- An authorized reader may access committed members directly in `TargetStore` using exact manifest identities or time-limited URLs.
|
||
- A time-limited URL is bound to one exact object identity, including its version when present, and has bounded expiry.
|
||
- Raw target listing is operational storage visibility, not result resolution. Clients with independent target credentials may observe uncommitted attempt objects.
|
||
- A stable user-chosen object key or atomically visible raw target prefix is not part of a job result. Providing one requires a separate export contract.
|
||
- Remora writes attempt members and manifests under a deployment-reserved service namespace using globally unique, create-only identities. It independently computes and persists SHA-256 for every completed object.
|
||
- Exact object and multipart identities are durable data. Version IDs are retained when available. A non-versioned namespace is supported only when Remora is its exclusive writer; otherwise immutable version addressing is required.
|
||
|
||
## Target Cleanup
|
||
|
||
- Concrete work for `CleanupStage` is materialized from exact attempt ownership and pending cleanup data in `DurableStore`, independently of `CoordinationStore`.
|
||
- Ambient cleanup processes are stateless and idempotent. Worker-local cleanup may reduce latency but is not the only cleanup path.
|
||
- Cleanup addresses exact attempt object or multipart identities. Eligibility comes only from `CleanupStage`; missing coordination state, target-object age, and raw prefix membership are not substitutes.
|
||
- Every reusable checkpoint has one durable logical staging owner. Its exact retained objects, multipart state, inventory rows, and pending cleanup identities are ineligible for cleanup while the job is nonterminal.
|
||
- A replacement is authorized only after the checkpoint owner is durably stopped. Authorization atomically transfers logical ownership and all retained exact identities to the replacement before it can perform I/O. Physical keys may retain their original globally unique attempt namespace; key prefixes do not determine current logical ownership.
|
||
- State beyond the checkpoint's validated boundary remains owned by the predecessor and becomes independently cleanup-eligible after transfer. A claimed or completed cleanup identity cannot be transferred or used as resume evidence.
|
||
- Target lifecycle rules are storage hygiene only. They do not substitute for `JobProtocol` publication or cleanup eligibility and must not expire objects retained by a committed aggregate.
|
||
- Independent writers and lifecycle mechanisms must be unable to replace or expire live or committed identities in Remora's reserved namespace. A deployment that cannot establish this property is unsupported.
|
||
|
||
## Source Identity and Resumption
|
||
|
||
- `checkpointValid` abstracts the concrete proof defined here. A reusable checkpoint binds an immutable `Source` identity, an exact completed boundary, validated destination state, and its current durable staging owner; absent or changed proof makes the checkpoint unavailable to a replacement attempt.
|
||
- HTTP identity includes the resolved final URL, a strong representation validator as defined by `RFC 9110`, representation length, and identity content encoding. Resume requires a conditional byte-range response whose validator, URL, encoding, and range exactly match the checkpoint. Weak validators, length, or modification time alone do not qualify.
|
||
- Object-store identity includes endpoint, bucket, key, size, and an immutable version ID when available. Without a version ID, resume requires a matching conditional range read. Destination multipart reuse additionally requires the same endpoint, bucket, key, upload ID, and exact re-listed completed parts.
|
||
- An HTTP or object-store single-object source supplies one explicit `result_path` validated under the manifest member-path rules. The adapter does not derive the result member path from a URL or object key.
|
||
- A Hugging Face branch or tag is resolved once to a full commit. The source identity includes repository type and ID, that commit, and the exact canonical file inventory. Retries and cache reuse remain pinned to the stored commit and member identities.
|
||
- Hugging Face file entries become byte members and directory entries are structural. Unknown or unsupported entry representations fail closed. Symlinks and submodules are unsupported unless a later contract defines their byte and path semantics.
|
||
- Hugging Face selection is the union of exact repository-relative paths and complete path-segment prefixes. Omitted selection means the complete snapshot; a supplied selection that matches no members is invalid.
|
||
- Selected Hugging Face paths use the manifest member-path rules. Normalization collisions are invalid, duplicate selections collapse to one member, and canonical inventory order is normalized-path order.
|
||
- A Hugging Face member identity records its Git blob ID, LFS SHA-256, or commit-bound Xet hash as applicable. Downloaded content always receives an independently computed SHA-256; Xet metadata is bound to the resolved full commit and path rather than treated as a free-standing content digest.
|
||
- A checkpoint never authorizes best-effort concatenation. The adapter maps an unavailable explicitly pinned or resolved snapshot to `source_changed` rather than silently resolving a newer snapshot.
|
||
|
||
## Retry and Structured Failures
|
||
|
||
- Structured failures use the stable classes `validation`, `authorization`, `source_not_found`, `source_changed`, `transient_source`, `transient_target`, `integrity`, `capacity`, `internal`, `cancelled`, and `lost_attempt`.
|
||
- Attempt-reported runtime classes map to the permanent or retryable disposition supplied to `JobProtocol`: authorization, source-not-found, and source-changed are permanent; transient-source, transient-target, integrity, capacity, and internal are retryable. Unrecognized runtime errors map to a redacted internal failure.
|
||
- Validation rejects submission before the job enters `JobProtocol`. Cancellation and lost attempts are protocol inputs rather than worker-selected failure classes.
|
||
- Source adapters select connect, response-start, no-progress, bounded full-jitter, and bounded server-hint timing. Invalid or excessive server hints are ignored or capped before the adapter supplies an outcome to `JobProtocol`.
|
||
- Remora supplies no elapsed wall-clock deadline to `JobProtocol`. Per-operation timeouts map to structured runtime failures.
|
||
- Public failure details are constructed from per-class allowlists. Credentials, upstream bodies, raw URLs, and internal exception text are excluded.
|
||
|
||
## Result Visibility and Deletion
|
||
|
||
- Deletion is an architecture lifecycle outside `JobProtocol`. It may begin only from a terminal protocol projection; active work uses the protocol cancellation input instead.
|
||
- Soft deletion tombstones Remora visibility immediately without deleting target objects. It is idempotent and may be restored until hard deletion begins.
|
||
- Hard deletion atomically tombstones visibility and creates one durable, idempotent deletion operation. Once that operation exists, the result cannot be restored even while physical deletion is pending or retrying.
|
||
- A hard-deletion operation uses the `ResultManifest` and durable attempt ownership to materialize exact deletion work with bounded memory. It removes committed members and deletion-eligible attempt data, treats an already-missing exact identity as success, and deletes the manifest only after its members.
|
||
- Physical deletion is asynchronous and recoverable across worker loss and transient target failures. It does not depend on client request lifetime or reconstruct work by listing a target prefix.
|
||
- A deletion operation that cannot progress because required target access or configuration is unavailable retains its exact remaining work and a redacted administrative reason. It resumes from that work when the dependency is restored; it never abandons the deletion obligation or restores result visibility.
|
||
- A permanent minimal tombstone retains job identity, owner, concrete target authorization tuple, terminal outcome, deletion identity, and audit timing. Retention of more detailed job metadata is deployment policy and never restores result visibility after hard deletion begins.
|
||
|
||
## Authorization and Credentials
|
||
|
||
- Submission canonicalizes a concrete target and authorizes it against the caller's current scopes. The accepted job stores an immutable execution grant containing the owner subject, target-store identity, bucket, prefix segments, policy version, and opaque credential references. It stores neither raw roles nor access tokens.
|
||
- A bucket is 3–63 lowercase ASCII letters, digits, dots, or hyphens, begins and ends with a letter or digit, and contains no adjacent dots. Bucket and target-store equality are exact and case-sensitive.
|
||
- A target prefix is represented as NFC-normalized, case-sensitive Unicode segments separated by `/`. Empty prefix means the complete bucket, and one trailing separator is insignificant. Leading or doubled separators, dot segments, backslashes, Unicode category-C characters, and deployment-reserved internal segments are invalid.
|
||
- A storage scope contains one target-store identity, bucket, and prefix segment tuple. It covers a target only when store and bucket match exactly and its segments are a complete prefix of the target segments; textual prefix coincidence is insufficient.
|
||
- An owner may list, read, cancel, soft-delete, restore, or hard-delete a job only while current authorization still covers its concrete target. Administrators may manage all jobs and tombstones. List operations apply the same rule as a filter and never reveal unauthorized job existence.
|
||
- Owner access to a deletion operation is authorized against the concrete target retained in its job tombstone. Missing and unauthorized deletion operations are indistinguishable.
|
||
- Workers execute an accepted job from its immutable grant without consulting current identity roles. Failure to establish current authorization fails new submission, reads, and management closed but does not reinterpret accepted execution authority.
|
||
- An administrator may explicitly and idempotently revoke an accepted execution grant. Revocation maps to the protocol cancellation input when applicable and does not add an identity-provider lookup to workers.
|
||
- A source credential reference identifies an owner-authorized `SourceCredential`, never a caller-selected secret-store path. Credential metadata and authorization state are durable; secret material is held behind an opaque secret-store location and immutable version.
|
||
- Submission resolves the credential's current authorized version and pins that exact version in the immutable execution grant. Rotation affects future grants and never silently changes an accepted grant.
|
||
- The trusted credential resolver accepts only the credential and immutable version carried by an active execution grant. Substitution, execution-grant revocation, credential-version destruction, absence, or secret-store unavailability fails closed and never falls forward to a newer version.
|
||
- Credential revocation prevents new execution grants. Revocation of an accepted execution grant follows the administrative cancellation rule above; destruction of a still-referenced version makes resolution fail closed.
|
||
- Every secret version created by Remora is either referenced by durable credential metadata or owned by a durable open write intent. A terminal abandoned intent has no live secret.
|
||
- Secret values and complete sensitive source inputs never appear in durable public job configuration, public API projections, events, logs, or structured failures. The complete submitted HTTP URL is sensitive execution input; public projections expose only a sanitized display URL. Target credentials remain internal to `StorageAdapter`.
|
||
- Secret-store topology, policy granularity, workload authentication, availability, and recovery are deployment choices. They may narrow access further but do not replace Remora's grant-bound resolver.
|
||
|
||
## Outbound Source Policy
|
||
|
||
- The default HTTP-style source policy permits HTTPS on port 443. Deployments may add explicit scheme and port pairs; userinfo, fragments, malformed hosts, and non-canonical numeric IP forms are rejected.
|
||
- Every new connection parses and canonicalizes its URL, resolves its hostname once through the policy resolver, and validates the complete returned address set. If any address is denied, the connection is denied.
|
||
- Unspecified, loopback, private, shared, link-local, documentation, multicast, reserved, and other non-global addresses are denied by default for both IPv4 and IPv6, including IPv4-mapped IPv6. Deployments may explicitly allow required private networks, but cloud metadata endpoints remain denied.
|
||
- The transport connects to one approved resolved address without a second name resolution while retaining the canonical hostname for the HTTP authority, TLS SNI, and certificate verification. Connection-pool reuse must retain the same approved address and policy identity.
|
||
- Redirects are bounded to five by default. Every hop repeats URL, scheme/port, DNS, and address validation before dialing. HTTPS-to-HTTP downgrade is denied, and credentials and cookies are removed when the origin changes.
|
||
- Ambient proxy environment variables are ignored. An outbound proxy is disabled unless configured as a trusted enforcement boundary that provides the same hostname, resolution, address, redirect, and credential controls. Infrastructure egress policy remains defense in depth.
|
||
|
||
## API and Live Updates
|
||
|
||
- Mutable HTTP and WebSocket implementation contracts live under [`docs/api`](../api/). They are non-authoritative projections of `JobProtocol` and this document and must be corrected when they conflict with either source.
|
||
- Public mutation idempotency refines `RemoraApiMutation`: operation scope is the caller subject and operation, the request fingerprint is RFC 8785 canonical JSON hashed with SHA-256, and records are retained for at least the advertised 24-hour default window.
|
||
- Public job status projects `JobProtocol` state and adds deletion lifecycle, concrete progress, checkpoint, result, and failure fields.
|
||
- Job listing applies current authorization before stable keyset pagination. Cursors are opaque, integrity-protected, and bound to the subject, filters, and membership snapshot.
|
||
- HTTP snapshots carry a live-event cursor. WebSocket replay is ordered and at least once; invalid, expired, authorization-incompatible, or unavailable history requires a new HTTP snapshot. Event emission exposes progress only when it matches the protocol projection being published.
|
||
- Internal mutation requests add transport identifiers and an idempotent operation ID to the corresponding `JobProtocol` action; their eligibility and effects are not redefined here.
|
||
- Claim reserves coordination capacity and durably authorizes an attempt, but its response is non-executable until a separate activation operation refines `ActivateAttempt`.
|
||
- Activation, resolved-source pinning, and checkpoint persistence are distinct internal operations. Activation refines `ActivateAttempt`; resolved-source pinning is a fenced single-assignment stuttering refinement; checkpoint persistence refines `PersistCheckpoint`.
|
||
- Exact replay returns the stored operation result. A changed kind, payload, attempt, fence, or Worker incarnation cannot reuse an operation identity.
|
||
|
||
## Requirements
|
||
|
||
### Functional Requirements
|
||
|
||
| ID | Requirement | Priority |
|
||
| ---- | ------------------------------------------------------------------------------------------------------------------------------------- | -------- |
|
||
| FR-1 | Authenticate users through `IdentityProvider` and derive administrative and storage scopes from its roles | High |
|
||
| FR-2 | Enforce bucket and prefix scopes for non-administrative users | High |
|
||
| FR-3 | Support `Job` sources using HTTP URLs, compatible object stores, and Hugging Face repositories | High |
|
||
| FR-4 | Allow per-`Job` target bucket and prefix overrides subject to authorization and validation | High |
|
||
| FR-5 | Provide attempt-aware live progress, validated resume progress, and cancellation for in-flight jobs | High |
|
||
| FR-6 | Retry failed or lost work according to `JobProtocol` | High |
|
||
| FR-7 | Maintain durable job and attempt history, including terminal outcomes and structured failure information | Medium |
|
||
| FR-8 | Support soft deletion by tombstoning job metadata without deleting stored objects | Medium |
|
||
| FR-9 | Support hard deletion by deleting stored objects while retaining a metadata tombstone | Medium |
|
||
| FR-10 | Stream data and construct result inventories with memory bounded independently of complete object and aggregate size | High |
|
||
| FR-11 | Expose independent configuration for the per-incarnation and deployment-wide limits modeled by `JobProtocol` | Medium |
|
||
| FR-12 | Isolate attempt target writes and expose exactly one immutable `ResultManifest` for each successful job | High |
|
||
| FR-13 | Ensure distributed job operations refine `JobProtocol` under duplicate or ambiguous requests | High |
|
||
| FR-14 | Reconstruct operational coordination state after loss or failover as required by `JobProtocol` | High |
|
||
| FR-15 | Enforce a configurable outbound-source policy for user-supplied locations and redirects | High |
|
||
|
||
### Non-Functional Requirements
|
||
|
||
| Category | Requirement |
|
||
| ------------- | ----------------------------------------------------------------------------------------------------------- |
|
||
| Reliability | Refine `JobProtocol` under worker or coordination failures and preserve concrete result data outside the model |
|
||
| Observability | Expose job status, progress, and errors through the UI and API |
|
||
| Scalability | Support multiple stateless API and worker replicas using shared state services |
|
||
| Performance | Stream multi-gigabyte objects without buffering complete objects in worker memory |
|
||
| Security | Authenticate and authorize all operations; do not expose long-lived source or target credentials to the UI |
|
||
| Deployment | Deploy as a standard Kubernetes service using externally managed identity, state, and object storage |
|
||
|
||
## Responsibility Boundaries
|
||
|
||
- [`docs/protocol`](../protocol/) is authoritative for distributed state and coordination semantics, including safety and liveness invariants.
|
||
- This architecture specification is authoritative for components, trust boundaries, externally visible behavior, and responsibility allocation below the model's abstraction, and must conform to `JobProtocol` where they overlap.
|
||
- Implementations may optimize storage layout but must refine `JobProtocol` behavior and preserve its linearization points.
|
||
- Mutable implementation contracts and documentation select technologies, wire shapes, and mappings from architectural components. They are not normative over `JobProtocol` or this specification.
|
||
- Deployment configuration supplies service credentials and endpoints, configures allowed source reachability, and provides infrastructure-level egress enforcement.
|
||
- Deployment configuration reserves and protects Remora's internal target namespace, excludes it from independent lifecycle deletion, and establishes either exclusive write ownership or immutable version addressing.
|
||
- Deployment configuration selects secret-store topology and policies while preserving the application-level grant-bound credential-resolution contract.
|