ms2k/remora
SHA256
Watch
0
Code Releases Activity

Separate protocol authority from mutable API contracts

Browse source 
Move API artifacts out of docs/spec, define concrete execution and deletion refinements, and document TLA+ abstraction boundaries so architecture prose cannot become a competing protocol specification.
This commit is contained in:
ms2k 2026-07-14 11:50:10 +09:00
commit dfd60cf104
6 changed files with 316 additions and 73 deletions
Download patch file Download diff file Expand all files Collapse all files

117
docs/api/README.md Normal file
View file

@ -0,0 +1,117 @@
# Remora API implementation contract
These files are mutable implementation artifacts. They describe the current
HTTP operations and WebSocket frames but are not authoritative over
[`Remora.tla`](../protocol/Remora.tla) or
[`architecture.md`](../spec/architecture.md). Conflicts are corrected here.
OpenAPI `3.2.0` describes HTTP operations and JSON Schema Draft 2020-12
describes WebSocket frames. The API contract version is `1.0.0`; additive
fields require clients to tolerate unknown response fields only after a
contract version that permits them, while incompatible changes use a new major
path or media type.
## Idempotency
Every public mutation requires `Idempotency-Key`. The identity is caller
subject, operation, and key. The server canonicalizes the semantic request,
including the authorization target, with RFC 8785 and stores its SHA-256
fingerprint and complete response atomically with the mutation. The default
retention is 24 hours and is never shorter than the advertised
`Idempotency-Expires-At`.
The same identity and fingerprint replays the original status, resource, and
body after duplicate or ambiguous delivery. A different fingerprint returns
`idempotency_conflict` and cannot affect either request. Naturally idempotent
deletion and cancellation semantics still use this request identity.
Internal claim uses job ID plus `operation_id` because no attempt exists yet.
Renew, progress, and outcome use attempt ID, fencing token, worker incarnation,
and `operation_id`. Each operation identity retains a request fingerprint. An
exact replay has the original effect; a changed worker, fence, or payload
returns `precondition_failed` without mutation. First observation is accepted
only when the corresponding `JobProtocol` action is enabled. Later eligibility
changes do not alter a stored response, while a fresh operation is evaluated
against current eligibility.
Renewal follows `RenewLease`: it may extend an existing live coordination lease
after durable authority changes. It never creates or restores a claim. Progress
and outcome operations require current fenced authority and return
`precondition_failed` for stale or non-authoritative tuples.
A successful claim returns the attempt authority and the attempt's
`ExecutionEnvelope`. The worker verifies that their job, attempt, fence, and
worker incarnation agree before source or target I/O. Opaque resolved-source
and checkpoint references identify durable streamable state; they are not
reconstructed from coordination data.
[`RemoraApiMutation`](../protocol/RemoraApiMutation.tla) models this durable
operation ledger around an abstract successful `JobProtocol` effect. It does
not redefine job phases or attempt-authority transitions. Its `Rejected`
response abstracts the endpoint-specific `idempotency_conflict` and
`precondition_failed` codes enforced by the executable contract.
## Authorization and errors
Public operations apply the authorization rules in the architecture. List is
filtered before pagination. A missing and an unauthorized resource both return
`not_found`; callers cannot probe job existence. Cursors and event replay are
bound to the authenticated subject and filter.
`ApiError` fields are an allowlist. Messages and field diagnostics never carry
credential values, bearer material, URL userinfo, raw upstream response bodies,
internal stack traces, or unredacted configured endpoints.
## Jobs, pages, and results
`Job.phase` uses the exact `JobProtocol` phase vocabulary; this contract does
not define phase transitions. `lifecycle`, current authoritative attempt,
current matching progress, reusable-checkpoint projection, result, and failure
are separate fields. Progress is present only when its attempt ID and fence
match the authoritative attempt in the same projection.
Job pages order by `(created_at DESC, id DESC)`. The first request fixes a
membership snapshot, authorization/filter fingerprint, and high-water mark.
`next_cursor` is an opaque integrity-protected keyset cursor for that snapshot;
new jobs do not enter later pages. Invalid, altered, cross-subject, or
cross-filter cursors return the corresponding structured cursor error.
A result endpoint returns only a visible `CommittedResult`. Tombstoned,
unauthorized, absent, and never-committed results are not exposed. The hard
delete endpoint returns the durable deletion resource established by the
architecture.
A deletion whose exact remaining work cannot progress because target access or
configuration is unavailable enters `blocked` with a redacted problem. It
retains the deletion obligation and can return to active processing when the
dependency is restored. Owner access remains authorized through the concrete
target retained in the tombstone.
Worker failure outcomes accept only attempt-reportable runtime classes and do
not accept a caller-supplied retryable flag. The service derives the durable
disposition from the class mapping in the architecture. Validation,
cancellation, and lost-attempt projection use their dedicated boundaries.
## WebSocket replay
An HTTP job or list snapshot carries `event_cursor` at its consistency
watermark. The client connects to `/v1/events` with that cursor; there is no
subscription frame. Server frames are ordered by increasing `sequence` for a
connection and carry an opaque next cursor. Delivery is at least once, and
clients deduplicate event IDs.
`job_changed` is an invalidation with an attempt-aware display hint, not a
resource replacement or patch. A client compares `job_revision` with the last
HTTP snapshot revision and fetches a new authorized snapshot when it is newer.
It advances its applied revision only from that snapshot. Matching progress may
be displayed provisionally while the fetch is pending.
`deletion_changed` is likewise an invalidation. Its state is a display hint;
the deletion resource remains the authoritative HTTP projection.
Replay applies current authorization before delivery. If a cursor is absent,
invalid, expired, authorization-incompatible, or no longer available after
coordination loss, the server sends `resync_required`; the client obtains a new
HTTP snapshot and reconnects from its cursor. Progress frames for a stale
attempt/fence are suppressed before emission and ignored defensively by the
client.

138
docs/api/events.schema.json Normal file
View file

@ -0,0 +1,138 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://remora.invalid/api/events.schema.json",
"title": "Remora WebSocket frame",
"oneOf": [
{"$ref": "#/$defs/JobChanged"},
{"$ref": "#/$defs/DeletionChanged"},
{"$ref": "#/$defs/ResyncRequired"},
{"$ref": "#/$defs/Heartbeat"}
],
"$defs": {
"OpaqueId": {
"type": "string",
"pattern": "^[A-Za-z0-9][A-Za-z0-9_-]{7,127}$"
},
"Cursor": {
"type": "string",
"minLength": 16,
"maxLength": 2048
},
"Attempt": {
"type": "object",
"additionalProperties": false,
"required": ["attempt_id", "fence"],
"properties": {
"attempt_id": {"$ref": "#/$defs/OpaqueId"},
"fence": {"type": "integer", "minimum": 1}
}
},
"Progress": {
"type": "object",
"additionalProperties": false,
"required": ["attempt_id", "fence", "completed_bytes", "completed_members"],
"properties": {
"attempt_id": {"$ref": "#/$defs/OpaqueId"},
"fence": {"type": "integer", "minimum": 1},
"completed_bytes": {"type": "integer", "minimum": 0},
"total_bytes": {"type": ["integer", "null"], "minimum": 0},
"completed_members": {"type": "integer", "minimum": 0},
"total_members": {"type": ["integer", "null"], "minimum": 0}
}
},
"ServerBase": {
"type": "object",
"required": ["schema_version", "kind", "event_id", "sequence", "cursor", "occurred_at"],
"properties": {
"schema_version": {"const": 1},
"kind": {"type": "string"},
"event_id": {"$ref": "#/$defs/OpaqueId"},
"sequence": {"type": "integer", "minimum": 1},
"cursor": {"$ref": "#/$defs/Cursor"},
"occurred_at": {"type": "string", "format": "date-time"}
}
},
"JobChanged": {
"allOf": [
{"$ref": "#/$defs/ServerBase"},
{
"type": "object",
"additionalProperties": false,
"required": ["schema_version", "kind", "event_id", "sequence", "cursor", "occurred_at", "job_id", "job_revision", "phase", "lifecycle", "authoritative_attempt", "progress"],
"properties": {
"schema_version": {"const": 1},
"kind": {"const": "job_changed"},
"event_id": {"$ref": "#/$defs/OpaqueId"},
"sequence": {"type": "integer", "minimum": 1},
"cursor": {"$ref": "#/$defs/Cursor"},
"occurred_at": {"type": "string", "format": "date-time"},
"job_id": {"$ref": "#/$defs/OpaqueId"},
"job_revision": {"type": "integer", "minimum": 1},
"phase": {"enum": ["Queued", "Running", "RetryWait", "Cancelling", "Succeeded", "Failed", "Cancelled"]},
"lifecycle": {"enum": ["visible", "soft_deleted", "hard_deleting", "hard_deleted"]},
"authoritative_attempt": {"oneOf": [{"$ref": "#/$defs/Attempt"}, {"type": "null"}]},
"progress": {"oneOf": [{"$ref": "#/$defs/Progress"}, {"type": "null"}]}
}
}
]
},
"DeletionChanged": {
"allOf": [
{"$ref": "#/$defs/ServerBase"},
{
"type": "object",
"additionalProperties": false,
"required": ["schema_version", "kind", "event_id", "sequence", "cursor", "occurred_at", "job_id", "operation_id", "state"],
"properties": {
"schema_version": {"const": 1},
"kind": {"const": "deletion_changed"},
"event_id": {"$ref": "#/$defs/OpaqueId"},
"sequence": {"type": "integer", "minimum": 1},
"cursor": {"$ref": "#/$defs/Cursor"},
"occurred_at": {"type": "string", "format": "date-time"},
"job_id": {"$ref": "#/$defs/OpaqueId"},
"operation_id": {"$ref": "#/$defs/OpaqueId"},
"state": {"enum": ["queued", "running", "retry_wait", "blocked", "completed"]}
}
}
]
},
"ResyncRequired": {
"allOf": [
{"$ref": "#/$defs/ServerBase"},
{
"type": "object",
"additionalProperties": false,
"required": ["schema_version", "kind", "event_id", "sequence", "cursor", "occurred_at", "reason"],
"properties": {
"schema_version": {"const": 1},
"kind": {"const": "resync_required"},
"event_id": {"$ref": "#/$defs/OpaqueId"},
"sequence": {"type": "integer", "minimum": 1},
"cursor": {"$ref": "#/$defs/Cursor"},
"occurred_at": {"type": "string", "format": "date-time"},
"reason": {"enum": ["cursor_absent", "cursor_invalid", "cursor_expired", "authorization_changed", "history_unavailable"]}
}
}
]
},
"Heartbeat": {
"allOf": [
{"$ref": "#/$defs/ServerBase"},
{
"type": "object",
"additionalProperties": false,
"required": ["schema_version", "kind", "event_id", "sequence", "cursor", "occurred_at"],
"properties": {
"schema_version": {"const": 1},
"kind": {"const": "heartbeat"},
"event_id": {"$ref": "#/$defs/OpaqueId"},
"sequence": {"type": "integer", "minimum": 1},
"cursor": {"$ref": "#/$defs/Cursor"},
"occurred_at": {"type": "string", "format": "date-time"}
}
}
]
}
}
}

964
docs/api/openapi.yaml Normal file
View file

@ -0,0 +1,964 @@
openapi: 3.2.0
$self: https://remora.invalid/api/openapi.yaml
jsonSchemaDialect: https://spec.openapis.org/oas/3.2/dialect/2025-09-17
info:
title: Remora API
version: 1.0.0
description: Mutable implementation contract for public job management and service-authenticated worker operations.
servers:
- url: /api
security:
- oidc: []
paths:
/v1/jobs:
post:
operationId: submitJob
parameters:
- $ref: '#/components/parameters/IdempotencyKey'
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/JobSubmission'
responses:
'202':
description: Accepted, or the original accepted response for the same key and fingerprint.
headers:
Idempotency-Expires-At:
$ref: '#/components/headers/IdempotencyExpiresAt'
content:
application/json:
schema:
$ref: '#/components/schemas/Job'
'409':
$ref: '#/components/responses/IdempotencyConflict'
default:
$ref: '#/components/responses/Error'
get:
operationId: listJobs
parameters:
- $ref: '#/components/parameters/Cursor'
- $ref: '#/components/parameters/Limit'
- name: include_deleted
in: query
schema:
type: boolean
default: false
responses:
'200':
description: Authorized keyset page bound to one snapshot and filter.
content:
application/json:
schema:
$ref: '#/components/schemas/JobPage'
default:
$ref: '#/components/responses/Error'
/v1/jobs/{job_id}:
parameters:
- $ref: '#/components/parameters/JobId'
get:
operationId: getJob
responses:
'200':
description: Current authorized job projection.
content:
application/json:
schema:
$ref: '#/components/schemas/Job'
'404':
$ref: '#/components/responses/NotFound'
default:
$ref: '#/components/responses/Error'
/v1/jobs/{job_id}/result:
parameters:
- $ref: '#/components/parameters/JobId'
get:
operationId: getJobResult
responses:
'200':
description: Visible committed manifest-rooted result.
content:
application/json:
schema:
$ref: '#/components/schemas/ResultDescriptor'
'404':
$ref: '#/components/responses/NotFound'
'409':
$ref: '#/components/responses/PreconditionFailed'
default:
$ref: '#/components/responses/Error'
/v1/jobs/{job_id}/cancel:
parameters:
- $ref: '#/components/parameters/JobId'
post:
operationId: cancelJob
parameters:
- $ref: '#/components/parameters/IdempotencyKey'
responses:
'202':
description: Cancellation intent accepted or prior response replayed.
headers:
Idempotency-Expires-At:
$ref: '#/components/headers/IdempotencyExpiresAt'
content:
application/json:
schema:
$ref: '#/components/schemas/Job'
default:
$ref: '#/components/responses/Error'
/v1/jobs/{job_id}/soft-delete:
parameters:
- $ref: '#/components/parameters/JobId'
post:
operationId: softDeleteJob
parameters:
- $ref: '#/components/parameters/IdempotencyKey'
responses:
'200':
description: Job hidden, or prior response replayed.
headers:
Idempotency-Expires-At:
$ref: '#/components/headers/IdempotencyExpiresAt'
content:
application/json:
schema:
$ref: '#/components/schemas/Job'
default:
$ref: '#/components/responses/Error'
/v1/jobs/{job_id}/restore:
parameters:
- $ref: '#/components/parameters/JobId'
post:
operationId: restoreJob
parameters:
- $ref: '#/components/parameters/IdempotencyKey'
responses:
'200':
description: Soft-deleted job restored, or prior response replayed.
headers:
Idempotency-Expires-At:
$ref: '#/components/headers/IdempotencyExpiresAt'
content:
application/json:
schema:
$ref: '#/components/schemas/Job'
default:
$ref: '#/components/responses/Error'
/v1/jobs/{job_id}/hard-delete:
parameters:
- $ref: '#/components/parameters/JobId'
post:
operationId: hardDeleteJob
parameters:
- $ref: '#/components/parameters/IdempotencyKey'
responses:
'202':
description: Durable deletion operation created or replayed.
headers:
Idempotency-Expires-At:
$ref: '#/components/headers/IdempotencyExpiresAt'
content:
application/json:
schema:
$ref: '#/components/schemas/DeletionOperation'
default:
$ref: '#/components/responses/Error'
/v1/jobs/{job_id}/revoke:
parameters:
- $ref: '#/components/parameters/JobId'
post:
operationId: revokeJobExecutionGrant
description: Administrator-only revocation of an accepted execution grant.
parameters:
- $ref: '#/components/parameters/IdempotencyKey'
responses:
'202':
description: Durable cancellation intent recorded or prior response replayed.
headers:
Idempotency-Expires-At:
$ref: '#/components/headers/IdempotencyExpiresAt'
content:
application/json:
schema:
$ref: '#/components/schemas/Job'
default:
$ref: '#/components/responses/Error'
/v1/jobs/{job_id}/attempts:
parameters:
- $ref: '#/components/parameters/JobId'
get:
operationId: listJobAttempts
parameters:
- $ref: '#/components/parameters/Cursor'
- $ref: '#/components/parameters/Limit'
responses:
'200':
description: Durable attempt history ordered by fencing token.
content:
application/json:
schema:
$ref: '#/components/schemas/AttemptPage'
'404':
$ref: '#/components/responses/NotFound'
default:
$ref: '#/components/responses/Error'
/v1/deletions/{operation_id}:
parameters:
- name: operation_id
in: path
required: true
schema:
$ref: '#/components/schemas/OpaqueId'
get:
operationId: getDeletionOperation
responses:
'200':
description: Current deletion operation.
content:
application/json:
schema:
$ref: '#/components/schemas/DeletionOperation'
'404':
$ref: '#/components/responses/NotFound'
default:
$ref: '#/components/responses/Error'
/v1/events:
get:
operationId: connectEvents
parameters:
- name: cursor
in: query
description: Opaque authorization-bound replay cursor from an HTTP snapshot or prior frame.
schema:
type: string
minLength: 16
maxLength: 2048
responses:
'101':
description: WebSocket frames conform to events.schema.json. An absent, invalid, expired, or unauthorized cursor yields resync_required.
default:
$ref: '#/components/responses/Error'
/internal/v1/jobs/{job_id}/claim:
parameters:
- $ref: '#/components/parameters/JobId'
post:
operationId: claimJob
security:
- serviceAuth: []
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/ClaimRequest'
responses:
'200':
description: Allocated authority or replay of the same claim operation.
content:
application/json:
schema:
$ref: '#/components/schemas/ClaimResponse'
default:
$ref: '#/components/responses/Error'
/internal/v1/attempts/{attempt_id}/renew:
parameters:
- $ref: '#/components/parameters/AttemptId'
post:
operationId: renewAttempt
security:
- serviceAuth: []
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/AttemptRenewRequest'
responses:
'204':
description: Existing live coordination lease renewed or duplicate operation accepted; durable authority is unchanged.
'409':
$ref: '#/components/responses/PreconditionFailed'
default:
$ref: '#/components/responses/Error'
/internal/v1/attempts/{attempt_id}/progress:
parameters:
- $ref: '#/components/parameters/AttemptId'
post:
operationId: reportAttemptProgress
security:
- serviceAuth: []
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/AttemptProgressRequest'
responses:
'204':
description: Current progress accepted or duplicate ignored.
'409':
$ref: '#/components/responses/PreconditionFailed'
default:
$ref: '#/components/responses/Error'
/internal/v1/attempts/{attempt_id}/outcome:
parameters:
- $ref: '#/components/parameters/AttemptId'
post:
operationId: reportAttemptOutcome
security:
- serviceAuth: []
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/AttemptOutcomeRequest'
responses:
'200':
description: Durable outcome projection or replay of the same operation.
content:
application/json:
schema:
$ref: '#/components/schemas/Job'
'409':
$ref: '#/components/responses/PreconditionFailed'
default:
$ref: '#/components/responses/Error'
components:
securitySchemes:
oidc:
type: openIdConnect
openIdConnectUrl: https://identity.invalid/.well-known/openid-configuration
serviceAuth:
type: mutualTLS
parameters:
IdempotencyKey:
name: Idempotency-Key
in: header
required: true
description: Subject- and operation-scoped key retained for at least 24 hours.
schema:
type: string
pattern: '^[A-Za-z0-9._~-]{16,128}$'
Cursor:
name: cursor
in: query
schema:
type: string
minLength: 16
maxLength: 2048
Limit:
name: limit
in: query
schema:
type: integer
minimum: 1
maximum: 200
default: 50
JobId:
name: job_id
in: path
required: true
schema:
$ref: '#/components/schemas/OpaqueId'
AttemptId:
name: attempt_id
in: path
required: true
schema:
$ref: '#/components/schemas/OpaqueId'
headers:
IdempotencyExpiresAt:
description: Earliest time at which the server may forget this key.
schema:
type: string
format: date-time
responses:
Error:
description: Structured error with user-safe detail only.
content:
application/problem+json:
schema:
$ref: '#/components/schemas/ApiError'
NotFound:
description: Missing or unauthorized resource; the response does not distinguish them.
content:
application/problem+json:
schema:
$ref: '#/components/schemas/ApiError'
IdempotencyConflict:
description: The key exists with a different canonical request fingerprint.
content:
application/problem+json:
schema:
$ref: '#/components/schemas/ApiError'
PreconditionFailed:
description: Durable authority, fence, phase, or lifecycle precondition failed.
content:
application/problem+json:
schema:
$ref: '#/components/schemas/ApiError'
schemas:
OpaqueId:
type: string
pattern: '^[A-Za-z0-9][A-Za-z0-9_-]{7,127}$'
Target:
type: object
additionalProperties: false
required: [store, bucket, prefix]
properties:
store:
type: string
minLength: 1
maxLength: 128
bucket:
type: string
pattern: '^(?!.*\.\.)[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$'
prefix:
type: string
maxLength: 1024
description: NFC-normalized target segments; semantic validation follows architecture.md.
TargetOverride:
type: object
additionalProperties: false
minProperties: 1
properties:
bucket:
type: string
pattern: '^(?!.*\.\.)[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$'
prefix:
type: string
maxLength: 1024
description: NFC-normalized target segments; semantic validation follows architecture.md.
StagingTarget:
type: object
additionalProperties: false
required: [store, bucket, prefix]
properties:
store:
type: string
minLength: 1
maxLength: 128
bucket:
type: string
pattern: '^(?!.*\.\.)[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$'
prefix:
type: string
minLength: 1
maxLength: 1024
description: Exact attempt-scoped internal prefix selected by the service.
HttpSource:
type: object
additionalProperties: false
required: [kind, url]
properties:
kind:
const: http
url:
type: string
format: uri
maxLength: 8192
credential_ref:
type: string
maxLength: 512
ObjectStoreSource:
type: object
additionalProperties: false
required: [kind, store, bucket, key]
properties:
kind:
const: object_store
store:
type: string
minLength: 1
maxLength: 128
bucket:
type: string
minLength: 3
maxLength: 63
key:
type: string
minLength: 1
maxLength: 4096
version_id:
type: string
maxLength: 1024
credential_ref:
type: string
maxLength: 512
HuggingFaceSource:
type: object
additionalProperties: false
required: [kind, repository_type, repository_id, revision]
properties:
kind:
const: hugging_face
repository_type:
type: string
enum: [model, dataset, space]
repository_id:
type: string
pattern: '^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$'
revision:
type: string
minLength: 1
maxLength: 256
include:
type: array
maxItems: 1024
uniqueItems: true
items:
type: string
minLength: 1
maxLength: 1024
credential_ref:
type: string
maxLength: 512
Source:
oneOf:
- $ref: '#/components/schemas/HttpSource'
- $ref: '#/components/schemas/ObjectStoreSource'
- $ref: '#/components/schemas/HuggingFaceSource'
discriminator:
propertyName: kind
mapping:
http: '#/components/schemas/HttpSource'
object_store: '#/components/schemas/ObjectStoreSource'
hugging_face: '#/components/schemas/HuggingFaceSource'
JobSubmission:
type: object
additionalProperties: false
required: [source]
properties:
source:
$ref: '#/components/schemas/Source'
target:
description: Fields omitted here are taken from the configured default target before authorization.
$ref: '#/components/schemas/TargetOverride'
label:
type: string
minLength: 1
maxLength: 200
JobPhase:
type: string
enum: [Queued, Running, RetryWait, Cancelling, Succeeded, Failed, Cancelled]
JobLifecycle:
type: string
enum: [visible, soft_deleted, hard_deleting, hard_deleted]
AttemptAuthority:
type: object
additionalProperties: false
required: [attempt_id, fence, worker_incarnation]
properties:
attempt_id:
$ref: '#/components/schemas/OpaqueId'
fence:
type: integer
minimum: 1
worker_incarnation:
$ref: '#/components/schemas/OpaqueId'
Progress:
type: object
additionalProperties: false
required: [attempt_id, fence, completed_bytes, completed_members, observed_at]
properties:
attempt_id:
$ref: '#/components/schemas/OpaqueId'
fence:
type: integer
minimum: 1
completed_bytes:
type: integer
minimum: 0
total_bytes:
type: [integer, 'null']
minimum: 0
completed_members:
type: integer
minimum: 0
total_members:
type: [integer, 'null']
minimum: 0
observed_at:
type: string
format: date-time
ProgressValue:
type: object
additionalProperties: false
required: [completed_bytes, completed_members]
properties:
completed_bytes:
type: integer
minimum: 0
total_bytes:
type: [integer, 'null']
minimum: 0
completed_members:
type: integer
minimum: 0
total_members:
type: [integer, 'null']
minimum: 0
CheckpointProjection:
type: object
additionalProperties: false
required: [reusable, validated_bytes, validated_members]
properties:
reusable:
type: boolean
validated_bytes:
type: integer
minimum: 0
validated_members:
type: integer
minimum: 0
JobFailure:
oneOf:
- type: object
additionalProperties: false
required: [class, retryable, message]
properties:
class:
type: string
enum: [validation, authorization, source_not_found, source_changed, cancelled]
retryable:
const: false
message:
type: string
maxLength: 500
safe_details:
type: object
additionalProperties:
type: [string, integer, boolean, 'null']
- type: object
additionalProperties: false
required: [class, retryable, message]
properties:
class:
type: string
enum: [transient_source, transient_target, integrity, capacity, internal, lost_attempt]
retryable:
const: true
message:
type: string
maxLength: 500
safe_details:
type: object
additionalProperties:
type: [string, integer, boolean, 'null']
AttemptRuntimeFailure:
type: object
additionalProperties: false
required: [class, message]
properties:
class:
type: string
enum: [authorization, source_not_found, source_changed, transient_source, transient_target, integrity, capacity, internal]
message:
type: string
maxLength: 500
safe_details:
type: object
additionalProperties:
type: [string, integer, boolean, 'null']
ResultDescriptor:
type: object
additionalProperties: false
required: [kind, manifest, member_count, total_bytes]
properties:
kind:
const: manifest
manifest:
$ref: '#/components/schemas/ObjectIdentity'
member_count:
type: integer
minimum: 1
total_bytes:
type: integer
minimum: 0
ObjectIdentity:
type: object
additionalProperties: false
required: [store, bucket, key]
properties:
store:
type: string
bucket:
type: string
key:
type: string
version_id:
type: string
checksum:
type: string
Job:
type: object
additionalProperties: false
required: [id, owner, label, revision, phase, lifecycle, source, target, authoritative_attempt, progress, checkpoint, result, failure, created_at, updated_at, event_cursor]
properties:
id:
$ref: '#/components/schemas/OpaqueId'
owner:
$ref: '#/components/schemas/OpaqueId'
label:
type: [string, 'null']
minLength: 1
maxLength: 200
revision:
type: integer
minimum: 1
phase:
$ref: '#/components/schemas/JobPhase'
lifecycle:
$ref: '#/components/schemas/JobLifecycle'
source:
$ref: '#/components/schemas/Source'
target:
$ref: '#/components/schemas/Target'
authoritative_attempt:
oneOf:
- $ref: '#/components/schemas/AttemptAuthority'
- type: 'null'
progress:
oneOf:
- $ref: '#/components/schemas/Progress'
- type: 'null'
checkpoint:
oneOf:
- $ref: '#/components/schemas/CheckpointProjection'
- type: 'null'
result:
oneOf:
- $ref: '#/components/schemas/ResultDescriptor'
- type: 'null'
failure:
oneOf:
- $ref: '#/components/schemas/JobFailure'
- type: 'null'
created_at:
type: string
format: date-time
updated_at:
type: string
format: date-time
event_cursor:
type: string
minLength: 16
JobPage:
type: object
additionalProperties: false
required: [items, next_cursor, event_cursor]
properties:
items:
type: array
items:
$ref: '#/components/schemas/Job'
next_cursor:
type: [string, 'null']
event_cursor:
type: string
minLength: 16
AttemptRecord:
type: object
additionalProperties: false
required: [id, fence, worker_incarnation, state, progress, result, failure, created_at, updated_at]
properties:
id:
$ref: '#/components/schemas/OpaqueId'
fence:
type: integer
minimum: 1
worker_incarnation:
$ref: '#/components/schemas/OpaqueId'
state:
type: string
enum: [AttemptAuthorized, AttemptExecuting, AttemptStopped, AttemptCommitted]
progress:
oneOf:
- $ref: '#/components/schemas/Progress'
- type: 'null'
result:
oneOf:
- $ref: '#/components/schemas/ResultDescriptor'
- type: 'null'
failure:
oneOf:
- $ref: '#/components/schemas/JobFailure'
- type: 'null'
created_at:
type: string
format: date-time
updated_at:
type: string
format: date-time
AttemptPage:
type: object
additionalProperties: false
required: [items, next_cursor]
properties:
items:
type: array
items:
$ref: '#/components/schemas/AttemptRecord'
next_cursor:
type: [string, 'null']
DeletionProblem:
type: object
additionalProperties: false
required: [class, message]
properties:
class:
type: string
enum: [authorization, target_unavailable, configuration, internal]
message:
type: string
maxLength: 500
DeletionOperation:
type: object
additionalProperties: false
required: [id, job_id, state, problem, created_at, updated_at]
properties:
id:
$ref: '#/components/schemas/OpaqueId'
job_id:
$ref: '#/components/schemas/OpaqueId'
state:
type: string
enum: [queued, running, retry_wait, blocked, completed]
problem:
oneOf:
- $ref: '#/components/schemas/DeletionProblem'
- type: 'null'
created_at:
type: string
format: date-time
updated_at:
type: string
format: date-time
ApiError:
type: object
additionalProperties: false
required: [code, message, request_id]
properties:
code:
type: string
enum: [invalid_request, unauthenticated, forbidden, not_found, idempotency_conflict, precondition_failed, cursor_invalid, cursor_expired, rate_limited, unavailable, internal]
message:
type: string
maxLength: 500
request_id:
$ref: '#/components/schemas/OpaqueId'
fields:
type: array
maxItems: 100
items:
type: object
additionalProperties: false
required: [pointer, code]
properties:
pointer:
type: string
maxLength: 500
code:
type: string
maxLength: 100
ClaimRequest:
type: object
additionalProperties: false
required: [operation_id, worker_incarnation]
properties:
operation_id:
$ref: '#/components/schemas/OpaqueId'
worker_incarnation:
$ref: '#/components/schemas/OpaqueId'
ExecutionEnvelope:
type: object
additionalProperties: false
required: [id, job_id, attempt_id, fence, worker_incarnation, source, resolved_source_ref, target, staging_target, execution_grant_ref, checkpoint_ref]
properties:
id:
$ref: '#/components/schemas/OpaqueId'
job_id:
$ref: '#/components/schemas/OpaqueId'
attempt_id:
$ref: '#/components/schemas/OpaqueId'
fence:
type: integer
minimum: 1
worker_incarnation:
$ref: '#/components/schemas/OpaqueId'
source:
$ref: '#/components/schemas/Source'
resolved_source_ref:
oneOf:
- $ref: '#/components/schemas/OpaqueId'
- type: 'null'
target:
$ref: '#/components/schemas/Target'
staging_target:
$ref: '#/components/schemas/StagingTarget'
execution_grant_ref:
$ref: '#/components/schemas/OpaqueId'
checkpoint_ref:
oneOf:
- $ref: '#/components/schemas/OpaqueId'
- type: 'null'
ClaimResponse:
type: object
additionalProperties: false
required: [authority, execution]
properties:
authority:
$ref: '#/components/schemas/AttemptAuthority'
execution:
$ref: '#/components/schemas/ExecutionEnvelope'
AttemptMutation:
type: object
required: [operation_id, fence, worker_incarnation]
properties:
operation_id:
$ref: '#/components/schemas/OpaqueId'
fence:
type: integer
minimum: 1
worker_incarnation:
$ref: '#/components/schemas/OpaqueId'
AttemptRenewRequest:
unevaluatedProperties: false
allOf:
- $ref: '#/components/schemas/AttemptMutation'
AttemptProgressRequest:
unevaluatedProperties: false
allOf:
- $ref: '#/components/schemas/AttemptMutation'
- type: object
required: [progress]
properties:
progress:
$ref: '#/components/schemas/ProgressValue'
AttemptOutcomeRequest:
unevaluatedProperties: false
allOf:
- $ref: '#/components/schemas/AttemptMutation'
- type: object
required: [outcome]
properties:
outcome:
oneOf:
- type: object
additionalProperties: false
required: [kind, result]
properties:
kind:
const: success
result:
$ref: '#/components/schemas/ResultDescriptor'
- type: object
additionalProperties: false
required: [kind, failure]
properties:
kind:
const: failure
failure:
$ref: '#/components/schemas/AttemptRuntimeFailure'

9
docs/protocol/Remora.tla
View file

@ -97,6 +97,7 @@ NoOutcome == "NoOutcome"
WantsSuccess == "WantsSuccess"
WantsRetryableFailure == "WantsRetryableFailure"
WantsPermanentFailure == "WantsPermanentFailure"
\* Adapters map concrete failure classes to these dispositions before JobProtocol.
FailureOutcomes == {WantsRetryableFailure, WantsPermanentFailure}
OutcomeRequests == {NoOutcome, WantsSuccess} \cup FailureOutcomes
@ -113,7 +114,9 @@ TokenOf(a) == a.token
VARIABLES
\* DurableStore authority.
phase, cancelRequested, currentToken, authority, dispatchNeeded,
committedAttempt, checkpoint, checkpointValid,
committedAttempt, checkpoint,
\* Abstracts the complete source-identity and destination-state proof.
checkpointValid,
\* CoordinationStore (reconstructible).
queued, reservation, leaseState, leaseAge, cancelNotice,
@ -275,6 +278,7 @@ AuthorizeAttempt(j, w) ==
/\ UNCHANGED workerVars
/\ UNCHANGED pendingOutcome
\* Concrete ExecutionEnvelope validation is abstracted into activation.
ActivateAttempt(a) ==
LET j == JobOf(a)
w == attemptWorker[a]
@ -447,6 +451,7 @@ ReapLostAttempt(a) ==
/\ UNCHANGED <<attemptWorker, attemptProgress, initialProgress,
resumedFromCheckpoint, stageState, pendingOutcome>>
\* RetryWait abstracts backoff; operation timeouts are outcomes, not a job deadline.
ScheduleRetry(j) ==
/\ phase[j] = RetryWait
/\ phase' = [phase EXCEPT ![j] = Queued]
@ -721,6 +726,7 @@ RejectOutcome(a) ==
/\ UNCHANGED <<attemptState, attemptWorker, attemptProgress,
initialProgress, resumedFromCheckpoint, stageState>>
\* Concrete cleanup streams exact durable identities; this action models eligibility.
CleanupStage(a) ==
LET j == JobOf(a)
IN /\ stageState[a] \in {StagePartial, StageComplete}
@ -953,6 +959,7 @@ StagingOwnership ==
/\ (committedAttempt[JobOf(a)] = a =>
stageState[a] = StageComplete)
\* Deletion is outside JobProtocol and must preserve terminal-state immutability.
TerminalStep ==
\A j \in Jobs :
Terminal(j) =>

1
docs/protocol/RemoraApiMutation.tla
View file

@ -3,6 +3,7 @@ EXTENDS Naturals, FiniteSets
\* This model covers the durable API operation ledger around an abstract
\* JobProtocol effect; JobProtocol remains the sole phase-transition model.
\* Identities range over one retention epoch; record expiry is outside the model.
CONSTANTS
OperationScopes, EligibilityClaims, OperationIds, MutationKinds,

84
docs/spec/architecture.md
View file

@ -9,13 +9,14 @@ Terms in this glossary are stable identifiers for use across Remora documentatio
| Identifier | Definition |
| ---------- | ---------- |
| `Job` | A durable request to transfer one object or multi-object source snapshot from a `Source` to a `TargetStore`. |
| `Attempt` | One execution of a `Job`. At-least-once delivery may produce more than one `Attempt` for a `Job`. |
| `Attempt` | A concrete execution corresponding to an attempt allocated by `JobProtocol`. |
| `WorkerIncarnation` | One lifetime of a `Worker` process. A restarted process is a new incarnation even when it occupies the same replica slot. |
| `Claim` | Durable authority for one `Attempt`, identified by its attempt ID, worker incarnation, and monotonically increasing fencing token. |
| `Lease` | Reconstructible coordination state granting a live attempt a deployment-wide concurrency permit for a bounded interval. |
| `Claim` | The worker-facing projection of `JobProtocol` attempt authority, containing its opaque attempt ID, worker incarnation, and fencing token. |
| `Lease` | The concrete coordination record representing a `JobProtocol` lease. |
| `ExecutionEnvelope` | The attempt-bound projection of durable inputs that a worker must validate before source or target I/O. |
| `StagingObject` | An immutable, attempt-specific result member or manifest object that has not been selected by durable publication. |
| `ResultManifest` | The immutable exact inventory and root identity for one attempt's result aggregate. |
| `CommittedResult` | The durable descriptor that selects one `ResultManifest` for a successful job. |
| `CommittedResult` | The concrete durable descriptor corresponding to the attempt committed by `JobProtocol`. |
| `WebClient` | The browser-based user interface for submitting and managing jobs. |
| `API` | The authenticated service interface for job submission, status, cancellation, history, and deletion. |
| `Worker` | A stateless process that executes attempts and reports progress and outcomes. |
@ -29,9 +30,9 @@ Terms in this glossary are stable identifiers for use across Remora documentatio
## Scope
This document defines Remora's product requirements, system boundaries, major components, result representation and access contract, and allocation of responsibilities. It intentionally does not define the distributed job protocol or implementation-level technology choices.
This document is authoritative for Remora's product requirements, system boundaries, major components, result representation and access contract, and allocation of responsibilities below the abstraction of `JobProtocol`. It intentionally does not define implementation-level technology choices.
`JobProtocol` defines job state transitions, at-least-once delivery, claims and fencing, retry scheduling, cancellation races, recovery, and the associated safety and liveness properties. Concrete frameworks, libraries, wire formats, and deployment choices are outside the scope of this specification.
[`Remora.tla`](../protocol/Remora.tla) is the authoritative `JobProtocol`. This document references its symbols without restating their definitions or actions; where they overlap, the model controls. Concrete frameworks, libraries, wire formats, mutable API descriptions, and deployment choices are outside the scope of this specification.
## Problem Statement
@ -41,7 +42,7 @@ Pulling large objects from remote sources into S3-compatible storage is commonly
Remora exposes a browser UI and API for submitting, monitoring, and managing download jobs into a target object store. Supported source modes are HTTP URLs, compatible object stores, and Hugging Face repository snapshots.
Each download is an independent background job with streaming I/O, automatic retry, live progress, and cancellation. A failed attempt resumes when source and destination support safe resumption; otherwise its replacement starts from the beginning. Jobs default to a configured target bucket and allow per-request bucket and prefix overrides subject to authorization and validation. Every successful job publishes one immutable `ResultManifest`, including single-object jobs.
Each download is an independent background job with streaming I/O and the retry, progress, and cancellation behavior defined by `JobProtocol`. A replacement attempt can reuse concrete checkpoint state only when its source and destination proofs remain valid. Jobs default to a configured target bucket and allow per-request bucket and prefix overrides subject to authorization and validation. The attempt committed by `JobProtocol` is exposed through one immutable `ResultManifest`.
Identity, coordination, and durable state are external services. Multiple API and worker replicas share them for authenticated multi-user operation.
@ -49,9 +50,9 @@ Identity, coordination, and durable state are external services. Multiple API an
`WebClient` communicates with `API`, which authorizes requests through `IdentityProvider`, records jobs in `DurableStore`, and submits work through `CoordinationStore`. `Worker` replicas consume jobs and stream data from `Source` through `StorageAdapter` into `TargetStore`.
`API` and `Worker` replicas are stateless. `DurableStore` contains authoritative job records and committed result descriptors. `CoordinationStore` contains reconstructible operational state such as queue entries, leases, deployment-wide permits, cancellation notifications, and live progress.
`API` and `Worker` replicas are stateless. `DurableStore` realizes `durableVars` and retains architecture data outside the model. `CoordinationStore` realizes `coordinationVars` and may be reconstructed from durable state as permitted by `JobProtocol`.
`TargetStore` contains immutable attempt-scoped objects. A `ResultManifest` roots each attempt-scoped aggregate; `DurableStore` contains the manifest descriptor exposed as the `CommittedResult`. `JobProtocol` is the sole specification of attempt authority and publication behavior.
`TargetStore` contains immutable attempt-scoped objects. The model's staging identity abstracts each concrete aggregate, and its committed attempt maps to the manifest descriptor exposed as the `CommittedResult`. `JobProtocol` is the sole specification of attempt authority and publication behavior.
Workers enforce a configurable outbound-source policy for user-supplied locations on every connection, including redirects. Deployments also restrict worker egress as a defense-in-depth boundary. Private or otherwise sensitive destinations are denied by default unless explicitly allowed by deployment policy.
@ -96,50 +97,60 @@ flowchart TD
StorageAdapter --> TargetStore
```
## Worker Execution Envelope
- `ExecutionEnvelope` supplies the concrete inputs abstracted by `ActivateAttempt`. Before concrete source or target I/O, a worker verifies that its job ID, opaque attempt ID, fencing token, and worker incarnation match the activated attempt projection.
- The envelope projects the accepted source request and any durable resolved source identity, the immutable execution grant and concrete target, the attempt-scoped staging identity, opaque credential references, and the validated checkpoint and destination-resume evidence available to that attempt.
- A source identity or inventory that is initially unresolved is durably pinned before concrete checkpoint validation or reuse relies on it.
- Large source inventories, completed-part sets, and other unbounded collections are represented by durable streamable descriptors rather than materialized in the envelope.
- Envelope transport is an implementation choice. It may accompany queue delivery or claim, or be read separately, but a worker never infers it from reconstructible coordination state.
- A missing envelope, an identity mismatch, an inaccessible credential reference, or invalid resume evidence fails closed. None grants permission to use a different source, target, staging identity, or checkpoint.
## Result Aggregate and Direct Access
- Every successful job is represented by one immutable `ResultManifest`. A single-object transfer uses a one-member manifest.
- The attempt selected by `committedAttempt` is represented by one immutable `ResultManifest`. A single-object transfer uses a one-member manifest.
- A manifest is the exact inventory of immutable member identities and carries sufficient integrity metadata to validate the aggregate.
- This specification defines the manifest's logical invariants, not its serialization or target-object layout. An implementation that exposes the manifest object directly to clients publishes a versioned format contract; an implementation may instead keep the format private and expose member enumeration and access through the service.
- Manifest member paths are NFC-normalized relative POSIX segment sequences. Empty segments, dot segments, traversal, leading or trailing separators, and backslashes are invalid rather than normalized into aliases.
- Manifest construction uses memory bounded independently of member count and total result size.
- Remora resolves a result only from its `CommittedResult`; it never derives one by listing a target bucket or prefix.
- Remora resolves the concrete aggregate only from the selected `CommittedResult`; it never derives one by listing a target bucket or prefix.
- An authorized reader may access committed members directly in `TargetStore` using exact manifest identities or time-limited URLs.
- Raw target listing is operational storage visibility, not result resolution. Clients with independent target credentials may observe uncommitted attempt objects.
- A stable user-chosen object key or atomically visible raw target prefix is not part of a job result. Providing one requires a separate export contract.
## Target Cleanup
- `DurableStore` retains exact attempt ownership and pending cleanup work so cleanup can recover independently of `CoordinationStore`.
- Concrete work for `CleanupStage` is materialized from exact attempt ownership and pending cleanup data in `DurableStore`, independently of `CoordinationStore`.
- Ambient cleanup processes are stateless and idempotent. Worker-local cleanup may reduce latency but is not the only cleanup path.
- Cleanup revalidates authority in `DurableStore` and addresses exact attempt object or multipart identities. Missing coordination state, target-object age, and raw prefix membership do not grant cleanup authority.
- Target lifecycle rules are storage hygiene only. They do not determine result publication or cleanup eligibility and must not expire completed attempt objects that Remora may still protect.
- Cleanup addresses exact attempt object or multipart identities. Eligibility comes only from `CleanupStage`; missing coordination state, target-object age, and raw prefix membership are not substitutes.
- Target lifecycle rules are storage hygiene only. They do not substitute for `JobProtocol` publication or cleanup eligibility and must not expire objects retained by a committed aggregate.
## Source Identity and Resumption
- A reusable checkpoint binds an immutable `Source` identity, an exact completed boundary, and validated destination state. If any proof is absent or changed, the replacement restarts that member from the beginning.
- `checkpointValid` abstracts the concrete proof defined here. A reusable checkpoint binds an immutable `Source` identity, an exact completed boundary, and validated destination state; absent or changed proof makes the checkpoint unavailable to a replacement attempt.
- HTTP identity includes the resolved final URL, a strong representation validator as defined by `RFC 9110`, representation length, and identity content encoding. Resume requires a conditional byte-range response whose validator, URL, encoding, and range exactly match the checkpoint. Weak validators, length, or modification time alone do not qualify.
- Object-store identity includes endpoint, bucket, key, size, and an immutable version ID when available. Without a version ID, resume requires a matching conditional range read. Destination multipart reuse additionally requires the same endpoint, bucket, key, upload ID, and exact re-listed completed parts.
- A Hugging Face branch or tag is resolved once to a full commit. The source identity includes repository type and ID, that commit, and the exact file inventory with Git blob or LFS identities. Retries and cache reuse remain pinned to the stored commit and member identities.
- A checkpoint never authorizes best-effort concatenation. An unavailable explicitly pinned or resolved snapshot produces a structured source-changed outcome rather than silently resolving a newer snapshot.
- A checkpoint never authorizes best-effort concatenation. The adapter maps an unavailable explicitly pinned or resolved snapshot to `source_changed` rather than silently resolving a newer snapshot.
## Retry and Structured Failures
- Structured failures use the stable classes `validation`, `authorization`, `source_not_found`, `source_changed`, `transient_source`, `transient_target`, `integrity`, `capacity`, `internal`, `cancelled`, and `lost_attempt`.
- Attempt-reported runtime failures carry a durable permanent or retryable disposition into `JobProtocol`. The model is the sole definition of how that disposition affects execution phase and attempt budget.
- Authorization, source-not-found, and source-changed runtime failures are permanent. Transient-source, transient-target, integrity, capacity, and internal failures are retryable. Unrecognized runtime errors map to a redacted internal failure.
- Validation normally rejects submission before a job exists. Cancellation and lost attempts follow their dedicated `JobProtocol` behavior rather than a worker-selected failure disposition.
- All retryable work uses the model-wide attempt bound. Source adapters may vary connect, response-start, no-progress, bounded full-jitter, and bounded server-hint timing without changing that bound. Invalid or excessive server hints are ignored or capped; they never change a retryable disposition into a permanent failure.
- Remora has no elapsed wall-clock job deadline. Per-operation timeouts produce a structured runtime failure and remain subject to `JobProtocol`.
- Attempt-reported runtime classes map to the permanent or retryable disposition supplied to `JobProtocol`: authorization, source-not-found, and source-changed are permanent; transient-source, transient-target, integrity, capacity, and internal are retryable. Unrecognized runtime errors map to a redacted internal failure.
- Validation rejects submission before the job enters `JobProtocol`. Cancellation and lost attempts are protocol inputs rather than worker-selected failure classes.
- Source adapters select connect, response-start, no-progress, bounded full-jitter, and bounded server-hint timing. Invalid or excessive server hints are ignored or capped before the adapter supplies an outcome to `JobProtocol`.
- Remora supplies no elapsed wall-clock deadline to `JobProtocol`. Per-operation timeouts map to structured runtime failures.
- Public failure details are constructed from per-class allowlists. Credentials, upstream bodies, raw URLs, and internal exception text are excluded.
## Result Visibility and Deletion
- `JobProtocol` remains the sole owner of execution phase. Deletion never rewrites a terminal execution outcome and is available only after a job is terminal; cancelling active work remains a separate protocol operation.
- Deletion is an architecture lifecycle outside `JobProtocol`. It may begin only from a terminal protocol projection; active work uses the protocol cancellation input instead.
- Soft deletion tombstones Remora visibility immediately without deleting target objects. It is idempotent and may be restored until hard deletion begins.
- Hard deletion atomically tombstones visibility and creates one durable, idempotent deletion operation. Once that operation exists, the result cannot be restored even while physical deletion is pending or retrying.
- A hard-deletion operation uses the `ResultManifest` and durable attempt ownership to materialize exact deletion work with bounded memory. It removes committed members and eligible attempt data, treats an already-missing exact identity as success, and deletes the manifest only after its members.
- A hard-deletion operation uses the `ResultManifest` and durable attempt ownership to materialize exact deletion work with bounded memory. It removes committed members and deletion-eligible attempt data, treats an already-missing exact identity as success, and deletes the manifest only after its members.
- Physical deletion is asynchronous and recoverable across worker loss and transient target failures. It does not depend on client request lifetime or reconstruct work by listing a target prefix.
- A permanent minimal tombstone retains job identity, owner, terminal outcome, deletion identity, and audit timing. Retention of more detailed job metadata is deployment policy and never restores result visibility after hard deletion begins.
- A deletion operation that cannot progress because required target access or configuration is unavailable retains its exact remaining work and a redacted administrative reason. It resumes from that work when the dependency is restored; it never abandons the deletion obligation or restores result visibility.
- A permanent minimal tombstone retains job identity, owner, concrete target authorization tuple, terminal outcome, deletion identity, and audit timing. Retention of more detailed job metadata is deployment policy and never restores result visibility after hard deletion begins.
## Authorization and Credentials
@ -148,8 +159,9 @@ flowchart TD
- A target prefix is represented as NFC-normalized, case-sensitive Unicode segments separated by `/`. Empty prefix means the complete bucket, and one trailing separator is insignificant. Leading or doubled separators, dot segments, backslashes, Unicode category-C characters, and deployment-reserved internal segments are invalid.
- A storage scope contains one target-store identity, bucket, and prefix segment tuple. It covers a target only when store and bucket match exactly and its segments are a complete prefix of the target segments; textual prefix coincidence is insufficient.
- An owner may list, read, cancel, soft-delete, restore, or hard-delete a job only while current authorization still covers its concrete target. Administrators may manage all jobs and tombstones. List operations apply the same rule as a filter and never reveal unauthorized job existence.
- Owner access to a deletion operation is authorized against the concrete target retained in its job tombstone. Missing and unauthorized deletion operations are indistinguishable.
- Workers execute an accepted job from its immutable grant without consulting current identity roles. Failure to establish current authorization fails new submission, reads, and management closed but does not reinterpret accepted execution authority.
- An administrator may explicitly and idempotently revoke an accepted execution grant. Revocation records durable cancellation intent through `JobProtocol`; it does not add an identity-provider lookup to workers or rewrite a terminal outcome.
- An administrator may explicitly and idempotently revoke an accepted execution grant. Revocation maps to the protocol cancellation input when applicable and does not add an identity-provider lookup to workers.
- Source credentials are server-side opaque references resolved only by the worker component that needs them. Secret values never appear in job configuration, API data, events, logs, or structured failures. Target credentials remain internal to `StorageAdapter`.
## Outbound Source Policy
@ -163,12 +175,12 @@ flowchart TD
## API and Live Updates
- [`api/openapi.yaml`](api/openapi.yaml) is the normative HTTP contract and [`api/events.schema.json`](api/events.schema.json) is the normative WebSocket frame contract. Their compatibility and replay semantics are defined in [`api/README.md`](api/README.md).
- Every public mutation uses a caller- and operation-scoped idempotency key. The server atomically retains the canonical request fingerprint and original response for at least the advertised 24-hour default window; changed payload reuse is a conflict.
- Public job status projects the `JobProtocol` phase without redefining its transitions and separately exposes deletion lifecycle, current authoritative attempt, matching progress, validated checkpoint, committed result, and structured failure.
- Mutable HTTP and WebSocket implementation contracts live under [`docs/api`](../api/). They are non-authoritative projections of `JobProtocol` and this document and must be corrected when they conflict with either source.
- Public mutation idempotency refines `RemoraApiMutation`: operation scope is the caller subject and operation, the request fingerprint is RFC 8785 canonical JSON hashed with SHA-256, and records are retained for at least the advertised 24-hour default window.
- Public job status projects `JobProtocol` state and adds deletion lifecycle, concrete progress, checkpoint, result, and failure fields.
- Job listing applies current authorization before stable keyset pagination. Cursors are opaque, integrity-protected, and bound to the subject, filters, and membership snapshot.
- HTTP snapshots carry a live-event cursor. WebSocket replay is ordered and at least once; invalid, expired, authorization-incompatible, or unavailable history requires a new HTTP snapshot. Stale-attempt progress is suppressed before emission.
- Internal claim identifies the job, worker incarnation, and idempotent operation because no attempt exists yet. Renew, progress, and outcome additionally require the opaque attempt ID and fence defined by `JobProtocol`; stale authority has no effect.
- HTTP snapshots carry a live-event cursor. WebSocket replay is ordered and at least once; invalid, expired, authorization-incompatible, or unavailable history requires a new HTTP snapshot. Event emission exposes progress only when it matches the protocol projection being published.
- Internal mutation requests add transport identifiers and an idempotent operation ID to the corresponding `JobProtocol` action; their eligibility and effects are not redefined here.
## Requirements
@ -186,7 +198,7 @@ flowchart TD
| FR-8 | Support soft deletion by tombstoning job metadata without deleting stored objects | Medium |
| FR-9 | Support hard deletion by deleting stored objects while retaining a metadata tombstone | Medium |
| FR-10 | Stream data and construct result inventories with memory bounded independently of complete object and aggregate size | High |
| FR-11 | Enforce independently configurable per-incarnation task limits and deployment-wide authoritative lease limits | Medium |
| FR-11 | Expose independent configuration for the per-incarnation and deployment-wide limits modeled by `JobProtocol` | Medium |
| FR-12 | Isolate attempt target writes and expose exactly one immutable `ResultManifest` for each successful job | High |
| FR-13 | Ensure distributed job operations refine `JobProtocol` under duplicate or ambiguous requests | High |
| FR-14 | Reconstruct operational coordination state after loss or failover as required by `JobProtocol` | High |
@ -196,7 +208,7 @@ flowchart TD
| Category | Requirement |
| ------------- | ----------------------------------------------------------------------------------------------------------- |
| Reliability | Preserve durable authority and committed results across worker or coordination failures; reconcile in-flight work by protocol |
| Reliability | Refine `JobProtocol` under worker or coordination failures and preserve concrete result data outside the model |
| Observability | Expose job status, progress, and errors through the UI and API |
| Scalability | Support multiple stateless API and worker replicas using shared state services |
| Performance | Stream multi-gigabyte objects without buffering complete objects in worker memory |
@ -205,8 +217,8 @@ flowchart TD
## Responsibility Boundaries
- This architecture specification defines components, trust boundaries, externally visible behavior, and responsibility allocation.
- `JobProtocol` defines distributed state and coordination semantics, including safety and liveness invariants.
- `JobProtocol` is the normative implementation contract for durable transitions, fencing, leases, retries, cancellation, publication, and reconciliation. Implementations may optimize storage layout but must refine its observable behavior and preserve its linearization points.
- Implementation documentation selects technologies and maps architectural components onto them.
- [`docs/protocol`](../protocol/) is authoritative for distributed state and coordination semantics, including safety and liveness invariants.
- This architecture specification is authoritative for components, trust boundaries, externally visible behavior, and responsibility allocation below the model's abstraction, and must conform to `JobProtocol` where they overlap.
- Implementations may optimize storage layout but must refine `JobProtocol` behavior and preserve its linearization points.
- Mutable implementation contracts and documentation select technologies, wire shapes, and mappings from architectural components. They are not normative over `JobProtocol` or this specification.
- Deployment configuration supplies service credentials and endpoints, configures allowed source reachability, and provides infrastructure-level egress enforcement.