ms2k/remora
SHA256
Watch
0
Code Releases Activity

Specify fenced activation and source credential contracts

Browse source 
Define bounded reservation recovery, exact activation replay fencing, immutable source identities, and deployment-owned secret and target policy boundaries. Align the OpenAPI contract and add a split-store activation safety refinement.
This commit is contained in:
ms2k 2026-07-14 18:12:28 +09:00
commit 033cfeaac6
15 changed files with 877 additions and 49 deletions
Download patch file Download diff file Expand all files Collapse all files

482
docs/api/openapi.yaml
View file

@ -257,6 +257,113 @@ paths:
$ref: '#/components/responses/NotFound' $ref: '#/components/responses/NotFound'
default: default:
$ref: '#/components/responses/Error' $ref: '#/components/responses/Error'
/v1/source-credentials:
post:
operationId: createSourceCredential
parameters:
- $ref: '#/components/parameters/IdempotencyKey'
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/SourceCredentialCreate'
responses:
'201':
description: Credential metadata and first immutable version created, or the original response replayed.
headers:
Idempotency-Expires-At:
$ref: '#/components/headers/IdempotencyExpiresAt'
content:
application/json:
schema:
$ref: '#/components/schemas/SourceCredential'
'409':
$ref: '#/components/responses/IdempotencyConflict'
default:
$ref: '#/components/responses/Error'
get:
operationId: listSourceCredentials
parameters:
- $ref: '#/components/parameters/Cursor'
- $ref: '#/components/parameters/Limit'
responses:
'200':
description: Authorized credential metadata page without secret material.
content:
application/json:
schema:
$ref: '#/components/schemas/SourceCredentialPage'
default:
$ref: '#/components/responses/Error'
/v1/source-credentials/{credential_id}:
parameters:
- $ref: '#/components/parameters/CredentialId'
get:
operationId: getSourceCredential
responses:
'200':
description: Authorized credential metadata without secret material.
content:
application/json:
schema:
$ref: '#/components/schemas/SourceCredential'
'404':
$ref: '#/components/responses/NotFound'
default:
$ref: '#/components/responses/Error'
/v1/source-credentials/{credential_id}/versions:
parameters:
- $ref: '#/components/parameters/CredentialId'
post:
operationId: rotateSourceCredential
parameters:
- $ref: '#/components/parameters/IdempotencyKey'
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/SourceCredentialRotate'
responses:
'201':
description: New immutable version created and selected for future grants, or the original response replayed.
headers:
Idempotency-Expires-At:
$ref: '#/components/headers/IdempotencyExpiresAt'
content:
application/json:
schema:
$ref: '#/components/schemas/SourceCredential'
'404':
$ref: '#/components/responses/NotFound'
'409':
$ref: '#/components/responses/IdempotencyConflict'
default:
$ref: '#/components/responses/Error'
/v1/source-credentials/{credential_id}/revoke:
parameters:
- $ref: '#/components/parameters/CredentialId'
post:
operationId: revokeSourceCredential
parameters:
- $ref: '#/components/parameters/IdempotencyKey'
responses:
'200':
description: Credential revoked for future grants, or the original response replayed.
headers:
Idempotency-Expires-At:
$ref: '#/components/headers/IdempotencyExpiresAt'
content:
application/json:
schema:
$ref: '#/components/schemas/SourceCredential'
'404':
$ref: '#/components/responses/NotFound'
'409':
$ref: '#/components/responses/IdempotencyConflict'
default:
$ref: '#/components/responses/Error'
/v1/deletions/{operation_id}: /v1/deletions/{operation_id}:
parameters: parameters:
- name: operation_id - name: operation_id
@ -308,11 +415,13 @@ paths:
$ref: '#/components/schemas/ClaimRequest' $ref: '#/components/schemas/ClaimRequest'
responses: responses:
'200': '200':
description: Allocated authority or replay of the same claim operation. description: Reserved and durably authorized attempt, or replay of the same claim operation. The authority is non-executable until activateAttempt succeeds.
content: content:
application/json: application/json:
schema: schema:
$ref: '#/components/schemas/ClaimResponse' $ref: '#/components/schemas/ClaimResponse'
'409':
$ref: '#/components/responses/IdempotencyConflict'
default: default:
$ref: '#/components/responses/Error' $ref: '#/components/responses/Error'
/internal/v1/attempts/{attempt_id}/renew: /internal/v1/attempts/{attempt_id}/renew:
@ -335,6 +444,70 @@ paths:
$ref: '#/components/responses/PreconditionFailed' $ref: '#/components/responses/PreconditionFailed'
default: default:
$ref: '#/components/responses/Error' $ref: '#/components/responses/Error'
/internal/v1/attempts/{attempt_id}/activate:
parameters:
- $ref: '#/components/parameters/AttemptId'
post:
operationId: activateAttempt
security:
- serviceAuth: []
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/AttemptActivateRequest'
responses:
'200':
description: Exact authority activated, or a prior successful activation replayed while its lease and durable executing tuple still agree.
content:
application/json:
schema:
$ref: '#/components/schemas/ActivationResponse'
'409':
$ref: '#/components/responses/PreconditionFailed'
default:
$ref: '#/components/responses/Error'
/internal/v1/attempts/{attempt_id}/resolved-source:
parameters:
- $ref: '#/components/parameters/AttemptId'
post:
operationId: pinAttemptResolvedSource
security:
- serviceAuth: []
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/AttemptResolvedSourceRequest'
responses:
'204':
description: Resolved source pinned once, or the exact operation replayed.
'409':
$ref: '#/components/responses/PreconditionFailed'
default:
$ref: '#/components/responses/Error'
/internal/v1/attempts/{attempt_id}/checkpoint:
parameters:
- $ref: '#/components/parameters/AttemptId'
post:
operationId: persistAttemptCheckpoint
security:
- serviceAuth: []
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/AttemptCheckpointRequest'
responses:
'204':
description: Validated checkpoint persisted, or the exact operation replayed.
'409':
$ref: '#/components/responses/PreconditionFailed'
default:
$ref: '#/components/responses/Error'
/internal/v1/attempts/{attempt_id}/progress: /internal/v1/attempts/{attempt_id}/progress:
parameters: parameters:
- $ref: '#/components/parameters/AttemptId' - $ref: '#/components/parameters/AttemptId'
@ -422,6 +595,12 @@ components:
required: true required: true
schema: schema:
$ref: '#/components/schemas/OpaqueId' $ref: '#/components/schemas/OpaqueId'
CredentialId:
name: credential_id
in: path
required: true
schema:
$ref: '#/components/schemas/CredentialId'
headers: headers:
IdempotencyExpiresAt: IdempotencyExpiresAt:
description: Earliest time at which the server may forget this key. description: Earliest time at which the server may forget this key.
@ -461,6 +640,14 @@ components:
description: Internal stable principal identifier mapped to one exact OIDC issuer and subject pair. description: Internal stable principal identifier mapped to one exact OIDC issuer and subject pair.
allOf: allOf:
- $ref: '#/components/schemas/OpaqueId' - $ref: '#/components/schemas/OpaqueId'
CredentialId:
description: Owner-authorized source credential registry identifier.
allOf:
- $ref: '#/components/schemas/OpaqueId'
CredentialVersionId:
description: Immutable application credential-version identifier; backend secret-store coordinates are never exposed.
allOf:
- $ref: '#/components/schemas/OpaqueId'
Target: Target:
type: object type: object
additionalProperties: false additionalProperties: false
@ -506,10 +693,15 @@ components:
minLength: 1 minLength: 1
maxLength: 1024 maxLength: 1024
description: Exact attempt-scoped internal prefix selected by the service. description: Exact attempt-scoped internal prefix selected by the service.
HttpSource: ResultPath:
type: string
minLength: 1
maxLength: 4096
description: NFC-normalized relative POSIX path validated under architecture.md.
HttpSourceSubmission:
type: object type: object
additionalProperties: false additionalProperties: false
required: [kind, url] required: [kind, url, result_path]
properties: properties:
kind: kind:
const: http const: http
@ -517,13 +709,16 @@ components:
type: string type: string
format: uri format: uri
maxLength: 8192 maxLength: 8192
credential_ref: writeOnly: true
type: string description: Sensitive execution input. Public projections expose only display_url.
maxLength: 512 result_path:
ObjectStoreSource: $ref: '#/components/schemas/ResultPath'
credential_id:
$ref: '#/components/schemas/CredentialId'
ObjectStoreSourceSubmission:
type: object type: object
additionalProperties: false additionalProperties: false
required: [kind, store, bucket, key] required: [kind, store, bucket, key, result_path]
properties: properties:
kind: kind:
const: object_store const: object_store
@ -542,10 +737,33 @@ components:
version_id: version_id:
type: string type: string
maxLength: 1024 maxLength: 1024
credential_ref: result_path:
type: string $ref: '#/components/schemas/ResultPath'
maxLength: 512 credential_id:
HuggingFaceSource: $ref: '#/components/schemas/CredentialId'
HuggingFaceSelection:
type: object
additionalProperties: false
minProperties: 1
anyOf:
- required: [paths]
- required: [prefixes]
properties:
paths:
type: array
minItems: 1
maxItems: 1024
uniqueItems: true
items:
$ref: '#/components/schemas/ResultPath'
prefixes:
type: array
minItems: 1
maxItems: 1024
uniqueItems: true
items:
$ref: '#/components/schemas/ResultPath'
HuggingFaceSourceSubmission:
type: object type: object
additionalProperties: false additionalProperties: false
required: [kind, repository_type, repository_id, revision] required: [kind, repository_type, repository_id, revision]
@ -562,35 +780,102 @@ components:
type: string type: string
minLength: 1 minLength: 1
maxLength: 256 maxLength: 256
include: selection:
type: array $ref: '#/components/schemas/HuggingFaceSelection'
maxItems: 1024 credential_id:
uniqueItems: true $ref: '#/components/schemas/CredentialId'
items: SourceSubmission:
type: string
minLength: 1
maxLength: 1024
credential_ref:
type: string
maxLength: 512
Source:
oneOf: oneOf:
- $ref: '#/components/schemas/HttpSource' - $ref: '#/components/schemas/HttpSourceSubmission'
- $ref: '#/components/schemas/ObjectStoreSource' - $ref: '#/components/schemas/ObjectStoreSourceSubmission'
- $ref: '#/components/schemas/HuggingFaceSource' - $ref: '#/components/schemas/HuggingFaceSourceSubmission'
discriminator: discriminator:
propertyName: kind propertyName: kind
mapping: mapping:
http: '#/components/schemas/HttpSource' http: '#/components/schemas/HttpSourceSubmission'
object_store: '#/components/schemas/ObjectStoreSource' object_store: '#/components/schemas/ObjectStoreSourceSubmission'
hugging_face: '#/components/schemas/HuggingFaceSource' hugging_face: '#/components/schemas/HuggingFaceSourceSubmission'
HttpSourceProjection:
type: object
additionalProperties: false
required: [kind, display_url, result_path]
properties:
kind:
const: http
display_url:
type: string
format: uri
maxLength: 8192
description: Sanitized URL without userinfo, sensitive query values, or fragments.
result_path:
$ref: '#/components/schemas/ResultPath'
credential_id:
$ref: '#/components/schemas/CredentialId'
ObjectStoreSourceProjection:
type: object
additionalProperties: false
required: [kind, store, bucket, key, result_path]
properties:
kind:
const: object_store
store:
type: string
minLength: 1
maxLength: 128
bucket:
type: string
minLength: 3
maxLength: 63
key:
type: string
minLength: 1
maxLength: 4096
version_id:
type: string
maxLength: 1024
result_path:
$ref: '#/components/schemas/ResultPath'
credential_id:
$ref: '#/components/schemas/CredentialId'
HuggingFaceSourceProjection:
type: object
additionalProperties: false
required: [kind, repository_type, repository_id, revision]
properties:
kind:
const: hugging_face
repository_type:
type: string
enum: [model, dataset, space]
repository_id:
type: string
pattern: '^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$'
revision:
type: string
minLength: 1
maxLength: 256
selection:
$ref: '#/components/schemas/HuggingFaceSelection'
credential_id:
$ref: '#/components/schemas/CredentialId'
SourceProjection:
oneOf:
- $ref: '#/components/schemas/HttpSourceProjection'
- $ref: '#/components/schemas/ObjectStoreSourceProjection'
- $ref: '#/components/schemas/HuggingFaceSourceProjection'
discriminator:
propertyName: kind
mapping:
http: '#/components/schemas/HttpSourceProjection'
object_store: '#/components/schemas/ObjectStoreSourceProjection'
hugging_face: '#/components/schemas/HuggingFaceSourceProjection'
JobSubmission: JobSubmission:
type: object type: object
additionalProperties: false additionalProperties: false
required: [source] required: [source]
properties: properties:
source: source:
$ref: '#/components/schemas/Source' $ref: '#/components/schemas/SourceSubmission'
target: target:
description: Fields omitted here are taken from the configured default target before authorization. description: Fields omitted here are taken from the configured default target before authorization.
$ref: '#/components/schemas/TargetOverride' $ref: '#/components/schemas/TargetOverride'
@ -598,6 +883,97 @@ components:
type: string type: string
minLength: 1 minLength: 1
maxLength: 200 maxLength: 200
SourceCredentialKind:
type: string
enum: [basic_auth, bearer_token, http_headers, object_store, hugging_face]
CredentialMaterial:
type: object
minProperties: 1
maxProperties: 64
additionalProperties:
type: string
minLength: 1
maxLength: 16384
writeOnly: true
description: Write-only adapter credential fields. Values are never returned or logged.
SourceCredentialCreate:
type: object
additionalProperties: false
required: [label, kind, material]
properties:
label:
type: string
minLength: 1
maxLength: 200
kind:
$ref: '#/components/schemas/SourceCredentialKind'
material:
$ref: '#/components/schemas/CredentialMaterial'
SourceCredentialRotate:
type: object
additionalProperties: false
required: [material]
properties:
material:
$ref: '#/components/schemas/CredentialMaterial'
SourceCredentialVersion:
type: object
additionalProperties: false
required: [id, created_at]
properties:
id:
$ref: '#/components/schemas/CredentialVersionId'
created_at:
type: string
format: date-time
CredentialGrantVersion:
type: object
additionalProperties: false
required: [credential_id, version_id]
properties:
credential_id:
$ref: '#/components/schemas/CredentialId'
version_id:
$ref: '#/components/schemas/CredentialVersionId'
SourceCredential:
type: object
additionalProperties: false
required: [id, owner, label, kind, status, active_version, created_at, updated_at]
properties:
id:
$ref: '#/components/schemas/CredentialId'
owner:
$ref: '#/components/schemas/PrincipalId'
label:
type: string
minLength: 1
maxLength: 200
kind:
$ref: '#/components/schemas/SourceCredentialKind'
status:
type: string
enum: [active, revoked]
active_version:
oneOf:
- $ref: '#/components/schemas/SourceCredentialVersion'
- type: 'null'
created_at:
type: string
format: date-time
updated_at:
type: string
format: date-time
SourceCredentialPage:
type: object
additionalProperties: false
required: [items, next_cursor]
properties:
items:
type: array
items:
$ref: '#/components/schemas/SourceCredential'
next_cursor:
type: [string, 'null']
JobPhase: JobPhase:
type: string type: string
enum: [Queued, Running, RetryWait, Cancelling, Succeeded, Failed, Cancelled] enum: [Queued, Running, RetryWait, Cancelling, Succeeded, Failed, Cancelled]
@ -802,7 +1178,7 @@ components:
lifecycle: lifecycle:
$ref: '#/components/schemas/JobLifecycle' $ref: '#/components/schemas/JobLifecycle'
source: source:
$ref: '#/components/schemas/Source' $ref: '#/components/schemas/SourceProjection'
target: target:
$ref: '#/components/schemas/Target' $ref: '#/components/schemas/Target'
authoritative_attempt: authoritative_attempt:
@ -978,7 +1354,7 @@ components:
ExecutionEnvelope: ExecutionEnvelope:
type: object type: object
additionalProperties: false additionalProperties: false
required: [id, job_id, attempt_id, fence, worker_incarnation, source, resolved_source_ref, target, staging_target, execution_grant_ref, checkpoint_ref] required: [id, job_id, attempt_id, fence, worker_incarnation, source, sensitive_source_ref, credential_version, resolved_source_ref, target, staging_target, execution_grant_ref, checkpoint_ref]
properties: properties:
id: id:
$ref: '#/components/schemas/OpaqueId' $ref: '#/components/schemas/OpaqueId'
@ -992,7 +1368,14 @@ components:
worker_incarnation: worker_incarnation:
$ref: '#/components/schemas/OpaqueId' $ref: '#/components/schemas/OpaqueId'
source: source:
$ref: '#/components/schemas/Source' $ref: '#/components/schemas/SourceProjection'
sensitive_source_ref:
description: Opaque grant-bound reference to the complete accepted source input; credential_version identifies any pinned credential separately.
$ref: '#/components/schemas/OpaqueId'
credential_version:
oneOf:
- $ref: '#/components/schemas/CredentialGrantVersion'
- type: 'null'
resolved_source_ref: resolved_source_ref:
oneOf: oneOf:
- $ref: '#/components/schemas/OpaqueId' - $ref: '#/components/schemas/OpaqueId'
@ -1016,6 +1399,15 @@ components:
$ref: '#/components/schemas/AttemptAuthority' $ref: '#/components/schemas/AttemptAuthority'
execution: execution:
$ref: '#/components/schemas/ExecutionEnvelope' $ref: '#/components/schemas/ExecutionEnvelope'
ActivationResponse:
type: object
additionalProperties: false
required: [authority, execution]
properties:
authority:
$ref: '#/components/schemas/AttemptAuthority'
execution:
$ref: '#/components/schemas/ExecutionEnvelope'
AttemptMutation: AttemptMutation:
type: object type: object
required: [operation_id, fence, worker_incarnation] required: [operation_id, fence, worker_incarnation]
@ -1031,6 +1423,28 @@ components:
unevaluatedProperties: false unevaluatedProperties: false
allOf: allOf:
- $ref: '#/components/schemas/AttemptMutation' - $ref: '#/components/schemas/AttemptMutation'
AttemptActivateRequest:
unevaluatedProperties: false
allOf:
- $ref: '#/components/schemas/AttemptMutation'
AttemptResolvedSourceRequest:
unevaluatedProperties: false
allOf:
- $ref: '#/components/schemas/AttemptMutation'
- type: object
required: [resolved_source_ref]
properties:
resolved_source_ref:
$ref: '#/components/schemas/OpaqueId'
AttemptCheckpointRequest:
unevaluatedProperties: false
allOf:
- $ref: '#/components/schemas/AttemptMutation'
- type: object
required: [checkpoint_ref]
properties:
checkpoint_ref:
$ref: '#/components/schemas/OpaqueId'
AttemptProgressRequest: AttemptProgressRequest:
unevaluatedProperties: false unevaluatedProperties: false
allOf: allOf:

2
docs/protocol/Remora.cfg
View file

@ -14,6 +14,7 @@ CONSTANTS
MaxAttempts = 2 MaxAttempts = 2
MaxProgress = 1 MaxProgress = 1
LeaseLimit = 1 LeaseLimit = 1
ReservationLimit = 1
LocalLimit = 1 LocalLimit = 1
GlobalLimit = 1 GlobalLimit = 1
FaultsEnabled = TRUE FaultsEnabled = TRUE
@ -42,6 +43,7 @@ PROPERTIES
FailureDispositionHonored FailureDispositionHonored
EventuallyDispatched EventuallyDispatched
EventuallyReaped EventuallyReaped
EventuallyAuthorizedResolved
EventuallyTerminal EventuallyTerminal
EventuallyNonWinnerStagingCleaned EventuallyNonWinnerStagingCleaned
EventuallySystemQuiescent EventuallySystemQuiescent

63
docs/protocol/Remora.tla
View file

@ -29,7 +29,7 @@ EXTENDS Naturals, FiniteSets, TLC
CONSTANTS CONSTANTS
Jobs, Workers, InitialWorkers, NoJob, NoWorker, Jobs, Workers, InitialWorkers, NoJob, NoWorker,
MaxAttempts, MaxProgress, LeaseLimit, MaxAttempts, MaxProgress, LeaseLimit, ReservationLimit,
LocalLimit, GlobalLimit, FaultsEnabled, FaultServices, LocalLimit, GlobalLimit, FaultsEnabled, FaultServices,
MaxCoordLosses, MaxPartitions MaxCoordLosses, MaxPartitions
@ -40,6 +40,7 @@ ASSUME /\ Jobs # {}
/\ MaxAttempts \in Nat \ {0} /\ MaxAttempts \in Nat \ {0}
/\ MaxProgress \in Nat \ {0} /\ MaxProgress \in Nat \ {0}
/\ LeaseLimit \in Nat \ {0} /\ LeaseLimit \in Nat \ {0}
/\ ReservationLimit \in Nat \ {0}
/\ LocalLimit \in Nat \ {0} /\ LocalLimit \in Nat \ {0}
/\ GlobalLimit \in Nat \ {0} /\ GlobalLimit \in Nat \ {0}
/\ FaultsEnabled \in BOOLEAN /\ FaultsEnabled \in BOOLEAN
@ -119,7 +120,7 @@ VARIABLES
checkpointValid, checkpointValid,
\* CoordinationStore (reconstructible). \* CoordinationStore (reconstructible).
queued, reservation, leaseState, leaseAge, cancelNotice, queued, reservation, reservationAge, leaseState, leaseAge, cancelNotice,
liveProgress, liveProgressAttempt, coordLossCount, liveProgress, liveProgressAttempt, coordLossCount,
\* Worker incarnations, per-service links, partition budget. \* Worker incarnations, per-service links, partition budget.
@ -133,7 +134,7 @@ durableVars ==
<<phase, cancelRequested, currentToken, authority, dispatchNeeded, <<phase, cancelRequested, currentToken, authority, dispatchNeeded,
committedAttempt, checkpoint, checkpointValid>> committedAttempt, checkpoint, checkpointValid>>
coordinationVars == coordinationVars ==
<<queued, reservation, leaseState, leaseAge, cancelNotice, <<queued, reservation, reservationAge, leaseState, leaseAge, cancelNotice,
liveProgress, liveProgressAttempt, coordLossCount>> liveProgress, liveProgressAttempt, coordLossCount>>
workerVars == <<workerState, links, partitionCount>> workerVars == <<workerState, links, partitionCount>>
attemptVars == attemptVars ==
@ -168,6 +169,7 @@ OtherCapacity(w) ==
CoordinationClean(j) == CoordinationClean(j) ==
/\ ~queued[j] /\ ~queued[j]
/\ reservation[j] = NoWorker /\ reservation[j] = NoWorker
/\ reservationAge[j] = 0
/\ ~cancelNotice[j] /\ ~cancelNotice[j]
/\ liveProgress[j] = 0 /\ liveProgress[j] = 0
/\ liveProgressAttempt[j] = NoAttempt /\ liveProgressAttempt[j] = NoAttempt
@ -195,6 +197,7 @@ Init ==
/\ checkpointValid = [j \in Jobs |-> FALSE] /\ checkpointValid = [j \in Jobs |-> FALSE]
/\ queued = [j \in Jobs |-> FALSE] /\ queued = [j \in Jobs |-> FALSE]
/\ reservation = [j \in Jobs |-> NoWorker] /\ reservation = [j \in Jobs |-> NoWorker]
/\ reservationAge = [j \in Jobs |-> 0]
/\ leaseState = [a \in AttemptIds |-> NoLease] /\ leaseState = [a \in AttemptIds |-> NoLease]
/\ leaseAge = [a \in AttemptIds |-> 0] /\ leaseAge = [a \in AttemptIds |-> 0]
/\ cancelNotice = [j \in Jobs |-> FALSE] /\ cancelNotice = [j \in Jobs |-> FALSE]
@ -222,7 +225,7 @@ Enqueue(j) ==
/\ reservation[j] = NoWorker /\ reservation[j] = NoWorker
/\ queued' = [queued EXCEPT ![j] = TRUE] /\ queued' = [queued EXCEPT ![j] = TRUE]
/\ UNCHANGED durableVars /\ UNCHANGED durableVars
/\ UNCHANGED <<reservation, leaseState, leaseAge, cancelNotice, /\ UNCHANGED <<reservation, reservationAge, leaseState, leaseAge, cancelNotice,
liveProgress, liveProgressAttempt, coordLossCount>> liveProgress, liveProgressAttempt, coordLossCount>>
/\ UNCHANGED workerVars /\ UNCHANGED workerVars
/\ UNCHANGED attemptVars /\ UNCHANGED attemptVars
@ -235,6 +238,7 @@ Reserve(j, w) ==
/\ CapacityAvailable(w) /\ CapacityAvailable(w)
/\ queued' = [queued EXCEPT ![j] = FALSE] /\ queued' = [queued EXCEPT ![j] = FALSE]
/\ reservation' = [reservation EXCEPT ![j] = w] /\ reservation' = [reservation EXCEPT ![j] = w]
/\ reservationAge' = [reservationAge EXCEPT ![j] = 0]
/\ UNCHANGED durableVars /\ UNCHANGED durableVars
/\ UNCHANGED <<leaseState, leaseAge, cancelNotice, liveProgress, /\ UNCHANGED <<leaseState, leaseAge, cancelNotice, liveProgress,
liveProgressAttempt, coordLossCount>> liveProgressAttempt, coordLossCount>>
@ -289,6 +293,7 @@ ActivateAttempt(a) ==
/\ workerState[w] = WorkerUp /\ workerState[w] = WorkerUp
/\ CanReach(w, CoordSvc) /\ CanReach(w, CoordSvc)
/\ reservation' = [reservation EXCEPT ![j] = NoWorker] /\ reservation' = [reservation EXCEPT ![j] = NoWorker]
/\ reservationAge' = [reservationAge EXCEPT ![j] = 0]
/\ leaseState' = [leaseState EXCEPT ![a] = LeaseLive] /\ leaseState' = [leaseState EXCEPT ![a] = LeaseLive]
/\ leaseAge' = [leaseAge EXCEPT ![a] = 0] /\ leaseAge' = [leaseAge EXCEPT ![a] = 0]
/\ liveProgress' = [liveProgress EXCEPT ![j] = attemptProgress[a]] /\ liveProgress' = [liveProgress EXCEPT ![j] = attemptProgress[a]]
@ -309,24 +314,44 @@ RenewLease(a) ==
/\ CanReach(w, CoordSvc) /\ CanReach(w, CoordSvc)
/\ leaseAge' = [leaseAge EXCEPT ![a] = 0] /\ leaseAge' = [leaseAge EXCEPT ![a] = 0]
/\ UNCHANGED durableVars /\ UNCHANGED durableVars
/\ UNCHANGED <<queued, reservation, leaseState, cancelNotice, /\ UNCHANGED <<queued, reservation, reservationAge, leaseState, cancelNotice,
liveProgress, liveProgressAttempt, coordLossCount>> liveProgress, liveProgressAttempt, coordLossCount>>
/\ UNCHANGED workerVars /\ UNCHANGED workerVars
/\ UNCHANGED attemptVars /\ UNCHANGED attemptVars
Tick == Tick ==
/\ \E a \in AttemptIds : /\ (\E a \in AttemptIds :
leaseState[a] = LeaseLive /\ leaseAge[a] < LeaseLimit leaseState[a] = LeaseLive /\ leaseAge[a] < LeaseLimit)
\/ (\E j \in Jobs :
reservation[j] # NoWorker /\ reservationAge[j] < ReservationLimit)
/\ leaseAge' = /\ leaseAge' =
[a \in AttemptIds |-> [a \in AttemptIds |->
IF leaseState[a] = LeaseLive /\ leaseAge[a] < LeaseLimit IF leaseState[a] = LeaseLive /\ leaseAge[a] < LeaseLimit
THEN leaseAge[a] + 1 ELSE leaseAge[a]] THEN leaseAge[a] + 1 ELSE leaseAge[a]]
/\ reservationAge' =
[j \in Jobs |->
IF reservation[j] # NoWorker
/\ reservationAge[j] < ReservationLimit
THEN reservationAge[j] + 1 ELSE reservationAge[j]]
/\ UNCHANGED durableVars /\ UNCHANGED durableVars
/\ UNCHANGED <<queued, reservation, leaseState, cancelNotice, /\ UNCHANGED <<queued, reservation, leaseState, cancelNotice,
liveProgress, liveProgressAttempt, coordLossCount>> liveProgress, liveProgressAttempt, coordLossCount>>
/\ UNCHANGED workerVars /\ UNCHANGED workerVars
/\ UNCHANGED attemptVars /\ UNCHANGED attemptVars
\* Reservation expiry is coordination-only. Queued jobs become eligible for
\* Enqueue; authorized attempts become eligible for ReapLostAttempt.
ExpireReservation(j) ==
/\ reservation[j] # NoWorker
/\ reservationAge[j] = ReservationLimit
/\ reservation' = [reservation EXCEPT ![j] = NoWorker]
/\ reservationAge' = [reservationAge EXCEPT ![j] = 0]
/\ UNCHANGED durableVars
/\ UNCHANGED <<queued, leaseState, leaseAge, cancelNotice,
liveProgress, liveProgressAttempt, coordLossCount>>
/\ UNCHANGED workerVars
/\ UNCHANGED attemptVars
\* Lease expiry does not clear live progress: product may retain a stale \* Lease expiry does not clear live progress: product may retain a stale
\* (attempt, bytes) hint until overwrite, reader discard, or terminal GC. \* (attempt, bytes) hint until overwrite, reader discard, or terminal GC.
ExpireLease(a) == ExpireLease(a) ==
@ -335,7 +360,7 @@ ExpireLease(a) ==
/\ leaseState' = [leaseState EXCEPT ![a] = LeaseExpired] /\ leaseState' = [leaseState EXCEPT ![a] = LeaseExpired]
/\ leaseAge' = [leaseAge EXCEPT ![a] = 0] /\ leaseAge' = [leaseAge EXCEPT ![a] = 0]
/\ UNCHANGED durableVars /\ UNCHANGED durableVars
/\ UNCHANGED <<queued, reservation, cancelNotice, liveProgress, /\ UNCHANGED <<queued, reservation, reservationAge, cancelNotice, liveProgress,
liveProgressAttempt, coordLossCount>> liveProgressAttempt, coordLossCount>>
/\ UNCHANGED workerVars /\ UNCHANGED workerVars
/\ UNCHANGED attemptVars /\ UNCHANGED attemptVars
@ -347,6 +372,7 @@ LoseCoordinationState ==
/\ coordLossCount < MaxCoordLosses /\ coordLossCount < MaxCoordLosses
/\ queued' = [j \in Jobs |-> FALSE] /\ queued' = [j \in Jobs |-> FALSE]
/\ reservation' = [j \in Jobs |-> NoWorker] /\ reservation' = [j \in Jobs |-> NoWorker]
/\ reservationAge' = [j \in Jobs |-> 0]
/\ leaseState' = /\ leaseState' =
[a \in AttemptIds |-> [a \in AttemptIds |->
IF leaseState[a] = LeaseLive THEN LeaseExpired IF leaseState[a] = LeaseLive THEN LeaseExpired
@ -419,6 +445,7 @@ ReleaseReservation(j) ==
\/ (authority[j] # NoAttempt \/ (authority[j] # NoAttempt
/\ attemptState[authority[j]] # AttemptAuthorized)) /\ attemptState[authority[j]] # AttemptAuthorized))
/\ reservation' = [reservation EXCEPT ![j] = NoWorker] /\ reservation' = [reservation EXCEPT ![j] = NoWorker]
/\ reservationAge' = [reservationAge EXCEPT ![j] = 0]
/\ UNCHANGED durableVars /\ UNCHANGED durableVars
/\ UNCHANGED <<queued, leaseState, leaseAge, cancelNotice, /\ UNCHANGED <<queued, leaseState, leaseAge, cancelNotice,
liveProgress, liveProgressAttempt, coordLossCount>> liveProgress, liveProgressAttempt, coordLossCount>>
@ -502,7 +529,7 @@ PublishCancelNotice(j) ==
/\ ~cancelNotice[j] /\ ~cancelNotice[j]
/\ cancelNotice' = [cancelNotice EXCEPT ![j] = TRUE] /\ cancelNotice' = [cancelNotice EXCEPT ![j] = TRUE]
/\ UNCHANGED durableVars /\ UNCHANGED durableVars
/\ UNCHANGED <<queued, reservation, leaseState, leaseAge, liveProgress, /\ UNCHANGED <<queued, reservation, reservationAge, leaseState, leaseAge, liveProgress,
liveProgressAttempt, coordLossCount>> liveProgressAttempt, coordLossCount>>
/\ UNCHANGED workerVars /\ UNCHANGED workerVars
/\ UNCHANGED attemptVars /\ UNCHANGED attemptVars
@ -531,7 +558,7 @@ ReleaseLease(a) ==
/\ leaseState' = [leaseState EXCEPT ![a] = LeaseExpired] /\ leaseState' = [leaseState EXCEPT ![a] = LeaseExpired]
/\ leaseAge' = [leaseAge EXCEPT ![a] = 0] /\ leaseAge' = [leaseAge EXCEPT ![a] = 0]
/\ UNCHANGED durableVars /\ UNCHANGED durableVars
/\ UNCHANGED <<queued, reservation, cancelNotice, liveProgress, /\ UNCHANGED <<queued, reservation, reservationAge, cancelNotice, liveProgress,
liveProgressAttempt, coordLossCount>> liveProgressAttempt, coordLossCount>>
/\ UNCHANGED workerVars /\ UNCHANGED workerVars
/\ UNCHANGED attemptVars /\ UNCHANGED attemptVars
@ -546,7 +573,7 @@ ClearStaleLiveProgress(j) ==
/\ liveProgressAttempt' = /\ liveProgressAttempt' =
[liveProgressAttempt EXCEPT ![j] = NoAttempt] [liveProgressAttempt EXCEPT ![j] = NoAttempt]
/\ UNCHANGED durableVars /\ UNCHANGED durableVars
/\ UNCHANGED <<queued, reservation, leaseState, leaseAge, /\ UNCHANGED <<queued, reservation, reservationAge, leaseState, leaseAge,
cancelNotice, coordLossCount>> cancelNotice, coordLossCount>>
/\ UNCHANGED workerVars /\ UNCHANGED workerVars
/\ UNCHANGED attemptVars /\ UNCHANGED attemptVars
@ -557,6 +584,7 @@ CleanTerminalCoordination(j) ==
\/ liveProgress[j] # 0 \/ liveProgressAttempt[j] # NoAttempt) \/ liveProgress[j] # 0 \/ liveProgressAttempt[j] # NoAttempt)
/\ queued' = [queued EXCEPT ![j] = FALSE] /\ queued' = [queued EXCEPT ![j] = FALSE]
/\ reservation' = [reservation EXCEPT ![j] = NoWorker] /\ reservation' = [reservation EXCEPT ![j] = NoWorker]
/\ reservationAge' = [reservationAge EXCEPT ![j] = 0]
/\ cancelNotice' = [cancelNotice EXCEPT ![j] = FALSE] /\ cancelNotice' = [cancelNotice EXCEPT ![j] = FALSE]
/\ liveProgress' = [liveProgress EXCEPT ![j] = 0] /\ liveProgress' = [liveProgress EXCEPT ![j] = 0]
/\ liveProgressAttempt' = /\ liveProgressAttempt' =
@ -628,7 +656,7 @@ ReportProgress(a) ==
/\ liveProgressAttempt' = /\ liveProgressAttempt' =
[liveProgressAttempt EXCEPT ![j] = a] [liveProgressAttempt EXCEPT ![j] = a]
/\ UNCHANGED durableVars /\ UNCHANGED durableVars
/\ UNCHANGED <<queued, reservation, leaseState, leaseAge, /\ UNCHANGED <<queued, reservation, reservationAge, leaseState, leaseAge,
cancelNotice, coordLossCount>> cancelNotice, coordLossCount>>
/\ UNCHANGED workerVars /\ UNCHANGED workerVars
/\ UNCHANGED attemptVars /\ UNCHANGED attemptVars
@ -774,6 +802,7 @@ Next ==
\/ \E a \in AttemptIds : RenewLease(a) \/ \E a \in AttemptIds : RenewLease(a)
\/ Tick \/ Tick
\/ \E a \in AttemptIds : ExpireLease(a) \/ \E a \in AttemptIds : ExpireLease(a)
\/ \E j \in Jobs : ExpireReservation(j)
\/ LoseCoordinationState \/ LoseCoordinationState
\/ \E w \in Workers, s \in Services : DropLink(w, s) \/ \E w \in Workers, s \in Services : DropLink(w, s)
\/ \E w \in Workers, s \in Services : RestoreLink(w, s) \/ \E w \in Workers, s \in Services : RestoreLink(w, s)
@ -812,6 +841,7 @@ Fairness ==
/\ \A j \in Jobs : SF_vars(ExecuteJob(j)) /\ \A j \in Jobs : SF_vars(ExecuteJob(j))
/\ \A a \in AttemptIds : WF_vars(ProcessOutcome(a)) /\ \A a \in AttemptIds : WF_vars(ProcessOutcome(a))
/\ \A a \in AttemptIds : WF_vars(ExpireLease(a)) /\ \A a \in AttemptIds : WF_vars(ExpireLease(a))
/\ \A j \in Jobs : WF_vars(ExpireReservation(j))
/\ \A a \in AttemptIds : WF_vars(CooperativeStop(a)) /\ \A a \in AttemptIds : WF_vars(CooperativeStop(a))
/\ \A a \in AttemptIds : WF_vars(ReleaseLease(a)) /\ \A a \in AttemptIds : WF_vars(ReleaseLease(a))
/\ \A a \in AttemptIds : WF_vars(CleanupStage(a)) /\ \A a \in AttemptIds : WF_vars(CleanupStage(a))
@ -840,6 +870,7 @@ TypeOK ==
/\ checkpointValid \in [Jobs -> BOOLEAN] /\ checkpointValid \in [Jobs -> BOOLEAN]
/\ queued \in [Jobs -> BOOLEAN] /\ queued \in [Jobs -> BOOLEAN]
/\ reservation \in [Jobs -> Workers \cup {NoWorker}] /\ reservation \in [Jobs -> Workers \cup {NoWorker}]
/\ reservationAge \in [Jobs -> 0..ReservationLimit]
/\ leaseState \in [AttemptIds -> LeaseStates] /\ leaseState \in [AttemptIds -> LeaseStates]
/\ leaseAge \in [AttemptIds -> 0..LeaseLimit] /\ leaseAge \in [AttemptIds -> 0..LeaseLimit]
/\ cancelNotice \in [Jobs -> BOOLEAN] /\ cancelNotice \in [Jobs -> BOOLEAN]
@ -874,6 +905,7 @@ StateConsistency ==
/\ (committedAttempt[j] # NoAttempt <=> phase[j] = Succeeded) /\ (committedAttempt[j] # NoAttempt <=> phase[j] = Succeeded)
/\ (phase[j] = Queued => dispatchNeeded[j]) /\ (phase[j] = Queued => dispatchNeeded[j])
/\ (reservation[j] # NoWorker => ~queued[j]) /\ (reservation[j] # NoWorker => ~queued[j])
/\ (reservation[j] = NoWorker => reservationAge[j] = 0)
/\ \A a \in AttemptIds : /\ \A a \in AttemptIds :
/\ (attemptState[a] = AttemptUnused <=> /\ (attemptState[a] = AttemptUnused <=>
attemptWorker[a] = NoWorker) attemptWorker[a] = NoWorker)
@ -1024,6 +1056,13 @@ EventuallyReaped ==
\A a \in AttemptIds : \A a \in AttemptIds :
(CurrentAuthority(a) /\ leaseState[a] # LeaseLive) (CurrentAuthority(a) /\ leaseState[a] # LeaseLive)
~> (~CurrentAuthority(a) \/ Terminal(JobOf(a))) ~> (~CurrentAuthority(a) \/ Terminal(JobOf(a)))
EventuallyAuthorizedResolved ==
RecoveryAssumption =>
\A a \in AttemptIds :
(CurrentAuthority(a) /\ attemptState[a] = AttemptAuthorized)
~> (attemptState[a] = AttemptExecuting
\/ ~CurrentAuthority(a)
\/ Terminal(JobOf(a)))
EventuallyTerminal == EventuallyTerminal ==
RecoveryAssumption => \A j \in Jobs : <>Terminal(j) RecoveryAssumption => \A j \in Jobs : <>Terminal(j)

28
docs/protocol/RemoraClaimActivation.cfg Normal file
View file

@ -0,0 +1,28 @@
SPECIFICATION Spec
CHECK_DEADLOCK TRUE
CONSTANTS
Claims = {c1}
Fences = {f1, f2}
Workers = {w1, w2}
Operations = {op1, op2}
NoWorker = NoWorker
NoOwner = NoOwner
NoRequest = NoRequest
NoResponse = NoResponse
Accepted = Accepted
Rejected = Rejected
INVARIANTS
TypeOK
LiveLeaseMatchesAuthority
ExecutingIsCommitted
AcceptedResponseIsExactReplay
ConflictIsChangedRequest
ReapedClaimHasNoActivationEffect
ReapedCoordinationIsClean
PROPERTIES
ActivationAppliesAtMostOnce
OperationRequestIsImmutable

307
docs/protocol/RemoraClaimActivation.tla Normal file
View file

@ -0,0 +1,307 @@
---------------------- MODULE RemoraClaimActivation ----------------------
EXTENDS Naturals
\* Split-store refinement of the atomic JobProtocol ActivateAttempt action.
\* Durable execution is the linearization point; lease creation alone is not authority.
CONSTANTS
Claims, Fences, Workers, Operations,
NoWorker, NoOwner, NoRequest, NoResponse, Accepted, Rejected
RequestKeys == Claims \X Fences \X Workers
ASSUME /\ Claims # {}
/\ Fences # {}
/\ Workers # {}
/\ Operations # {}
/\ NoWorker \notin Workers
/\ NoRequest \notin RequestKeys
Authorized == "Authorized"
Executing == "Executing"
Reaped == "Reaped"
DurableStates == {Authorized, Executing, Reaped}
Idle == "Idle"
Validated == "Validated"
LeaseCreated == "LeaseCreated"
Committed == "Committed"
Finished == "Finished"
ActivationStages == {Idle, Validated, LeaseCreated, Committed, Finished}
Activator == "Activator"
Recovery == "Recovery"
LockOwners == {NoOwner, Activator, Recovery}
Responses == {NoResponse, Accepted, Rejected}
VARIABLES
durableState, attemptFence, attemptWorker, reservation, lease,
activationStage, durableLock, operationRequest, operationResponse,
conflictRequest, activationEffectCount
vars == <<durableState, attemptFence, attemptWorker, reservation, lease,
activationStage, durableLock, operationRequest, operationResponse,
conflictRequest, activationEffectCount>>
Request(c, f, w) == <<c, f, w>>
ExactRequest(c) == Request(c, attemptFence[c], attemptWorker[c])
Init ==
/\ durableState = [c \in Claims |-> Authorized]
/\ attemptFence \in [Claims -> Fences]
/\ attemptWorker \in [Claims -> Workers]
/\ reservation = attemptWorker
/\ lease = [c \in Claims |-> NoWorker]
/\ activationStage = [c \in Claims |-> Idle]
/\ durableLock = [c \in Claims |-> NoOwner]
/\ operationRequest = [op \in Operations |-> NoRequest]
/\ operationResponse = [op \in Operations |-> NoResponse]
/\ conflictRequest = [op \in Operations |-> NoRequest]
/\ activationEffectCount = [c \in Claims |-> 0]
BindOperation(op, c, f, w) ==
/\ operationRequest[op] = NoRequest
/\ operationRequest' =
[operationRequest EXCEPT ![op] = Request(c, f, w)]
/\ UNCHANGED <<durableState, attemptFence, attemptWorker, reservation,
lease, activationStage, durableLock, operationResponse,
conflictRequest, activationEffectCount>>
RejectOperationReuse(op, c, f, w) ==
/\ operationRequest[op] # NoRequest
/\ operationRequest[op] # Request(c, f, w)
/\ conflictRequest' =
[conflictRequest EXCEPT ![op] = Request(c, f, w)]
/\ UNCHANGED <<durableState, attemptFence, attemptWorker, reservation,
lease, activationStage, durableLock, operationRequest,
operationResponse, activationEffectCount>>
StartActivation(op, c, f, w) ==
/\ operationRequest[op] = Request(c, f, w)
/\ operationResponse[op] = NoResponse
/\ Request(c, f, w) = ExactRequest(c)
/\ activationStage[c] = Idle
/\ durableState[c] = Authorized
/\ reservation[c] = w
/\ activationStage' = [activationStage EXCEPT ![c] = Validated]
/\ UNCHANGED <<durableState, attemptFence, attemptWorker, reservation,
lease, durableLock, operationRequest, operationResponse,
conflictRequest, activationEffectCount>>
CreateLease(c) ==
LET w == attemptWorker[c]
IN /\ activationStage[c] = Validated
/\ durableState[c] = Authorized
/\ reservation[c] = w
/\ lease[c] = NoWorker
/\ lease' = [lease EXCEPT ![c] = w]
/\ activationStage' = [activationStage EXCEPT ![c] = LeaseCreated]
/\ UNCHANGED <<durableState, attemptFence, attemptWorker, reservation,
durableLock, operationRequest, operationResponse,
conflictRequest, activationEffectCount>>
AcquireActivationLock(c) ==
/\ activationStage[c] = LeaseCreated
/\ durableState[c] = Authorized
/\ durableLock[c] = NoOwner
/\ durableLock' = [durableLock EXCEPT ![c] = Activator]
/\ UNCHANGED <<durableState, attemptFence, attemptWorker, reservation,
lease, activationStage, operationRequest,
operationResponse, conflictRequest,
activationEffectCount>>
LoseActivatorLock(c) ==
/\ durableLock[c] = Activator
/\ durableLock' = [durableLock EXCEPT ![c] = NoOwner]
/\ UNCHANGED <<durableState, attemptFence, attemptWorker, reservation,
lease, activationStage, operationRequest,
operationResponse, conflictRequest,
activationEffectCount>>
CommitActivation(c) ==
LET w == attemptWorker[c]
IN /\ activationStage[c] = LeaseCreated
/\ durableLock[c] = Activator
/\ durableState[c] = Authorized
/\ reservation[c] = w
/\ lease[c] = w
/\ durableState' = [durableState EXCEPT ![c] = Executing]
/\ reservation' = [reservation EXCEPT ![c] = NoWorker]
/\ activationStage' = [activationStage EXCEPT ![c] = Committed]
/\ durableLock' = [durableLock EXCEPT ![c] = NoOwner]
/\ activationEffectCount' =
[activationEffectCount EXCEPT ![c] = @ + 1]
/\ UNCHANGED <<attemptFence, attemptWorker, lease,
operationRequest, operationResponse, conflictRequest>>
ReplayActivation(op, c, f, w) ==
/\ operationRequest[op] = Request(c, f, w)
/\ Request(c, f, w) = ExactRequest(c)
/\ durableState[c] = Executing
/\ lease[c] = w
/\ activationEffectCount[c] = 1
/\ operationResponse' =
[operationResponse EXCEPT ![op] = Accepted]
/\ activationStage' = [activationStage EXCEPT ![c] = Finished]
/\ UNCHANGED <<durableState, attemptFence, attemptWorker, reservation,
lease, durableLock, operationRequest, conflictRequest,
activationEffectCount>>
RejectPrecondition(op, c, f, w) ==
/\ operationRequest[op] = Request(c, f, w)
/\ operationResponse[op] = NoResponse
/\ \/ Request(c, f, w) # ExactRequest(c)
\/ durableState[c] = Reaped
\/ /\ durableState[c] = Executing
/\ lease[c] # w
/\ operationResponse' =
[operationResponse EXCEPT ![op] = Rejected]
/\ UNCHANGED <<durableState, attemptFence, attemptWorker, reservation,
lease, activationStage, durableLock, operationRequest,
conflictRequest, activationEffectCount>>
ExpireReservation(c) ==
/\ durableState[c] = Authorized
/\ reservation[c] # NoWorker
/\ lease[c] = NoWorker
/\ reservation' = [reservation EXCEPT ![c] = NoWorker]
/\ UNCHANGED <<durableState, attemptFence, attemptWorker, lease,
activationStage, durableLock, operationRequest,
operationResponse, conflictRequest,
activationEffectCount>>
ExpireLease(c) ==
/\ durableState[c] = Executing
/\ lease[c] = attemptWorker[c]
/\ lease' = [lease EXCEPT ![c] = NoWorker]
/\ operationResponse' =
[op \in Operations |->
IF operationResponse[op] = Accepted
/\ operationRequest[op][1] = c
THEN NoResponse ELSE operationResponse[op]]
/\ UNCHANGED <<durableState, attemptFence, attemptWorker, reservation,
activationStage, durableLock, operationRequest,
conflictRequest, activationEffectCount>>
AcquireRecoveryLock(c) ==
/\ durableState[c] = Authorized
/\ durableLock[c] = NoOwner
/\ durableLock' = [durableLock EXCEPT ![c] = Recovery]
/\ UNCHANGED <<durableState, attemptFence, attemptWorker, reservation,
lease, activationStage, operationRequest,
operationResponse, conflictRequest,
activationEffectCount>>
FinishRecovery(c) ==
/\ durableLock[c] = Recovery
/\ durableState[c] = Authorized
/\ durableState' = [durableState EXCEPT ![c] = Reaped]
/\ reservation' = [reservation EXCEPT ![c] = NoWorker]
/\ lease' = [lease EXCEPT ![c] = NoWorker]
/\ durableLock' = [durableLock EXCEPT ![c] = NoOwner]
/\ UNCHANGED <<attemptFence, attemptWorker, activationStage,
operationRequest, operationResponse, conflictRequest,
activationEffectCount>>
CompensateRejectedActivation(c) ==
LET w == attemptWorker[c]
IN /\ durableState[c] = Reaped
/\ activationStage[c] \in {Validated, LeaseCreated}
/\ lease[c] \in {NoWorker, w}
/\ lease' = [lease EXCEPT ![c] = NoWorker]
/\ activationStage' = [activationStage EXCEPT ![c] = Finished]
/\ UNCHANGED <<durableState, attemptFence, attemptWorker, reservation,
durableLock, operationRequest, operationResponse,
conflictRequest, activationEffectCount>>
Done ==
/\ \A c \in Claims : durableState[c] \in {Executing, Reaped}
/\ UNCHANGED vars
Next ==
\/ \E op \in Operations, c \in Claims, f \in Fences, w \in Workers :
BindOperation(op, c, f, w)
\/ \E op \in Operations, c \in Claims, f \in Fences, w \in Workers :
RejectOperationReuse(op, c, f, w)
\/ \E op \in Operations, c \in Claims, f \in Fences, w \in Workers :
StartActivation(op, c, f, w)
\/ \E c \in Claims : CreateLease(c)
\/ \E c \in Claims : AcquireActivationLock(c)
\/ \E c \in Claims : LoseActivatorLock(c)
\/ \E c \in Claims : CommitActivation(c)
\/ \E op \in Operations, c \in Claims, f \in Fences, w \in Workers :
ReplayActivation(op, c, f, w)
\/ \E op \in Operations, c \in Claims, f \in Fences, w \in Workers :
RejectPrecondition(op, c, f, w)
\/ \E c \in Claims : ExpireReservation(c)
\/ \E c \in Claims : ExpireLease(c)
\/ \E c \in Claims : AcquireRecoveryLock(c)
\/ \E c \in Claims : FinishRecovery(c)
\/ \E c \in Claims : CompensateRejectedActivation(c)
\/ Done
Spec == Init /\ [][Next]_vars
TypeOK ==
/\ durableState \in [Claims -> DurableStates]
/\ attemptFence \in [Claims -> Fences]
/\ attemptWorker \in [Claims -> Workers]
/\ reservation \in [Claims -> Workers \cup {NoWorker}]
/\ lease \in [Claims -> Workers \cup {NoWorker}]
/\ activationStage \in [Claims -> ActivationStages]
/\ durableLock \in [Claims -> LockOwners]
/\ operationRequest \in [Operations -> RequestKeys \cup {NoRequest}]
/\ operationResponse \in [Operations -> Responses]
/\ conflictRequest \in [Operations -> RequestKeys \cup {NoRequest}]
/\ activationEffectCount \in [Claims -> 0..1]
LiveLeaseMatchesAuthority ==
\A c \in Claims :
lease[c] # NoWorker => lease[c] = attemptWorker[c]
ExecutingIsCommitted ==
\A c \in Claims :
durableState[c] = Executing =>
/\ reservation[c] = NoWorker
/\ activationStage[c] \in {Committed, Finished}
/\ activationEffectCount[c] = 1
AcceptedResponseIsExactReplay ==
\A op \in Operations :
operationResponse[op] = Accepted =>
\E c \in Claims :
/\ operationRequest[op] = ExactRequest(c)
/\ durableState[c] = Executing
/\ lease[c] = attemptWorker[c]
/\ activationEffectCount[c] = 1
ConflictIsChangedRequest ==
\A op \in Operations :
conflictRequest[op] # NoRequest =>
/\ operationRequest[op] # NoRequest
/\ conflictRequest[op] # operationRequest[op]
ReapedClaimHasNoActivationEffect ==
\A c \in Claims :
durableState[c] = Reaped => activationEffectCount[c] = 0
ReapedCoordinationIsClean ==
\A c \in Claims :
durableState[c] = Reaped =>
/\ reservation[c] = NoWorker
/\ lease[c] = NoWorker
ActivationAppliesAtMostOnceAction ==
\A c \in Claims :
activationEffectCount[c] = 1 => activationEffectCount'[c] = 1
ActivationAppliesAtMostOnce == [][ActivationAppliesAtMostOnceAction]_vars
OperationRequestIsImmutableAction ==
\A op \in Operations :
operationRequest[op] # NoRequest =>
operationRequest'[op] = operationRequest[op]
OperationRequestIsImmutable == [][OperationRequestIsImmutableAction]_vars
=============================================================================

1
docs/protocol/RemoraCombined.cfg
View file

@ -14,6 +14,7 @@ CONSTANTS
MaxAttempts = 1 MaxAttempts = 1
MaxProgress = 1 MaxProgress = 1
LeaseLimit = 1 LeaseLimit = 1
ReservationLimit = 1
LocalLimit = 1 LocalLimit = 1
GlobalLimit = 2 GlobalLimit = 2
FaultsEnabled = FALSE FaultsEnabled = FALSE

1
docs/protocol/RemoraDisposition.cfg
View file

@ -11,6 +11,7 @@ CONSTANTS
MaxAttempts = 2 MaxAttempts = 2
MaxProgress = 1 MaxProgress = 1
LeaseLimit = 1 LeaseLimit = 1
ReservationLimit = 1
LocalLimit = 1 LocalLimit = 1
GlobalLimit = 1 GlobalLimit = 1
FaultsEnabled = FALSE FaultsEnabled = FALSE

1
docs/protocol/RemoraDurableFault.cfg
View file

@ -13,6 +13,7 @@ CONSTANTS
MaxAttempts = 2 MaxAttempts = 2
MaxProgress = 1 MaxProgress = 1
LeaseLimit = 1 LeaseLimit = 1
ReservationLimit = 1
LocalLimit = 1 LocalLimit = 1
GlobalLimit = 1 GlobalLimit = 1
FaultsEnabled = TRUE FaultsEnabled = TRUE

1
docs/protocol/RemoraFencing.cfg
View file

@ -13,6 +13,7 @@ CONSTANTS
MaxAttempts = 3 MaxAttempts = 3
MaxProgress = 1 MaxProgress = 1
LeaseLimit = 1 LeaseLimit = 1
ReservationLimit = 1
LocalLimit = 1 LocalLimit = 1
GlobalLimit = 1 GlobalLimit = 1
FaultsEnabled = TRUE FaultsEnabled = TRUE

1
docs/protocol/RemoraGlobal.cfg
View file

@ -12,6 +12,7 @@ CONSTANTS
MaxAttempts = 1 MaxAttempts = 1
MaxProgress = 1 MaxProgress = 1
LeaseLimit = 1 LeaseLimit = 1
ReservationLimit = 1
LocalLimit = 2 LocalLimit = 2
GlobalLimit = 1 GlobalLimit = 1
FaultsEnabled = FALSE FaultsEnabled = FALSE

1
docs/protocol/RemoraLocal.cfg
View file

@ -12,6 +12,7 @@ CONSTANTS
MaxAttempts = 1 MaxAttempts = 1
MaxProgress = 1 MaxProgress = 1
LeaseLimit = 1 LeaseLimit = 1
ReservationLimit = 1
LocalLimit = 1 LocalLimit = 1
GlobalLimit = 3 GlobalLimit = 3
FaultsEnabled = FALSE FaultsEnabled = FALSE

1
docs/protocol/RemoraMultiShot.cfg
View file

@ -12,6 +12,7 @@ CONSTANTS
MaxAttempts = 2 MaxAttempts = 2
MaxProgress = 1 MaxProgress = 1
LeaseLimit = 1 LeaseLimit = 1
ReservationLimit = 1
LocalLimit = 1 LocalLimit = 1
GlobalLimit = 1 GlobalLimit = 1
FaultsEnabled = TRUE FaultsEnabled = TRUE

1
docs/protocol/RemoraStress.cfg
View file

@ -14,6 +14,7 @@ CONSTANTS
MaxAttempts = 2 MaxAttempts = 2
MaxProgress = 2 MaxProgress = 2
LeaseLimit = 1 LeaseLimit = 1
ReservationLimit = 1
LocalLimit = 1 LocalLimit = 1
GlobalLimit = 1 GlobalLimit = 1
FaultsEnabled = TRUE FaultsEnabled = TRUE

1
docs/protocol/RemoraTargetFault.cfg
View file

@ -13,6 +13,7 @@ CONSTANTS
MaxAttempts = 2 MaxAttempts = 2
MaxProgress = 1 MaxProgress = 1
LeaseLimit = 1 LeaseLimit = 1
ReservationLimit = 1
LocalLimit = 1 LocalLimit = 1
GlobalLimit = 1 GlobalLimit = 1
FaultsEnabled = TRUE FaultsEnabled = TRUE

37
docs/spec/architecture.md
View file

@ -14,6 +14,8 @@ Terms in this glossary are stable identifiers for use across Remora documentatio
| `Claim` | The worker-facing projection of `JobProtocol` attempt authority, containing its opaque attempt ID, worker incarnation, and fencing token. | | `Claim` | The worker-facing projection of `JobProtocol` attempt authority, containing its opaque attempt ID, worker incarnation, and fencing token. |
| `Lease` | The concrete coordination record representing a `JobProtocol` lease. | | `Lease` | The concrete coordination record representing a `JobProtocol` lease. |
| `ExecutionEnvelope` | The attempt-bound projection of durable inputs that a worker must validate before source or target I/O. | | `ExecutionEnvelope` | The attempt-bound projection of durable inputs that a worker must validate before source or target I/O. |
| `ResolvedSource` | The immutable source identity or inventory pinned from an accepted source request. |
| `SourceCredential` | An owner-authorized registry record whose immutable secret versions are referenced by execution grants. |
| `StagingObject` | An immutable, attempt-specific result member or manifest object that has not been selected by durable publication. | | `StagingObject` | An immutable, attempt-specific result member or manifest object that has not been selected by durable publication. |
| `ResultManifest` | The immutable exact inventory and root identity for one attempt's result aggregate. | | `ResultManifest` | The immutable exact inventory and root identity for one attempt's result aggregate. |
| `CommittedResult` | The concrete durable descriptor corresponding to the attempt committed by `JobProtocol`. | | `CommittedResult` | The concrete durable descriptor corresponding to the attempt committed by `JobProtocol`. |
@ -99,12 +101,20 @@ flowchart TD
## Worker Execution Envelope ## Worker Execution Envelope
- Every Worker process creates a fresh, non-reused `WorkerIncarnation`. A restarted process cannot reuse authority issued to an earlier lifetime, including when it occupies the same replica slot.
- `ExecutionEnvelope` supplies the concrete inputs abstracted by `ActivateAttempt`. Before concrete source or target I/O, a worker verifies that its job ID, opaque attempt ID, fencing token, and worker incarnation match the activated attempt projection. - `ExecutionEnvelope` supplies the concrete inputs abstracted by `ActivateAttempt`. Before concrete source or target I/O, a worker verifies that its job ID, opaque attempt ID, fencing token, and worker incarnation match the activated attempt projection.
- The envelope projects the accepted source request and any durable resolved source identity, the immutable execution grant and concrete target, the attempt-scoped staging identity, opaque credential references, and the validated checkpoint and destination-resume evidence available to that attempt. - Authorization alone is not executable. The exact authority becomes usable only after coordination activation and the durable attempt state both identify it as executing.
- A source identity or inventory that is initially unresolved is durably pinned before concrete checkpoint validation or reuse relies on it. - Activation and inactive-attempt recovery serialize against the same durable authority. Recovery revalidates coordination after entering that boundary and reaps an attempt that remains authorized; compensation removes any pre-commit lease. Activation removes any lease that cannot be paired with the durable executing tuple.
- Reservations have a bounded lifetime. Expiry makes unallocated work eligible for redispatch and makes a durably authorized attempt without its exact live lease eligible for reaping; it never reverses durable authority from coordination state alone.
- A lost successful activation response replays success only while the exact live lease and complete durable executing tuple still agree. A changed attempt, fence, or worker incarnation is a precondition failure.
- The envelope projects the accepted non-secret source metadata and a sensitive-input reference, any durable `ResolvedSource`, the immutable execution grant and concrete target, the attempt-scoped staging identity, pinned credential-version references, and the validated checkpoint and destination-resume evidence available to that attempt.
- A source identity or inventory that is initially unresolved is durably pinned once before concrete checkpoint validation or reuse relies on it. Pinning is a fenced durable mutation and a conflicting value is rejected.
- Large source inventories, completed-part sets, and other unbounded collections are represented by durable streamable descriptors rather than materialized in the envelope. - Large source inventories, completed-part sets, and other unbounded collections are represented by durable streamable descriptors rather than materialized in the envelope.
- Envelope transport is an implementation choice. It may accompany queue delivery or claim, or be read separately, but a worker never infers it from reconstructible coordination state. - Envelope transport is an implementation choice. It may accompany queue delivery or claim, or be read separately, but a worker never infers it from reconstructible coordination state.
- A missing envelope, an identity mismatch, an inaccessible credential reference, or invalid resume evidence fails closed. None grants permission to use a different source, target, staging identity, or checkpoint. - A missing envelope, an identity mismatch, an inaccessible credential reference, or invalid resume evidence fails closed. None grants permission to use a different source, target, staging identity, or checkpoint.
- Every resolved-source, checkpoint, progress, outcome, and publication mutation validates the complete attempt, fence, and Worker-incarnation tuple against current durable authority and the required attempt state.
- An adapter validates the activated envelope and exact live lease when it enters or restarts source or target I/O. Lease loss may leave harmless attempt-scoped staging writes in progress, as permitted by `JobProtocol`; it never authorizes a durable mutation or publication.
- Coordination projection is idempotent and driven from durable projection work. Missing coordination state never reverses authority into `DurableStore`.
## Result Aggregate and Direct Access ## Result Aggregate and Direct Access
@ -115,8 +125,11 @@ flowchart TD
- Manifest construction uses memory bounded independently of member count and total result size. - Manifest construction uses memory bounded independently of member count and total result size.
- Remora resolves the concrete aggregate only from the selected `CommittedResult`; it never derives one by listing a target bucket or prefix. - Remora resolves the concrete aggregate only from the selected `CommittedResult`; it never derives one by listing a target bucket or prefix.
- An authorized reader may access committed members directly in `TargetStore` using exact manifest identities or time-limited URLs. - An authorized reader may access committed members directly in `TargetStore` using exact manifest identities or time-limited URLs.
- A time-limited URL is bound to one exact object identity, including its version when present, and has bounded expiry.
- Raw target listing is operational storage visibility, not result resolution. Clients with independent target credentials may observe uncommitted attempt objects. - Raw target listing is operational storage visibility, not result resolution. Clients with independent target credentials may observe uncommitted attempt objects.
- A stable user-chosen object key or atomically visible raw target prefix is not part of a job result. Providing one requires a separate export contract. - A stable user-chosen object key or atomically visible raw target prefix is not part of a job result. Providing one requires a separate export contract.
- Remora writes attempt members and manifests under a deployment-reserved service namespace using globally unique, create-only identities. It independently computes and persists SHA-256 for every completed object.
- Exact object and multipart identities are durable data. Version IDs are retained when available. A non-versioned namespace is supported only when Remora is its exclusive writer; otherwise immutable version addressing is required.
## Target Cleanup ## Target Cleanup
@ -124,13 +137,19 @@ flowchart TD
- Ambient cleanup processes are stateless and idempotent. Worker-local cleanup may reduce latency but is not the only cleanup path. - Ambient cleanup processes are stateless and idempotent. Worker-local cleanup may reduce latency but is not the only cleanup path.
- Cleanup addresses exact attempt object or multipart identities. Eligibility comes only from `CleanupStage`; missing coordination state, target-object age, and raw prefix membership are not substitutes. - Cleanup addresses exact attempt object or multipart identities. Eligibility comes only from `CleanupStage`; missing coordination state, target-object age, and raw prefix membership are not substitutes.
- Target lifecycle rules are storage hygiene only. They do not substitute for `JobProtocol` publication or cleanup eligibility and must not expire objects retained by a committed aggregate. - Target lifecycle rules are storage hygiene only. They do not substitute for `JobProtocol` publication or cleanup eligibility and must not expire objects retained by a committed aggregate.
- Independent writers and lifecycle mechanisms must be unable to replace or expire live or committed identities in Remora's reserved namespace. A deployment that cannot establish this property is unsupported.
## Source Identity and Resumption ## Source Identity and Resumption
- `checkpointValid` abstracts the concrete proof defined here. A reusable checkpoint binds an immutable `Source` identity, an exact completed boundary, and validated destination state; absent or changed proof makes the checkpoint unavailable to a replacement attempt. - `checkpointValid` abstracts the concrete proof defined here. A reusable checkpoint binds an immutable `Source` identity, an exact completed boundary, and validated destination state; absent or changed proof makes the checkpoint unavailable to a replacement attempt.
- HTTP identity includes the resolved final URL, a strong representation validator as defined by `RFC 9110`, representation length, and identity content encoding. Resume requires a conditional byte-range response whose validator, URL, encoding, and range exactly match the checkpoint. Weak validators, length, or modification time alone do not qualify. - HTTP identity includes the resolved final URL, a strong representation validator as defined by `RFC 9110`, representation length, and identity content encoding. Resume requires a conditional byte-range response whose validator, URL, encoding, and range exactly match the checkpoint. Weak validators, length, or modification time alone do not qualify.
- Object-store identity includes endpoint, bucket, key, size, and an immutable version ID when available. Without a version ID, resume requires a matching conditional range read. Destination multipart reuse additionally requires the same endpoint, bucket, key, upload ID, and exact re-listed completed parts. - Object-store identity includes endpoint, bucket, key, size, and an immutable version ID when available. Without a version ID, resume requires a matching conditional range read. Destination multipart reuse additionally requires the same endpoint, bucket, key, upload ID, and exact re-listed completed parts.
- A Hugging Face branch or tag is resolved once to a full commit. The source identity includes repository type and ID, that commit, and the exact file inventory with Git blob or LFS identities. Retries and cache reuse remain pinned to the stored commit and member identities. - An HTTP or object-store single-object source supplies one explicit `result_path` validated under the manifest member-path rules. The adapter does not derive the result member path from a URL or object key.
- A Hugging Face branch or tag is resolved once to a full commit. The source identity includes repository type and ID, that commit, and the exact canonical file inventory. Retries and cache reuse remain pinned to the stored commit and member identities.
- Hugging Face file entries become byte members and directory entries are structural. Unknown or unsupported entry representations fail closed. Symlinks and submodules are unsupported unless a later contract defines their byte and path semantics.
- Hugging Face selection is the union of exact repository-relative paths and complete path-segment prefixes. Omitted selection means the complete snapshot; a supplied selection that matches no members is invalid.
- Selected Hugging Face paths use the manifest member-path rules. Normalization collisions are invalid, duplicate selections collapse to one member, and canonical inventory order is normalized-path order.
- A Hugging Face member identity records its Git blob ID, LFS SHA-256, or commit-bound Xet hash as applicable. Downloaded content always receives an independently computed SHA-256; Xet metadata is bound to the resolved full commit and path rather than treated as a free-standing content digest.
- A checkpoint never authorizes best-effort concatenation. The adapter maps an unavailable explicitly pinned or resolved snapshot to `source_changed` rather than silently resolving a newer snapshot. - A checkpoint never authorizes best-effort concatenation. The adapter maps an unavailable explicitly pinned or resolved snapshot to `source_changed` rather than silently resolving a newer snapshot.
## Retry and Structured Failures ## Retry and Structured Failures
@ -162,7 +181,12 @@ flowchart TD
- Owner access to a deletion operation is authorized against the concrete target retained in its job tombstone. Missing and unauthorized deletion operations are indistinguishable. - Owner access to a deletion operation is authorized against the concrete target retained in its job tombstone. Missing and unauthorized deletion operations are indistinguishable.
- Workers execute an accepted job from its immutable grant without consulting current identity roles. Failure to establish current authorization fails new submission, reads, and management closed but does not reinterpret accepted execution authority. - Workers execute an accepted job from its immutable grant without consulting current identity roles. Failure to establish current authorization fails new submission, reads, and management closed but does not reinterpret accepted execution authority.
- An administrator may explicitly and idempotently revoke an accepted execution grant. Revocation maps to the protocol cancellation input when applicable and does not add an identity-provider lookup to workers. - An administrator may explicitly and idempotently revoke an accepted execution grant. Revocation maps to the protocol cancellation input when applicable and does not add an identity-provider lookup to workers.
- Source credentials are server-side opaque references resolved only by the worker component that needs them. Secret values never appear in job configuration, API data, events, logs, or structured failures. Target credentials remain internal to `StorageAdapter`. - A source credential reference identifies an owner-authorized `SourceCredential`, never a caller-selected secret-store path. Credential metadata and authorization state are durable; secret material is held behind an opaque secret-store location and immutable version.
- Submission resolves the credential's current authorized version and pins that exact version in the immutable execution grant. Rotation affects future grants and never silently changes an accepted grant.
- The trusted credential resolver accepts only the credential and immutable version carried by an active execution grant. Substitution, execution-grant revocation, credential-version destruction, absence, or secret-store unavailability fails closed and never falls forward to a newer version.
- Credential revocation prevents new execution grants. Revocation of an accepted execution grant follows the administrative cancellation rule above; destruction of a still-referenced version makes resolution fail closed.
- Secret values and complete sensitive source inputs never appear in durable public job configuration, public API projections, events, logs, or structured failures. The complete submitted HTTP URL is sensitive execution input; public projections expose only a sanitized display URL. Target credentials remain internal to `StorageAdapter`.
- Secret-store topology, policy granularity, workload authentication, availability, and recovery are deployment choices. They may narrow access further but do not replace Remora's grant-bound resolver.
## Outbound Source Policy ## Outbound Source Policy
@ -181,6 +205,9 @@ flowchart TD
- Job listing applies current authorization before stable keyset pagination. Cursors are opaque, integrity-protected, and bound to the subject, filters, and membership snapshot. - Job listing applies current authorization before stable keyset pagination. Cursors are opaque, integrity-protected, and bound to the subject, filters, and membership snapshot.
- HTTP snapshots carry a live-event cursor. WebSocket replay is ordered and at least once; invalid, expired, authorization-incompatible, or unavailable history requires a new HTTP snapshot. Event emission exposes progress only when it matches the protocol projection being published. - HTTP snapshots carry a live-event cursor. WebSocket replay is ordered and at least once; invalid, expired, authorization-incompatible, or unavailable history requires a new HTTP snapshot. Event emission exposes progress only when it matches the protocol projection being published.
- Internal mutation requests add transport identifiers and an idempotent operation ID to the corresponding `JobProtocol` action; their eligibility and effects are not redefined here. - Internal mutation requests add transport identifiers and an idempotent operation ID to the corresponding `JobProtocol` action; their eligibility and effects are not redefined here.
- Claim reserves coordination capacity and durably authorizes an attempt, but its response is non-executable until a separate activation operation refines `ActivateAttempt`.
- Activation, resolved-source pinning, and checkpoint persistence are distinct internal operations. Activation refines `ActivateAttempt`; resolved-source pinning is a fenced single-assignment stuttering refinement; checkpoint persistence refines `PersistCheckpoint`.
- Exact replay returns the stored operation result. A changed kind, payload, attempt, fence, or Worker incarnation cannot reuse an operation identity.
## Requirements ## Requirements
@ -222,3 +249,5 @@ flowchart TD
- Implementations may optimize storage layout but must refine `JobProtocol` behavior and preserve its linearization points. - Implementations may optimize storage layout but must refine `JobProtocol` behavior and preserve its linearization points.
- Mutable implementation contracts and documentation select technologies, wire shapes, and mappings from architectural components. They are not normative over `JobProtocol` or this specification. - Mutable implementation contracts and documentation select technologies, wire shapes, and mappings from architectural components. They are not normative over `JobProtocol` or this specification.
- Deployment configuration supplies service credentials and endpoints, configures allowed source reachability, and provides infrastructure-level egress enforcement. - Deployment configuration supplies service credentials and endpoints, configures allowed source reachability, and provides infrastructure-level egress enforcement.
- Deployment configuration reserves and protects Remora's internal target namespace, excludes it from independent lifecycle deletion, and establishes either exclusive write ownership or immutable version addressing.
- Deployment configuration selects secret-store topology and policies while preserving the application-level grant-bound credential-resolution contract.