Architecture

rs3 is organized around a narrow gateway, a repository service, and a provider-neutral storage boundary.

The overview below shows the three important boundaries. S3 backup clients call the gateway through the S3 API. Inside the trusted gateway process, the compatibility layer, namespace mapping, payload encryption, encrypted index state, signed commit publication, and path-redacted admin facts stay under operator control. The gateway writes encrypted format/, keyrings/, commits/, and opaque repository objects/ to the backend and reads or advances a separate Kubernetes Lease anchor. rs3-console only reads path-redacted admin posture and status.

The gateway is the trusted data-plane boundary. The object store keeps opaque encrypted objects and can still observe operational signals such as object counts, sizes, timing, and broad object classes. The external anchor is a separate latest-state authority for the preview. The gateway must fail closed when the anchor cannot be read, advanced, or reconciled with verified commit state.

Crate Boundaries

Crate	Responsibility
rs3-types	Typed identifiers and shared policy types.
rs3-crypto	Key derivation, encryption, metadata sealing, payload envelopes, and commit signatures.
rs3-storage	Provider-neutral object-store trait, local stores, S3 adapter, retention contracts.
rs3-index	Durable index and repository state model.
rs3-repository	Namespace, payload, anchor contracts, commit, replay, maintenance, and commit coordination.
rs3-k8s	Kubernetes Lease anchor integration surface.
rs3-server	Gateway process, configuration, identity, S3 boundary, core admin reports, metrics, and shutdown.
rs3-console	Read-only single-gateway operations UI over the authenticated admin report.
xtask	Integration, performance, and compatibility automation.

Cryptographic operations stay behind rs3-crypto; higher-level crates should not add ad hoc hashing, MAC, encryption, or key derivation logic.

Repository State

Format implementation status

commits/v01 has been removed and is unsupported. It has no production repositories and will not gain a migration path or dual reader. The runtime now publishes bounded normal batches as one compact PAYLOAD_PACK plus one framed INDEX_RUN, and bounded recovery replays those runs without reading payload bytes. Signed INDEX_ROOT checkpoints catalog exact embedded run sections; publication reads back and opens the exact signed root and every new compacted run before anchor adoption. They also drive exact maintenance reachability. Guarded metadata-only compaction and automatic active-run watermarks are implemented for packed and streamed payload carriers. Unknown-length and zero-length streaming writes publish the canonical [PAYLOAD, INDEX_RUN] shape. Large known-length requests publish an opaque standalone payload carrier plus a short [INDEX_RUN] commit containing its encrypted exact reference. Both participate in the same catalog, compaction, and GC graph. New bounded writes are partitioned by effective protection cohort, and guarded full GC renews exact restore dependencies before orphan deletion. Retained-provider restart/fault qualification and pinned-runner filesystem qualification remain release blockers. The current automatic-compaction 1M in-memory gate and three local separate-process filesystem runs pass.

Normal writes are append-friendly and value-separated:

1. Put every non-empty bounded value in the batch into one encrypted payload pack and stage one compact framed binary index run. Empty bounded values are index-only. An unknown-length or zero-length streamed request instead writes one encrypted PAYLOAD followed by its INDEX_RUN; large known-length requests use the standalone carrier flow described below.
2. Publish a signed v02 commit under a random path-private key.
3. Advance the external commit anchor.
4. Acknowledge the client write only after the covering commit is accepted.

This avoids rewriting many backend objects during normal operation and gives crash recovery a concrete boundary.

The payload pack is an immutable value log. It carries shared encryption and container facts once, keeps small-record overhead to one AEAD tag, randomizes record order, and retains segmented AEAD for large range-readable values. The pack itself is ciphertext-only. Its encrypted INDEX_RUN stores authenticated shared pack facts plus each record's ordinal, physical offset, and plaintext length. Per-segment AEAD binds the complete record and segment layout, so a second plaintext digest is not stored. The accepted record reference preserves the exact historical keyring-envelope object and digest used by the containing commit. This lets a cold read issue one exact range GET instead of fetching a pack directory first. Retention mode, expiry horizon, and legal-hold requirement define protection cohorts because the backend protects the containing object.

A streamed value is also immutable and value-separated. Unknown-length streams keep ciphertext in the PAYLOAD section of the same commit as the foreground run. While that run is embedded, a self-stream pointer carries the authenticated payload identity, section ordinal, and segmented-payload header. Before compaction removes the source-run boundary, the pointer is normalized to an exact external carrier: commit key and provider version, stored object length and body digest, historical keyring-envelope reference, section start, ordinal, offset, length, and digest, plus the payload identity and header. Compaction and checkpoints therefore move metadata references only. They do not read or rewrite streamed payload ciphertext.

For a known-length large request, the preview instead uploads one encrypted segmented objects/v02/ carrier outside the publication lock. It verifies the completed exact version, length, post-completion retention horizon, EOF, and full ciphertext digest before a short fenced commit publishes the encrypted reference. This allows distinct large uploads to overlap while keeping repository ordering at one atomic anchor transition per mutation. Cancellation before unambiguous multipart completion aborts the upload. Ambiguous completion, create-only races, or anchor failure can leave an invisible opaque orphan, which guarded maintenance reports and later reclaims. Same-process GC excludes registered in-flight carriers even with a zero minimum age.

The v02 preview does not publish new legal holds. It rejects client hold requests until every catalog, chain, format, and keyring dependency can be held and later released through one guarded lifecycle. The storage conformance layer still tests provider legal-hold mechanics independently. This is an explicit unsupported S3 capability, not an incomplete protection mode operators may enable.

The gateway does not deduplicate payloads. Deduplication would add equality leakage and shared-liveness policy; Kopia already performs chunking and deduplication for the primary client workload.

v02 replaces monolithic index snapshots with an encrypted LSM-style index. Recent immutable foreground runs are level 0. Each compaction selects at most the oldest 128 level-0 runs, merges that bounded window newest-wins, retains tombstones, and emits fewer bounded level-1 generation-range shards. Newer level-0 runs and existing level-1 shards remain exact-referenced and are not rewritten. Level is a storage tier, never a compaction epoch. The preview format accepts only level 0 and level 1, rejecting higher values until a future capability explicitly defines another tier. Equal-generation mutations remain indivisible so a root cannot publish half of one logical generation. Pointers to packs or streams embedded beside a source run are normalized to exact external historical commit, section, and keyring-envelope references before source boundaries are discarded. A retained level-1 tombstone masks older records in earlier level-1 shards. Reclaiming those bottom-tier tombstones and the records they mask remains future guarded or offline maintenance. A small signed INDEX_ROOT catalog names the complete active run set. It does not serialize every live path, and compaction never reads or rewrites payload ciphertext.

Runs contain two specialized encrypted binary projections linked by mutation ordinal. The blinded namespace projection answers HEAD and GET; the path-sorted listing projection answers prefix listings. Frame-local container tables share exact object references. Values never live in an index frame, so LSM compaction is metadata-only and cold recovery does not read user data. Run wire version 6 includes canonical self-stream and exact external-stream carriers, an authenticated namespace-key table, and larger bounded small-object packs. It uses canonical bounded varints for generation and content length in both projections.

The runtime keeps one accepted compact state plus a hard-bounded 4,096-mutation overlay. An exclusive publication barrier freezes that overlay from pre-CAS validation through accepted-state installation. Publication failure discards the overlay instead of rolling back a second full state copy. This preserves commit atomicity without doubling steady-state namespace memory.

The bounded compaction path follows the same memory invariant. It does not clone the full accepted state before planning, verifies each source run with short-lived scratch state instead of accumulating replay state across the window, and interns shared exact container facts while selecting winners. Catalog checkpoints and metadata-only compaction validate exact read-back bytes without materializing another complete query state, then keep the accepted namespace in place and change only the run inventory and anchor. The scale harness enforces process peak RSS so these are measured constraints rather than allocator lore.

Cold recovery is descriptor-first. It walks bounded signed headers from the anchor to the newest catalog, then verifies and applies one encrypted index frame at a time. Signed section descriptors let recovery authenticate index ranges without downloading unrelated payload sections. The recovered state is installed only after a final anchor recheck. The storage boundary also checks the declared length of control-plane objects before allocation: format envelopes are capped at 1 MiB and keyring envelopes at 16 MiB, then the exact stream must terminate at the declared EOF. Provider implementations must opt into this pre-allocation-bounded read path; the storage trait fails closed rather than adapting through a whole-object buffered read. Bootstrap freshness checks and anchor-import commit inventories consume provider-private pages under a fixed 4,096-page/2,000,000-member budget. Emptiness probes stop after the first raw member and permit at most 4,096 empty pages, while unanchored keyring discovery admits at most two raw members. A provider that cannot page within the requested bound, returns an oversized page, or exceeds the applicable budget causes the control path to fail closed.

Automatic maintenance starts requesting packed-run compaction at 256 active runs. With no configured guard it degrades and retries at each additional 64-run boundary, then pauses new mutations at 896. The immutable format ceiling remains 1,024 active runs. Compaction requires the same live Kubernetes WriterFence used for anchor advancement. A fully validated bounded plan that cannot reduce run count may also defer below 896 and retry later. A configured guard rejection, corruption, storage or anchor failure, and every other compaction error poisons the coordinator immediately instead of allowing writes to run past an uncertain maintenance failure. Already accepted reads remain available.

The state-flow view below separates the normal write path from the restore read path. A normal write blinds the namespace lookup, encrypts payload segments, stages payload plus an index run, publishes a signed commit, advances the external anchor, and only then acknowledges the client write. A restore read starts from trusted anchor state, verifies the signed catalog and runs, finds the exact encrypted payload reference, range-reads the retained version when required, verifies AEAD segments, and returns restored bytes.

Path Privacy

The backend object store sees class-like prefixes and opaque object IDs. It must not see plaintext paths, directory names, Kubernetes namespaces, resource names, snapshot names, tenant names, or user metadata labels.

Logical lookup uses secret-derived namespace tokens inside the trusted gateway. Directory listing is answered from repository index state, not by exposing client paths as backend object keys.

In v02, encrypted runs carry a blinded lookup projection and a plaintext-path listing projection inside authenticated ciphertext. Run keys, public metadata, and signed headers expose neither paths nor plaintext projection bounds. The v02 runtime does not persist the legacy durable prefix-token representation.

Rollback Resistance

The object store can preserve encrypted bytes, but it cannot establish that it served the latest valid commit. rs3 therefore separates storage durability from latest-state authority:

• Object store: encrypted keyring envelopes, format roots, signed commits, and independently sealed index runs.
• External anchor: monotonic latest commit sequence and digest.
• Gateway: verification, replay, and fail-closed behavior when the anchor cannot be trusted.

Provider retention and Object Lock are useful for preventing deletion of object versions. They do not replace commit signatures or external anchors.

For v02, the external anchor stores the accepted commit key, body digest, provider version ID when needed, signing key ID, and active format-root reference. Recovery derives the exact catalog, run, and payload graph from that root. Anchor import from a trusted bundle verifies the graph before recreating a missing anchor.

Catalogs and effective index records are exact reachability roots. INDEX_ROOT catalogs name index runs only; effective highest-generation records name exact payload-pack or streamed-payload carriers. Maintenance marks the exact catalog and run versions plus the exact payload-containing object versions selected by live records. A payload reference does not keep its commit's entire ancestry reachable. This rule also protects a zero-length streamed carrier even though a client read can return an empty body without fetching payload bytes. GC completes a fail-closed mark before any deletion and rechecks both the maintenance fence and anchor before deleting an exact version.

Payload-pack cleaning is separate from index compaction. It rewrites live records from a sufficiently sparse pack into a new random pack, publishes new higher-generation physical references, and retains the old version until no current or protected historical root reaches it. It does not use mutable reference counts as deletion authority.

Writer Coordination

Read-write failover is supported only inside one Kubernetes apiserver and Lease coordination domain. The writer owns a monotonic fence epoch on the anchor Lease, and the same resource-version CAS checks that fence when advancing the anchor. Checkpointing and compaction use the same authority. Compaction writes metadata-only delta-carrier commits and the candidate signed root as direct children of the same accepted base. The candidate runs are exact-referenced by the root and are not individually anchored. The writer reads back and authenticates every new run plus the exact signed root, then rechecks the fence and base anchor before one CAS adopts the root. Recovery rejects a compacted carrier whose parent, sequence, level, or compaction generation does not match that sibling publication shape.

Inside one gateway process, a repository instance admits exactly one mutation owner. A commit coordinator holds an RAII lease that is also retained by every delayed publisher task; direct mutation and maintenance entry points fail while that lease exists. This prevents a cancelled request or a second local API path from publishing and clearing another batch's speculative overlay. All semantic installation checks occur before anchor CAS. If the anchor advances but local lock installation fails, callers receive an explicit recovery-required error, new mutations stop, and the process must restart from the accepted anchor.

The single owner does not serialize large request bodies. Distinct declared- length standalone uploads run concurrently, then queue for the short stage, commit, and anchor publication section. A stalled standalone request therefore does not block an unrelated buffered mutation. Conflicting create-only writes are resolved during fenced publication, with exactly one visible winner.

Disconnected writers that merely share S3 are unsupported. Conditional object creation can prevent a collision at one key, but it cannot fence a stale writer, order repository-wide state, or merge divergent encrypted namespace histories. S3 listing and timestamps are not coordination primitives. A future disconnected mode would need explicit branches, authenticated merge semantics, and deterministic conflict policy in a different repository contract.

Payload segmentation is recorded per pack record or streamed-payload header. Small packed values use one AEAD record; medium and large values use independently authenticated segments for bounded range reads. The authenticated index descriptor carries the physical pack layout or exact streamed section facts, so neither read path needs an unauthenticated directory lookup. Bounded commits use one single-part upload with a compact header; only genuinely streaming commits pay the fixed multipart header reservation.

Partial streamed reads fetch only the authenticated ciphertext segments that cover the requested plaintext range. The in-memory decrypted-segment cache uses an opaque identity derived from repository and historical keyring context plus the exact commit, version, body, section, payload-header, and content-length facts. The actual payload ID remains the AEAD identity. This prevents cache entries from aliasing across exact carriers without creating a backend object or exposing a new backend key.

S3 Compatibility

The first gateway surface focuses on the operations backup clients need:

• PUT Object
• GET Object, including ranges
• HEAD Object
• ListObjectsV2
• DELETE Object
• native conditional create behavior (PutObject with If-None-Match: *)
• retention and legal-hold plumbing where the backend supports it
• provider version IDs and exact-version reads for retained/Object Lock repositories

There are two preview-qualified storage safety profiles. The atomic-create profile requires native PutObject with If-None-Match: *. The retained-version profile requires Object Lock, provider version IDs for all restore-critical writes, and exact-version reads after a newer latest version exists. Non-atomic HEAD before PUT is not a security-equivalent fallback for either profile.

The preview compatibility workloads are Kopia and Velero with the Kopia uploader. Kopia drives the lower-level S3 behavior; Velero exercises the Kubernetes backup and restore workflow through that path. Broader S3 compatibility should be added behind tests that verify restored bytes and privacy invariants.

Maintenance Supervisor

Full maintenance is supervised inside the mutation-capable gateway. It is not a second controller, CronJob, or repository writer. The supervisor depends on the provider-neutral commit-anchor and maintenance-guard traits; Kubernetes supplies those contracts with the same fenced Lease used by normal writes.

Automatic mode evaluates bounded retention-deadline and reclaimable-orphan facts, adds deterministic jitter, and applies cooldown or failure backoff only when doing so cannot cross the renewal safety boundary. Manual mode performs no background inventory scans and accepts explicit operator runs. Restore-readonly mode forces the supervisor off. A missing guard parks the state machine rather than retrying or mutating without exclusion.

An operator dry run returns aggregates plus an opaque digest computed by the repository over the canonical exact private plan. Apply enters the coordinator maintenance window, drains pending commit work, verifies the guard and anchor, builds one fresh exact plan, compares its digest, and mutates that same plan instance. Object/version identities never cross the repository boundary. A different exact delete or renewal target therefore invalidates approval even when aggregate counts and bytes happen to match.

The break-glass offline command builds the same repository runtime but first fences the real anchor Lease. It refuses a live renewing writer and renews its own fence until the run and release complete. This path is for a stopped gateway and is not an alternative scheduler.

Admin Surface

Core code owns the path-redacted admin report model used by doctor checks, runtime status, backend and anchor posture, retention settings, and restore-trust summaries. That keeps the S3 data plane independent from operator UX and broader management concerns.

In rs3-server, src/admin.rs contains only the shared report builders and serializable summaries. A separate operator UI or platform integration should consume those summaries through an explicit admin boundary instead of sharing backup-client S3 credentials or browsing repository objects directly. The report shape remains a preview fact contract. Full maintenance is the first narrow workflow API: read routes accept the admin read or mutation token, while dry-run, digest-gated apply, cancel, pause, and resume require a distinct mutation token and emit path-redacted operation records. Recovery, key-management, and arbitrary storage workflows are not exposed.

rs3-console is the narrow single-gateway UI for these reports. It serves a browser interface and proxies GET /api/posture, GET /api/status, or GET /api/maintenance to the gateway admin listener. The browser authenticates to the console; the gateway read token remains server-side. The console has no repository browser, database, scheduler, work queue, or mutation routes.

Admin and platform surfaces are not part of the S3 data plane and should not expose client-visible object browsing, backend object IDs, configured bucket names, repository IDs, prefixes, or secret material.

Mutating workflows other than the narrowly authorized maintenance API, such as anchor import, recovery apply, key rewrap, data-key rotation, or arbitrary compaction, stay explicit local operator actions until they have a dedicated authorization and audit model.