private-rag

Outsource a confidential knowledge base to an untrusted cloud, embed inside an attested SEV-SNP CVM with an information-theoretically blind GPU, store as distance-preserving ciphertext, retrieve at near-plaintext cost.

workspace  9 Rust crates
tiers  T1 ✓ · T2 ✓ · T3 deferred
tests  150 passed · 0 failed
rev  2026·05·14
attested TEE · blind GPU · ciphertext at rest
GELO overhead
1.27×plain
Qwen3-Emb-0.6B · paper-parity · AOCL-BLIS · 100-doc NFCorpus
end-to-end wall
153ms/text
vs 121 ms plain ⇒ +32 ms / +26 % per text
U-Verify miss rate
2.4·10⁻⁷
k = 8 probes at L = 3 per offload (TwinShield §V-C)
CVM footprint
3.2GB
Qwen3 weights Arc-shared between embedder + U-Verify cache
§01

What this is

A working retrieval-only private RAG system: a regulated enterprise outsources a document corpus to a cloud provider, queries it interactively, and gets ranked results back — without the cloud ever seeing plaintext text, plaintext embeddings, or which document matched. The prototype combines a thin client, an attested embedding TEE, an untrusted-but-blind GPU, and distance-preserving ciphertext at rest.

The hard question this is built around: can openweight transformer embedding inference be served by an untrusted cloud GPU at near-plaintext cost, without the GPU learning the input? The literature offers three families of answer — full FHE (orders of magnitude too slow), confidential-compute GPUs (10× cost premium, vendor lock-in), and TEE-on-CPU paired with masked GPU offload. This codebase ships the third.

The chosen storage path is CAPRISE-at-rest: the cloud operator sees only distance-preserving ciphertext, and the CAPRISE seed is derived inside the SEV-SNP CVM per request from a two-party HKDF (storage page). A second strategy — RemoteRAG, where the doc index is plaintext but the query gets formal (n, ε)-DistanceDP via planar-Laplace noise + Paillier rerank — was prototyped end-to-end and parked. The two are mutually exclusive at the type level. See storage · RemoteRAG for the decision trail.

§02

Definitions & glossary

TermDefinition
TEE / CVMTrusted Execution Environment. This codebase uses an AMD SEV-SNP Confidential VM as the trust boundary — encrypted memory plus a hardware-signed attestation report.
RATLSRemote-Attested TLS. Client verifies the SEV-SNP attestation report inside the TLS handshake before any plaintext is sent.
CAPRISEDistance-preserving encryption (Ye et al. 2026). Server can run cosine over ciphertext directly; the production storage path uses CAPRISE with a two-party HKDF-derived seed.
GELOThe per-forward orthogonal-mask split-inference protocol used by embedding / reranking / generation. Hidden states cross PCIe as U = A·H with A a fresh Haar mask sampled inside the TEE.
RemoteRAGAn alternative storage protocol (Cheng et al., ACL Findings 2025). Plaintext doc index plus planar-Laplace DP on the query plus Paillier rerank. Prototyped and parked — mutually exclusive with CAPRISE.
CompassZhu, Patel, Zaharia, Popa (OSDI 2025). HNSW-over-Ring-ORAM private graph retrieval — the GraphRAG layer wraps it.
AloePriA static weight-obfuscation alternative to GELO (matrix-Γ obfuscated weights served via unmodified vLLM). Evaluated as a comparison generation path; no TEE in the loop.
HNSWHierarchical Navigable Small World graph — the graph-based ANN index used by every retrieval layer in this stack.
Vec2TextMorris et al., EMNLP 2023. Inverts embedding vectors back to near-exact source text — the canonical motivation for keeping embeddings off the storage server.
T1 / T2 / T3Hardware deployment tiers used in §05 hardware table. T1 = in-process mock, T2 = VM-simulated CVM, T3 = real SEV-SNP silicon with VFIO GPU passthrough.
§03

Full process flow

A request traverses four trust zones, in three stages. Stage 1 · Ingest takes raw chunks from the client, embeds them inside the attested TEE (offloading the matmuls to a blind GPU under the GELO mask), derives the CAPRISE + AES keys via two-party HKDF, and writes the resulting ciphertext records to the untrusted vector DB. Stage 2 · Retrieval embeds the query through the same shape, encrypts it under the query CAPRISE coefficient, runs cosine ANN over the ciphertext on the server, then decrypts the top-k′ hits back inside the TEE. Stage 3 · Rerank derives a per-session SessionKey via HKDF, scores each (query, chunk) pair under GELO+mask, sorts in the TEE, and AES-GCM-seals the top-k final list (padded with decoys to k_max, shuffled) under a per-query key — scores never leave the CVM.

CLIENT user device — holds user_x_sk, decrypts plaintext results
TRUSTED · TEE SEV-SNP CVM — sees text, activations, weights, derives keys per request
UNTRUSTED · GPU commodity Vulkan — sees masked U=A·H + public weights only
UNTRUSTED · STORAGE vector DB — sees CAPRISE ciphertext rows + AES-GCM chunk blobs · access patterns
CRYPTO PRIMITIVE value-class — HKDF, CAPRISE, AES-GCM
FIG. 01 — End-to-end process flow · Stage 1 Ingest + Stage 2 Retrieval + Stage 3 Rerank dashed red = trust boundary · dashed amber = masked PCIe · solid teal = ciphertext · dashed navy = attestation
Client holds keys · sends requests TEE · SEV-SNP CVM attested · embed + KDF + crypto Untrusted GPU Vulkan · masked matmuls only Untrusted Storage vector DB · ciphertext only RATLS PCIe · masked HTTP · ciphertext STAGE 1 · INGEST N text chunks → embed via GELO → CAPRISE / AES encrypt → write to encrypted vector DB SEV-SNP report + VCEK ① Ingest request tenant_id · user_x_sk chunks: [{id, text}…] ② Tokenize · GELO embed Pool + L2 normalize → pooled vector / chunk ③ HKDF derive tee_user_x_sk ‖ user_x_sk caprise_seed · aes_key ④ Encrypt per chunk CAPRISE-enc doc (3/8 coeff) AES-GCM chunk text ⑤ Write to encrypted vector DB InMemoryEncryptedIndex.insert(…) × N records (EncryptedEmbedding, ChunkCiphertext) server sees ciphertext + access log only CAPRISE + AES records · × N GPU · masked matmuls Q · K · V · O · FfnUp · FfnDown matmul_many bundles QKV + gate/up × 28 layers · × N chunks U · public W → masked output weights public · activations are A·H mix: U = A·H unmix: Aᵀ·(U·W) ⑥ Zeroize caprise_seed · aes_key · user_x_sk ⑦ 200 OK { ingested: N } all N inserts done STAGE 2 · RETRIEVAL attest → embed query → CAPRISE-encrypt → cosine over ciphertext → decrypt hits → top-k′ candidates inside CVM SEV-SNP report + VCEK ① Query request tenant_id · user_x_sk text · top_k ② Tokenize · GELO embed Pool + L2 normalize → pooled query vector ③ HKDF re-derive same two halves as ingest → same caprise_seed ④ CAPRISE-enc query query form · 1/8 coefficient fresh 16-byte nonce ⑤ Cosine ANN over ciphertext InMemoryEncryptedIndex.search(eq, top_k') distance-preserving — no decrypt → top-k′ candidate pairs (EncryptedEmbedding, ChunkCiphertext) CAPRISE-encrypted query GPU · masked matmuls Q · K · V · O · FfnUp · FfnDown matmul_many bundles QKV + gate/up × 28 layers · × 1 query U · public W → masked output mix: U = A·H unmix: Aᵀ·(U·W) ⑥ Hand off to Stage 3 candidates remain encrypted (EncryptedEmbedding, ChunkCiphertext) top-k′ ciphertexts STAGE 3 · RERANK no new client request · CVM decrypts candidates internally → score (q, doc) under GELO+mask → in-TEE sort → AES-GCM seal · padded · shuffled stays inside CVM · same session ① Decrypt candidates CAPRISE-decrypt embeddings AES-decrypt chunk text · in-CVM ② Score (q, doc) pairs GELO-masked forward × k′ cross-enc · or yes/no-disc ③ In-TEE sort + top-k tie-shuffle · scores zeroized ranked chunk list GPU · masked matmuls Q · K · V · O · FfnUp · FfnDown (per (q, doc) joint forward) × layers · × k′ candidates U · public W → masked output scores never cross PCIe mix: U = A·H unmix: Aᵀ·(U·W) ④ Seal + pad + shuffle HKDF QueryKey · AES-GCM-256 k_max items · rank in payload ⑤ Bundle response EncryptedRerankBundle over RATLS client decrypts · sorts by rank ranked chunks (plaintext) k_max AES-GCM ciphertexts
Stage # Step Who runs it What crosses the boundary
1 · Ingest Client opens RATLS, verifies attestation, posts {tenant_id, user_x_sk, chunks: [{id, text}, …]}Client → TEEplaintext text + 32-byte secret over RATLS
TEE tokenizes + GELO-embeds each chunk · GPU runs the masked matmuls (Q/K/V/O + FfnUp/FfnDown) under the mask · TEE re-masks per layer · pools + L2-normalizesTEE + GPUmasked U on PCIe outbound · U·W back
TEE looks up (or creates on first contact) tee_user_x_sk[tenant]; runs HkdfPolicy::derive(user_x_sk, tee_user_x_sk, tenant_id)(caprise_seed, aes_chunk_key)TEE
TEE CAPRISE-encrypts each pooled vector with the doc coefficient (3/8) + a fresh 16-byte nonce; AES-256-GCM encrypts each chunk textTEE
Storage server inserts × N records into InMemoryEncryptedIndex; the server only ever sees the ciphertext pair (EncryptedEmbedding, ChunkCiphertext) + access patternsStorage serverencrypted records over HTTP
TEE zeroizes caprise_seed, aes_chunk_key, and the incoming user_x_sk at end of requestTEE
TEE returns 200 OK { ingested: N } over the same RATLS sessionTEE → Clientresponse over RATLS
2 · Retrieval Client opens RATLS, verifies attestation (fresh nonce), posts {tenant_id, user_x_sk, text, top_k′, k_final, k_max} — a single request that drives Stages 2 + 3 togetherClient → TEEplaintext query + 32-byte secret over RATLS
Same embed pipeline as Stage 1 — GELO-masked matmuls offload to GPU for one inputTEE + GPUmasked U on PCIe outbound · U·W back
TEE re-derives the same (caprise_seed, aes_chunk_key) deterministically from tee_user_x_sk[tenant] ‖ user_x_sk — that's the property that lets queries decrypt what ingest encryptedTEE
TEE CAPRISE-encrypts the query with the query coefficient (1/8) and a fresh nonceTEE
Storage server runs cosine ANN over the ciphertext — CAPRISE is distance-preserving, so no decryption is needed. Returns top-k′ pairs (EncryptedEmbedding, ChunkCiphertext)Storage serverencrypted query in · encrypted hits out
Encrypted top-k′ candidates flow from the storage response back into the CVM and feed Stage 3 in the same in-CVM session; the CVM does not decrypt yet, and nothing crosses RATLS at this pointTEE (internal)— (no boundary crossed)
3 · Rerank TEE CAPRISE-decrypts each candidate embedding and AES-decrypts its chunk text inside the CVM; decrypted bytes never leave the trust boundaryTEE
For each of the k′ candidates the reranker (cross-encoder or causal-LM discriminator) runs the GELO-masked forward pass; matmuls offload to GPU under the same mask as Stage 1/2TEE + GPUmasked U on PCIe outbound · U·W back
Scores stay inside the CVM. In-TEE sort + tie-shuffle picks top-k_final; scores then zeroizedTEE
TEE derives QueryKey via HKDF (info "gelo-rerank.query.v1") and AES-GCM-encrypts each ranked chunk under it; the list is padded with decoys to k_max and shuffled so emission order conveys no rankTEE
Client receives the EncryptedRerankBundle as the response to the original Stage 2 request, decrypts each item with the locally-re-derived QueryKey, drops decoys, sorts by the embedded rank, and forwards the ordered plaintext to its generatorTEE → Clientk_max AES-GCM ciphertexts over RATLS
# Approach-4 step Current implementation Status
1Ingestion / chunkingClient passes raw text to TEE over RATLS; TEE chunk-encrypts AES-256-GCM with the derived aes_chunk_keygreen
2Session setup / attestationSEV-SNP attestation via /attest; SnpAttestationVerifier chain validates ARK→ASK→VCEK; REPORT_DATA binds (model_id, scheme_id, nonce)green
3Embedding (TEE-side)Primary variant chosen: full embedding model in TEE with GELO mask offloading linears to GPU. ObfuscaTune-encoder + GELO-encoder variants not built.green
4Storage — CAPRISE key managementOption 3 chosen: two-party HKDF (user_x_sk client + tee_user_x_sk TEE → caprise_seed_key + aes_chunk_key). In-memory persistence today; KMS-released variant deferred.green
5Query submissionClient POSTs (tenant_id, user_x_sk, text, top_k) over RATLS; TEE re-derives the same keys deterministically, embeds via GELO, CAPRISE-encrypts with the query coefficient (1/8)green
6RetrievalCosine over CAPRISE ciphertext (in-memory linear ANN today; HNSW swap planned at >100k docs). Distance-preserving — no server-side decryption ever.green
7Decryption / handoffTEE CAPRISE-decrypts the candidate embeddings + AES-decrypts the chunk texts inside the CVM, zeroizes the derived keys, returns plaintext top-k over the same RATLS session.green
8Reranking / filteringNot implemented in-TEE; client-side cross-encoder rerank is the design intent (TEE returns plaintext top-k that the client reranks locally). The TEE re-ranks Stage-1 candidates against the clean embedding only for accuracy verification in the bench.deferred
9Answer generationOut of scope for the prototype — client-side LLM is the assumed architecture. ObfuscaTune / GELO+OSNIP variants for hosted generation are designed in the spec, not built.deferred
§04

High-level components

Each entry below is a box in the diagram. Detail pages cover the per-component method, threat model, source paper, and measured overhead.

§A

Storage · CAPRISE-at-rest

The production storage path. Doc embeddings live on the untrusted server as distance-preserving CAPRISE ciphertext; cosine ANN runs over the ciphertext directly. The CAPRISE seed is HKDF-derived inside the SEV-SNP CVM per request from a two-party secret (Option 3) and zeroized at return.

  • core::caprise · distance-preserving encryption
  • core::keying · HkdfPolicy · two-party HKDF
  • gelo-rag · GeloRagTwoPartyService (runner-bound)
  • core::storage · InMemoryEncryptedIndex (cosine over ciphertext)
  • core::content · AES-256-GCM chunk-text cipher
read more
✕ alternative — not pursued
§B

Storage · RemoteRAG

A second storage strategy prototyped during the work and parked. Keeps the index plaintext on the server; the query gets formal (n, ε)-DistanceDP via planar-Laplace noise, with a Paillier homomorphic dot-product reranking against the clean query. Mutually exclusive with CAPRISE — the project chose CAPRISE.

  • remote-rag::planar_laplace · Stage-1 query DP
  • remote-rag::paillier · Stage-2 PHE rerank · 256-bit · CRT
  • remote-rag::service · RemoteRagService (parallel, not wired in runner)
  • 10k-doc bench: 96% recall vs ground truth · 23.5 ms / query
  • Documented for traceability; revivable if deployment shape inverts (public corpus, private queries)
read why not chosen
§C

Embedding · GELO

An attested SEV-SNP CVM runs a transformer encoder; every Q/K/V/O + FFN GEMM is masked with a Haar-uniform orthogonal A per forward pass and offloaded to a commodity Vulkan GPU. The GPU sees only U=A·H plus public weights.

  • gelo-protocol · mask · shield · U-Verify · OutAttnMult · permuted attention
  • gelo-embedder · GeloBertEmbedder · GeloQwenEmbedder
  • gelo-gpu-wgpu · WgpuVulkanEngine
  • gelo-tee-sev-snp · SnpTrustedExecutor · SnpAttestationVerifier
read more
§D

Reranking · GELO

Post-retrieval rerank inside the same SEV-SNP CVM. Two architecture-typed services — Qwen3-Reranker-0.6B (causal-LM, primary) and bge-reranker-v2-m3 (XLM-RoBERTa cross-encoder, parity bench) — both ride the embedder's GELO mask + TwinShield primitives. Scores never leave the TEE; the wire emission is a fixed-shape, shuffled, AES-GCM bundle with rank sealed inside each ciphertext.

  • gelo-reranker · causal_discriminator · cross_encoder · score · session
  • shared GELO substrate — ~98% code reuse with the embedder
  • Qwen3 single-pair: 1.83 s · k′=20 batched on rayon candidates
  • output: fixed-shape encrypted bundle · rank order hidden from operator
read more
§E

Generation · GELO LLM

Autoregressive answer generation inside the same SEV-SNP CVM. Gemma 4 E2B and E4B run under the embedder's GELO mask + TwinShield primitives, with hybrid attention keeping cheap sliding-window layers fully in-TEE and only the long-range global layers crossing PCIe under mask. The Per-Layer Embedding table lives in encrypted CVM DRAM so token-id gather addresses never reach the GPU; the KV cache stays in-TEE and tokens stream back to the client under per-query AES-GCM.

  • gelo-embedder::decoder · gemma4 · generation · kv_cache · sampler
  • gelo-protocol::ple · PLE table + gather inside the trusted executor
  • hybrid attention router · local SWA(W=512) in-TEE · global via OutAttnMult / fused permuted (deferred)
  • Gemma 4 E2B primary · E4B scaling · 31B stretch · MoE deferred
read more
§F

Generation · AloePri LLM

Forward-looking. Second private-inference path for the generative LLM step, complementary to the GELO TEE-GPU stack. The Qwen3 1.7B weights are rewritten once offline into an obfuscated GGUF (covariant obfuscation, arXiv 2603.01499); stock llama-server hosts it without a fork; the trusted client wrapper holds only a secret token permutation τ and the tokenizer. Prompts cross the wire as an integer-array of obfuscated IDs — the server never reads any plaintext. Trade vs GELO: no TEE required, in exchange for static obfuscation (TTRSR ≤ 15% empirical, not information-theoretic per-batch).

  • demonstrator: Qwen3 1.7B Q8_0 plaintext → fp32 obfuscated GGUF (8.6 GB)
  • obfuscate_qwen3_gguf · offline weight rewriter (Algorithm 1 keymat · §5.2.5 norm fusion · Π · αeh noise · inter-head shuffle)
  • AloePriClient · trusted-side τ-map over native /completion int-array endpoint
  • Qwen3 QK-norm blocks the paper's intra-head Algorithm 2 transforms — inter-head shuffle only deployed; ISA defense partial (see §08 of the doc)
  • Gemma 4 deferred: 5 residual norm sites per block push κ-compounding past comfortable accuracy bounds
read more
§G

GraphRAG — Compass

Shipped · benched. Private LightRAG retrieval inside the same SEV-SNP CVM. Three Compass-encrypted HNSW indexes (entities · relations · chunks) plus two XorMM volume-hiding multimaps (adjacency · src_chunks) plus an AES-GCM chunk store — every server-visible op reduces to a Ring-ORAM read, an EMM lookup, or a blob fetch. Local + Hybrid modes wired; 10–58 ms p50 in-memory at N=100–2000.

  • ring-oram · semi-honest baseline + lazy eviction + treetop cache · async BlockBackend
  • compass-index · layered HNSW · Directional Filter (~50 % read reduction)
  • xormm-emm · volume-hiding adjacency & src_chunks · cuckoo placement
  • compass-rest-backend · axum + sled storage server + reqwest client
  • light-kg-store + lightrag-private · LightRAG kg_query orchestrator
  • Wired through HkdfPolicyV2 + SnpTrustedExecutor + RATLS — no new trust assumptions
read more
§H

Hardware

AMD EPYC SEV-SNP for the TEE, commodity Vulkan GPU via VFIO passthrough for offload. No confidential-compute GPU needed — GELO masks make the GPU information-theoretically blind regardless.

  • AMD EPYC Genoa / Turin · SEV-SNP · attestation via virtee/sev
  • Vulkan via wgpu 26 · burn-cubecl · autotune cache
  • AOCL-BLIS (CPU mask GEMMs) · skx_asm AVX-512 dispatch on Zen
  • 3 simulation tiers — T1 in-process · T2 QEMU mock · T3 real silicon (deferred)
jump to section
§05

Hardware

The prototype is built for commodity hardware on both ends — EPYC SEV-SNP for the trusted side, any Vulkan-capable accelerator on the untrusted side.

SEV-SNP CVMtrusted · CPU

Encrypted-RAM VM with hardware attestation by the AMD Secure Processor.

vendor
AMD · EPYC 9004 (Genoa) / 9005 (Turin) — production target
technology
Secure Encrypted Virtualization with Secure Nested Paging (SEV-SNP)
attestation
ARK → ASK → VCEK via /dev/sev-guest · SNP_GET_EXT_REPORT
Rust SDK
virtee/sev 7 (Apache-2.0) · DCAP byte format · ECDSA-P-384 sig
memory
RMP-protected · per-CVM key never leaves AMD-SP · ~3.2 GB encrypted footprint for Qwen3 with Arc-share
SKU economics
Hetzner AX42 EPYC dedicated (~€100/mo) vs ~$5,800/mo for managed confidential-GPU SKUs
dev box
AMD Strix Halo (Ryzen AI Max+ 395) — non-SEV-SNP, same vendor; T2 uses QEMU mock

GPU offloaduntrusted · accelerator

Commodity Vulkan device — sees only masked activations + public weights.

API
Vulkan via wgpu 26 · Linux RADV / NVIDIA / Intel Mesa
kernel stack
burn-cubecl 0.20.1 over cubecl-wgpu 0.9.0 · SPIR-V codegen
autotune
cubecl-runtime::TuneCache · disk-persistent · configured by workspace cubecl.toml
precision
f32 (default) · f16 path behind new_fp16()
dev card
AMD Strix Halo iGPU (integrated; shared LPDDR5X) — no PCIe, no SWIOTLB
target
discrete GPU via VFIO passthrough: RTX 4090 / RX 7900 XTX / Intel Arc
DMA cost
~15 ms / Qwen3-text SWIOTLB bounce on real silicon; vanishes under TDISP
TierHostBinary modeStatusValidates
T1 — in-processany x86_64 Linuxcargo test --features mockgreenprotocol math · report format · parser/verifier round-trip · tamper rejection
T2 — VM-sim CVMregular QEMU/KVMSNP_MODE=mockgreenOS boundary · systemd lifecycle · weight loading · full HTTP service
T3 — real siliconHetzner EPYC + VFIO GPUSNP_MODE=productiondeferredreal /dev/sev-guest · ARK chain · RMP · SWIOTLB · GPU passthrough