generation
— GELO LLM

Autoregressive answer generation under the same GELO mask + TwinShield primitives the embedder and reranker already validated. Qwen3-1.7B runs end-to-end today inside an attested SEV-SNP CVM; every Q/K/V/O + MLP matmul rides through a per-batch Haar mask before crossing PCIe; QK-norm + RoPE + GQA + SwiGLU happen entirely in-TEE on plaintext tensors. Tokens stream back to the client over RATLS. Gemma 4 E2B/E4B remain the architecture north star — substrate landed; full decoder integration deferred.

protocol GELO + TwinShield + (optional) hybrid attention

TEE AMD SEV-SNP CVM

GPU commodity Vulkan via VFIO

model Qwen3-1.7B · Qwen3-4B (same path)

prompt → KV cache → token stream · everything sealed inside

Prefill tok/s

121 agg · B=8

Qwen3-4B at production-extraction shape (n = 2 048, DCT-IV cascade). Single-stream B = 1 long-context (n = 2 561) holds 81 tok/s.

Decode tok/s

4.62 agg · B=8

Qwen3-4B at n = 2 048 (in-TEE attention 54 % of wall). Single-stream B = 1 long-context: 1.44 tok/s — attention-bandwidth-bound at this hardware tier.

Batched decode — wall amortisation

2.6 – 5.2× vs B=1 × serial

Both at B = 8. 2.6× per-seq at the real-weight production shape (Qwen3-4B n = 2 048, per-step per-seq 217 ms batched vs ~557 ms scaled single-stream — compute-amortised regime, in-TEE attention dominates). 5.2× at the synth Qwen3-4B-shape n_kv = 64 bench (4 layers, 9.83 ms vs 50.56 ms per-step per-seq — launch-overhead-amortised regime, attention is small).

Auto vs Haar @ pow2-aligned

−36% TTFT

Qwen3-4B n = 2 040 · Auto routes to HD₃ at pad ratio 1.0 · 16.3 s vs ~25.6 s dense Haar QR baseline · σ < 12 ms across decode steps. Cascade doesn't touch HD₃ shapes so this gap is stable post-2026-05-26.

§01

Role in the project#

where this fits · what it closes · what currently runs

Generation is the final stage that closes the private-RAG loop: embed the query, retrieve ciphertext candidates, rerank them inside the CVM, and generate the answer inside the same trust boundary rather than handing decrypted top-k chunks to a client-side LLM. Model weights are public; the protected asset is the user's prompt, the retrieved chunk text, the per-token logits, and the sampled tokens — all of which must stay confidential against a co-located untrusted GPU and an untrusted CVM operator.

What runs today. Greedy decoding on Qwen/Qwen3-1.7B runs end-to-end under the GELO protocol: prefill + 8-token decode with bit-identical token-sequence output across PlaintextExecutor, InProcessTrustedExecutor (per-forward Haar mask + shield(8, 4.0)), and the full stack with U-Verify k = 2 enabled. The GELO mask itself is a +33 % TPOT overhead on the AMD RADV GFX1151 iGPU; argmax stability holds — the masked branch emits the same tokens as the plain branch on real weights. The same forward-pass code path that the embedder and reranker already use services generation, with one Qwen3-specific addition (per-head QK-norm pre-RoPE) optional on the loader.

What is parked. Gemma 4 E2B / E4B remain the architecture north star — they exercise PLE in encrypted CVM DRAM (an address-bus-leak mitigation the GPU can't help with), hybrid local/global attention with sliding-window in-TEE, p-RoPE, cross-layer KV sharing, GeGLU, AltUp, and final-logit softcap. Every protocol primitive needed to run them is already landed and tested on synthetic weights; what's missing is the Gemma-specific decoder refactor (per-class head_dim, per-class rope_theta, cross-layer KV sharing, GeGLU dispatch, AltUp). Tracked in gemma4-architecture-roadmap; summarised in §05.

What's open. Two protocol gaps that today's measurement surfaces: (a) U-Verify is unusable at 1.7B-parameter scale — k = 2 Freivalds probes against ~13 GB of TEE-side weights collapse decode to 0.04 tok/s. The protocol needs a sublinear-in-weight-size integrity primitive (randomised-block Freivalds, batched verification, or checkpoint-only attestation) before integrity rides production. (b) Decode-phase KV-cache bandwidth grows with context length and dominates past ~4k tokens — the SCX-style per-position encoding (Yuan et al., SIGCOMM 2025) is the candidate primitive; security analysis under the openweight + per-batch-fresh-A assumption is the gating spike before adoption. Both items are §10.

§02

Definitions & glossary#

terminology introduced by this stage

Only the terms specific to generation and to the model families involved (Qwen3 today, Gemma 4 as the architecture target). Protocol terms (mask, shield, U-Verify, OutAttnMult, permuted attention) are defined in embedding §05 and gelo.md. Rows tagged (Gemma) describe terms that only apply to the Gemma 4 architecture target.

Term	Short	Meaning
Prefill	—	The single forward pass over the full prompt (system message + retrieved RAG context + user question) that populates the KV cache for all subsequent decode steps. Compute-intensive, attention-dominated past ~2k tokens; the "long" workload of the two phases.
Decode	—	Per-token autoregressive step. The new token's query attends to all cached K, V; one new K/V pair is appended. Per-step compute is small, dispatch-bound; the "narrow" workload of the two phases.
Time-to-first-token	TTFT	Latency from request arrival to the first output token. Dominated by prefill cost. The user-visible "is the model alive yet" number.
Time-per-output-token	TPOT	Steady-state inter-token latency once generation starts. Dominated by decode cost. The user-visible "how fast does it read" number.
KV cache	—	The materialised `(K, V)` pair for every previously-seen position, per layer, per head. Lives in encrypted CVM DRAM; sized `2 · n_layers · n_kv_heads · n_cache · d_head`. ~58 MB for Qwen3-1.7B at `n_cache=4k` (28 layers · 8 KV heads · 128 head_dim); ~2 GB-class for Gemma E4B at long context. The heavy state of the decode loop.
QK-norm	—	Per-head RMSNorm applied to `Q` and `K` before RoPE. Introduced by Qwen3 as the only Qwen3-vs-Qwen2 architectural delta: each Q-head's `(head_dim,)` slice is normalised against `self_attn.q_norm.weight`, ditto K against `k_norm.weight`. Stays entirely in-TEE on plaintext tensors — no protocol surface change vs the embedder's RMSNorm pattern. Loader detects the tensors when present; absent on Qwen2/LLaMA/Mistral, so the step is a no-op for those families.
Grouped-query attention	GQA	Q has more heads than K/V; multiple Q heads share one K/V head. Qwen3-1.7B uses 2:1 (16 Q heads to 8 KV heads); Gemma 4 uses 8:1. Already handled unchanged by the existing decoder attention path; reused here.
Per-Layer Embedding (Gemma)	PLE	An additional embedding table introduced by Gemma 3n / Gemma 4 E-variants, shape `[vocab × n_layers × d_ple]` (262 144 × 35 × 256 for E4B), indexed by `token_id` per layer rather than once at input. Address-bus-leaking under our threat model — token-id-keyed gathers on a GPU reveal the prompt in plaintext with no inversion needed. The fix is structural: the table lives in CVM DRAM; the TEE performs the gather; only the public projection matrix that lifts the gathered row to `d_hidden` ever sees GPU. Qwen3 has no PLE — the box stays unused on Qwen3.
Hybrid attention (Gemma)	—	The Gemma 3 / Gemma 4 design choice that interleaves cheap local layers (sliding-window causal attention, window W tokens) with full-context global layers. E2B uses 4:1, E4B uses 5:1; the last layer is always global. Cuts per-token attention FLOPs ~3.86× at `n=8k` on E4B and shifts the bandwidth bottleneck onto the small minority of global layers. Qwen3 has no hybrid attention — every layer is full causal — so this dispatcher is a no-op on Qwen3.
Sliding-window attention (Gemma)	SWA	Causal attention restricted to the last `W` tokens (W = 512 on small Gemma 4). Reduces attention from `O(n²)` to `O(n·W)`. In-TEE-tractable at all our context lengths; does not compose with permuted attention because the band-diagonal mask is not permutation-invariant.
p-RoPE (Gemma)	—	Proportional RoPE. Rotation applied to the first `p · d_head` dimensions only; the remaining `(1−p) · d_head` dims pass through unrotated. Gemma 4 global layers use `p = 0.25`. One-line change to the existing RoPE kernel — defaults to full-rotation (`p = 1`) when not set, matching Qwen3.
K = V tying (Gemma)	—	Gemma 4 global layers materialise K and V as the same tensor (the "K equals V" trick). Halves global-layer KV-cache memory and lets the protocol sample one mask, one permutation, or one OutAttnMult partition for both — not two. Local layers keep separate K, V. Qwen3 has separate K and V everywhere.
Address-bus leak	—	An attack class where an observer on the memory or PCIe bus learns content from which addresses are accessed, not what is read at them. Defeats mask-based confidentiality on table lookups (PLE, MoE expert selection, embedding gather) because the index, not the value, is the secret. Mitigation here: every secret-indexed table read happens in-TEE; only public-key projections cross PCIe. Empirically validated by the TEE.Fail DDR5 bus-interposer result (Oct 2025), which is precisely the threat the per-batch-fresh-mask architecture is designed to survive — even if mask state is later extracted offline, no live forward pass is recoverable.
Streaming output	—	Per-token emission over the open RATLS connection as soon as each token is sampled, rather than batched-at-EOS. Surfaces TPOT to the user but exposes per-token timing on the wire — covered in §03.
SCX KV-cache encoding	SCX	Stateless per-user KV-cache encoding scheme (Yuan et al., SIGCOMM 2025) that derives the encoding key from `(session_id, layer_id, position)` at write time and reads the encoded K, V back without per-step re-permutation. Candidate decode-phase primitive; complements permuted attention on prefill. Not yet adopted; evaluated against the threat model when the decode bench lands.

§03

Threat model#

what each component sees · what it cannot · what's new vs embedding/rerank

Same trust posture as embedding and reranking: hardware-attested TEE on CPU, information-theoretically blind GPU, everything else hostile. The openweight assumption holds — model weights pin to a public HuggingFace revision (Qwen/Qwen3-1.7B today, google/gemma-4-* when the Gemma decoder integration lands); the SHA-256 of the loaded bytes rides as model_identity through the attestation report. Generation extends the model with three new exit channels (KV cache, per-token streaming, plus the Gemma-only PLE gather); each is closed structurally rather than statistically.

Component	Trust	What it sees	What it does NOT see
User-side prompt	confidential	—	—
TEE (SEV-SNP CVM)	trusted	prompt · retrieved chunk text · per-layer activations · scores · sampled tokens · KV cache · PLE table · mask state · model weights	—
GPU + driver + PCIe	untrusted	public model weights · per-batch masked activations `U = A·H` · public PLE projection matrices · integrity-probed matmul results	clean activations · mask `A` · prompt tokens · sampled tokens · PLE table contents · PLE gather indices · KV cache
TEE host (CVM operator)	untrusted	encrypted CVM memory · masked PCIe traffic · attestation evidence · per-token TLS write timing · prompt length (via TLS frame size) · generation length	plaintext memory · per-CVM encryption key · token content · KV-cache bytes · scores · routing
Network operator	untrusted	TLS-wrapped requests · attestation evidence · per-token TLS frame timing	RATLS contents

Six exit-channel leaks specific to generation, each closed by construction:

Leak	How it's closed
PLE address-bus leak (Gemma · P0)	The full `[262 144 × n_layers × 256]` int8 PLE table is provisioned into the `InProcessTrustedExecutor`'s encrypted CVM DRAM at model load. Per-token row selection (`PLE[token_id, layer_idx]`) happens inside the TEE; only the resulting `(n, d_ple)` tensor — projected up to `d_hidden` via the public per-layer projection matrix — ever crosses PCIe, and it crosses under the same per-batch mask `A` as the rest of the activation stream. A PCIe observer sees no token-id-keyed gather pattern. Verified by `tests/ple_pcie_leak.rs`, an attacker-simulator that asserts zero PLE-side reads on the offload trace. Qwen3 has no PLE, so this exit channel doesn't exist on Qwen3; the substrate piece is staged ahead of the Gemma decoder integration.
KV-cache content leak	The KV cache for both local and global layers lives entirely in encrypted CVM DRAM, indexed by `(layer_idx, head_idx, position)`. The GPU never touches KV bytes — attention's `Q · Kᵀ` and `P · V` for in-TEE local layers run on CPU under AOCL-BLIS; for global layers the per-step `Q` goes out under mask, but `K` and `V` are pulled from cache, masked freshly, and never persisted GPU-side. Decode-phase π handling — fresh π each step versus session-persistent π — is the decision point that gates SCX-style encoded-KV adoption; see §10.
Per-token TLS timing	Streaming output flushes one ciphertext frame per sampled token over the RATLS connection. An on-path observer sees per-token timing; combined with public knowledge of the GELO+Gemma forward-pass cost, this is mild side information about token entropy (cache-friendly tokens decode faster). Mitigations available at deployment time: (a) emit at a fixed cadence padded by sleep; (b) batch tokens into fixed-size frames at the cost of latency; (c) disable streaming entirely and return at EOS. The protocol surface is unchanged by the choice.
Generation-length leak	Total token count is observable from TLS frame count and connection duration. Treated as out-of-scope at the protocol layer — same posture as the existing reranker's `k_max` being a function of the deployment, not the request. Fixed-length deployments (always emit exactly `n_max` tokens, padded with EOS-after-stop) are available as a wrapper around the same generation loop.
Sampler-state leak	Sampling (greedy, top-k, top-p, temperature) runs entirely in-TEE on plaintext logits. The RNG is a ChaCha20 seeded from the per-session `SessionKey`; the GPU sees neither the logits nor the sample. Logit values are zeroized after sampling; only the sampled `token_id` survives to the next step and to the KV-cache append.
Tied-LM-head dot-product leak	The output projection is `H_final · token_embeddingᵀ` over the 262 144-row vocabulary. The mask GEMM at this shape is unchanged from the reranker's tied-LM-head pattern — masked offload of the same form as every other linear. The vocab-row gather that follows sampling (next-step input embedding) is again token-id-keyed and stays in-TEE for the same reason PLE does.

Not covered. Total prompt length and total generation length are visible on the wire and are treated as request metadata. Side channels in the TEE itself (cache timing, branch prediction) are the responsibility of the SEV-SNP hardware envelope. Whether a deployment chooses to pad-and-batch tokens to obscure per-token timing is a deployment policy, not a protocol property.

Validating result. The TEE.Fail DDR5 bus-interposer attack (Oct 2025) defeats SGX, TDX, and SEV-SNP key sealing in the offline case and uses extracted CPU-TEE keys to compromise H100 confidential GPU. The architectural choice that survives it is the protocol's per-batch fresh Haar mask: mask material exists only for the duration of one forward pass; an offline TEE compromise reveals nothing about any past forward and nothing about any future one until a fresh mask is sampled. Generation inherits this property unchanged — every prefill is one forward pass, every decode step is one forward pass. Today's Qwen3 measurement validates the property end-to-end: InProcessTrustedExecutor::with_seed samples a fresh Haar A at the start of each generate() step and discards it at end_forward_pass; the same code path runs unchanged on Gemma 4 once the decoder integration lands.

§04

Supported workloads & architectures#

the two phases of autoregressive generation · Qwen3 family

Workloads — prefill & decode

Generation has two phases with structurally different compute shapes and structurally different cost profiles. The protocol covers both, but with different attention paths and different cost trade-offs. The runner exposes them as a single generate(prompt, max_tokens) entry point that does prefill once and then loops decode.

Phase	Shape	Dominant cost	Attention path	Mask cadence
Prefill	`n_q = n_prompt` (full prompt)	attention `O(n²)` at long context · linear projections at short context	length-based auto-switch: in-TEE for short n · fused permuted attention on the GPU past threshold · OutAttnMult is the 3-dispatch fallback before the fused kernel lands	one Haar `A` for the whole prefill forward
Decode	`n_q = 1`, `n_kv = n_cache` (growing)	dispatch overhead · KV-cache bandwidth · linear projections	in-TEE always. Per-step attention math is microseconds-scale at `n_q = 1`; the auto-switch threshold doesn't engage on decode. KV-cache bandwidth (orthogonal axis) is what SCX in §10 optimises.	one fresh Haar `A` per decode step (per-token)

Why prefill and decode want different attention paths. Prefill at n = 4k materialises a (heads, n, n) score tensor of ~1 GB per layer that the 3-dispatch permuted-attention path forces through device memory three times — bandwidth-bound to ~3.2 GB/layer × 28 layers ≈ 90 GB on the integrated-GPU dev box (~1.8 s pure traffic per prefill). The FlashAttention-style fused permuted kernel drops that to ~130 MB/layer and lands the workload within ~2× of unprotected baseline. Decode at n_q = 1 never materialises a square score tensor — per-step attention compute is microseconds and stays in-TEE; the bottleneck instead is the KV-cache bandwidth read at every decode step, addressed by the SCX-class encoded-KV primitive (§10) rather than by any attention-dispatch change.

Sampling. Greedy and top-p / top-k / temperature sampling all run in-TEE on plaintext logits. The seed is derived from the per-session SessionKey; deterministic replay (same prompt, same session) is possible at temperature = 0 for debugging and reproducibility benches without breaking confidentiality of the live request. Today ships greedy only — top-p / top-k / temperature wire in alongside the production HTTP route.

Model architectures

The shipping surface is the Qwen3 family — vanilla Qwen3ForCausalLM with GQA, full causal attention, SwiGLU FFN, tied input/output embeddings, and one Qwen3-specific addition: per-head QK-norm before RoPE. The same DecoderWeights loader and forward path that services Qwen3-Embedding under the embedder/reranker protocol handles generation. The SHA-256 of the loaded safetensors bytes + tokenizer + the (q_norm, k_norm) presence flag rides as model_identity in every attestation report — a tokenizer drift or a re-numbered <eos> trips the attestation. Gemma 4 E2B / E4B remain the architecture target for the next phase — see the status block below.

Qwen3 family — shipping target

Variant	Layers	Hidden · Inter · d_head	Q · KV heads	Attention	RoPE θ	QK-norm	Status
Qwen3-1.7B	28	2048 · 6144 · 128	16 · 8	full causal	1 M	yes	running — primary target
Qwen3-4B	36	2560 · 9728 · 128	32 · 8	full causal	1 M	yes	stretch — same code path

Per-head QK-norm before RoPE. Each layer's self_attn.q_norm.weight and k_norm.weight are (head_dim,) RMS-norm gammas applied to each attention head's Q / K slice. The tensors are loaded Option<Array1<f32>> on DecoderLayerWeights; the forward path invokes rms_norm::apply_qk_norm only when populated, so Qwen2 / LLaMA / Mistral checkpoints continue loading byte-identically. Stays entirely in-TEE — the QK-norm step is a no-op for the protocol.

Qwen3.5 family is out of scope. The 2B / 4B / 9B / 35B-A3B variants ship as Qwen3_5ForConditionalGeneration — multimodal VLM with a 24-layer ViT, image / video tokens, and a text backbone using GDN-style linear_attention (Mamba/SSM) layers in 3:1 hybrid with full_attention, MRoPE (interleaved mrope_section), partial_rotary_factor: 0.25, an MTP head, and attn_output_gate. Strictly harder than Gemma 4, not easier.

Gemma 4 — next architecture target — click to expand: variants, developed protocol surface, decoder blockers

Gemma 4 is the architecture north star for the next phase. Every protocol primitive needed to run it has shipped and is tested on synthetic weights — what is missing is the Gemma-specific decoder refactor (a closed list of items, none of which affect the protocol surface). Full handoff in gemma4-architecture-roadmap.

Variant	Layers	Hidden · Inter · d_head (local / global)	Q · KV heads	Attention	RoPE θ (local / global)	PLE	Status
Gemma 4 E2B	35	1536 · 6144 · 256 / 512	8 · 1	4:1 local/global · W=512	10 K / 1 M	262 144 × 35 × 256 · int8	scaffolded · blocked on decoder refactor
Gemma 4 E4B	42	2560 · 10 240 · 256 / 512	8 · 2	5:1 local/global · W=512	10 K / 1 M	262 144 × 42 × 256 · int8	scaffolded · blocked on decoder refactor

Developed (protocol surface, complete)

AttentionClass::{Local{window}, Global} + per-layer attention-class vector on DecoderConfig
causal_gqa_attention_swa_cached(window, q_pos_offset) band-mask SWA kernel for local layers
HybridAttentionRouter dispatcher selecting per-layer kernel based on class
RoPE::apply_partial_at(rotated_dim) for p-RoPE; even-snap rotation on first p · head_dim
KvCache::new_with_sharing + Separate/Shared variant — halves K=V global-layer memory
final_logit_softcapping on DecoderConfig; wired through compute_logits
PleTable (int8 storage, dequant on gather) + TrustedExecutor::provision_ple_table / ple_gather
tests/ple_pcie_leak.rs — SpyEngine attacker-simulator confirms zero PLE-keyed PCIe activity
Gemma4Variant::{E2B, E4B} config builders pinned to real HF constants (audited 2026-05-18)

Blockers (Gemma-specific decoder refactor)

Per-class head_dim — Q / K / V projection shapes diverge per attention class (local 256, global 512); requires per-layer-class projection plumbing
Per-class rope_theta — two RoPE bases per model (local 10 K, global 1 M); needs class-aware RopeTables
Cross-layer KV sharing — num_kv_shared_layers reuses an earlier layer's KV instead of computing its own (distinct from within-layer K=V tying)
GeGLU dispatch — Gemma uses gelu_pytorch_tanh not silu; decoder/swiglu.rs needs an activation kind selector
use_double_wide_mlp semantics — exact FFN structure needs HF transformers source check
AltUp residual stream — alternating-update variant inherited from Gemma 3n
Gemma-aware safetensors loader — model.embed_tokens_per_layer.weight, per-layer PLE projection tensors, PLE int8 dequant scale layout

Estimate: ~4-5 weeks of decoder refactor (no protocol changes). All work tracked in gemma4-architecture-roadmap.

Deferred

Gemma 3n — same PLE-in-TEE-DRAM mechanism would apply; superseded by Gemma 4 E2B and not a target.
LLaMA / Mistral dense — already loadable via the same DecoderWeights path used by Qwen3-Embedding; trivially extensible (no PLE, no hybrid attention, no p-RoPE, no QK-norm). Add a variant when needed.
MoE generation — Qwen3-MoE, DeepSeek-V2, Gemma 4 MoE 26B-A4B: routing histogram is an orthogonal exit channel; needs CryptoMoE-style balanced dispatch as a separate ~3-6 month workstream.

§05

Components#

crate-side · what's in the codebase today

Each card pairs [component ↦ source ↦ what it does ↦ status]. Components scoped to the Qwen3 shipping path. Architecture-scaffold components for the Gemma 4 target (Gemma4Variant, PleTable) are landed and synthetic-tested in the crates; see §04 "Gemma 4" details for status.

model

Qwen3Variantgelo_embedder :: decoder :: qwen3

Variant config builder. Q1_7B and Q4B pin HF-verified constants (layer count, hidden / intermediate / head dims, GQA layout, RoPE θ, max_position, vocab) and emit a DecoderConfig. The HF model ID is exposed via hf_model_id() so re-pinning is one line.

file: decoder/qwen3.rs
tests: q1_7b_config_matches_real_hf_config · q4b_constants_pinned · hf_model_id_is_stable
used by: tests/qwen3_generation_e2e.rs · tests/qwen3_hf_parity.rs · tests/qwen3_generation_bench.rs

loader

DecoderWeightsgelo_embedder :: decoder :: weights

Safetensors loader for the Qwen3 / Qwen2 / LLaMA / Mistral family. Reads token_embedding, final_norm, and per-layer {q,k,v,o}_proj, {input,post_attention}_layernorm, mlp.{gate,up,down}_proj. Bf16 and f16 are dequantised to f32 on load. q_norm / k_norm read as Option<Array1<f32>> — populated for Qwen3, None elsewhere. SHA-256 of concatenated shard bytes folds into the attestation model_identity.

file: decoder/weights.rs
tokenizer: common/tokenizer.rs · HfTokenizer::{encode, decode, token_id}
back-compat: byte-identical for Qwen2 / LLaMA / Mistral (QK-norm tensors absent → step skipped)

attention

apply_qk_normgelo_embedder :: decoder :: rms_norm

Per-head RMSNorm helper for the Qwen3 QK-norm step. Views a (n_tokens, n_heads · head_dim) projection as n_tokens · n_heads rows of length head_dim and normalises each row against the shared (head_dim,) gamma. Stays in-TEE — invoked between QKV projection and RoPE on both prefill and decode-cached paths.

file: decoder/rms_norm.rs
callers: decoder::forward::{decoder_block, decoder_block_cached}
tests: apply_qk_norm_normalises_each_head_independently · apply_qk_norm_applies_gamma_per_channel

state

KvCachegelo_embedder :: decoder :: kv_cache

In-CVM KV cache. Per-layer LayerKvCache::{Separate, Shared} — separate K and V on the Qwen3 path. Grows by one position per decode step; reset / re-allocate per generate() invocation. Always RMP-protected CVM DRAM — never touched by GPU.

file: decoder/kv_cache.rs
Qwen3: all-Separate · 28 layers · 8 KV heads · 128 head_dim · ~58 MB at n_cache=4k
π handling: per-step decision (§10) — fresh π per step requires KV-cache-encoded primitive (SCX-class); session-persistent π trades one weaker decode-step argument for in-place cache reuse

attention

Cached attention kernelsgelo_embedder :: decoder :: attention

Asymmetric Q vs cached-KV attention. causal_gqa_attention_cached(n_q, n_kv, q_pos_offset) handles both prefill (n_q = n_prompt, n_kv = n_prompt) and decode (n_q = 1, n_kv = n_cache + 1) under one kernel.

kernels: causal_gqa_attention_cached
Qwen3 — current shape: all-Global path; OutAttnMult auto-switch at n ≥ hidden_size = 2048 · in-TEE at decode shapes

loop

generate / run_prefill / run_decode_stepgelo_embedder :: decoder :: generation

Orchestrates prefill + decode. run_prefill opens one begin_forward_pass, walks every layer with cache writes, returns the per-token hidden state. run_decode_step opens its own begin_forward_pass(1) per call, embeds the previous sampled token, walks layers (each appending one position to the cache), returns the last-layer hidden row. generate ties these together with a sampler. One fresh Haar A per forward pass — every decode step gets its own.

file: decoder/generation.rs · decoder/forward.rs
LM head: tied — h_last · token_embedding.T in-TEE
sampler: greedy today (SamplerConfig::Greedy); top-p / top-k / temperature share the same call-site, in-TEE on plaintext logits
EOS: GenerationConfig::eos_token_ids · short-circuit when sampled token matches

protocol

InProcessTrustedExecutorgelo_protocol :: sim

The masked executor — paper-parity defaults via with_seed: per-forward Haar A + shield(8, 4.0), no U-Verify. with_verify_probes(k) enables Freivalds-style integrity probes on top. Holds a WeightStore shared with the offload engine, a ChaCha20Rng session-seeded, and (optionally) a PleTable in encrypted CVM DRAM.

file: crates/gelo-protocol/src/sim.rs
session bracket: begin_forward_pass(n) samples the per-forward A · end_forward_pass() drops it
per-offload mode: opt-in with_per_offload_mask() for parity / BSS-recovery tests · not the production path

§06

Compute flow & trust boundaries#

prefill once · decode loop · per-layer block detail

Two figures trace one Qwen3-1.7B decoder block under each phase — the generation path running today. FIG. 02a covers prefill (full prompt, full causal attention O(n²)). FIG. 02b covers one decode step (n_q = 1, growing KV cache, attention O(n_cache · d_head)). The Gemma 4 variants (PLE gather, hybrid local/global dispatch, p-RoPE) are scaffolded but unused on this path; they're in the collapsible block below for comparison.

FIG. 02a — Qwen3-1.7B prefill · one decoder block · pre-LN · SwiGLU · GQA(16:8) · full causal attention · QK-norm before RoPE · 2026-05-18 · 4 GPU GEMMs per block · 28 blocks total blue arc = residual · solid red = TEE flow · dashed amber = masked PCIe transit (mix → / unmix ←)

FIG. 02b — Qwen3-1.7B decode · one step · n_q = 1 · n_kv = n_cache · same block topology · attention reads from CVM-resident KV cache · 2026-05-18 · one fresh Haar A per decode step same legend · decode step is shape-equivalent to a 1-token prefill against the cache

What to look for in the Qwen3 figures. ① Single attention class — no hybrid router, every layer is dense causal; OutAttnMult auto-switch decides offload-vs-in-TEE on each forward based on n vs hidden_size = 2048. At decode (n_q = 1) attention always stays in-TEE. ② QK-norm step (green) is the only Qwen3-specific addition vs Qwen2/LLaMA — per-head RMSNorm on Q and K, applied entirely in-TEE between the QKV projection and RoPE. ③ Per-block GEMM count is 4 (QKV merged, O, gate∥up merged, FfnDown) × 28 blocks = 112 GPU dispatches per forward pass. Each rides a single per-forward Haar A. ④ Decode-step parity — argmax-stable token agreement across PlaintextExecutor / InProcessTrustedExecutor / full-stack-with-U-Verify confirms the protocol surface is bit-stable at this scale.

Gemma 4 reference diagrams (scaffold · click to expand) — PLE gather, hybrid attention router, p-RoPE, K = V tying

The original Gemma 4 prefill / decode figures are kept here for design reference. They show the additional pieces the protocol substrate carries for the Gemma 4 target: a TEE-only PLE gather (green), a hybrid attention router that branches per-layer between sliding-window-in-TEE and global-offloaded, p-RoPE rotation on global-layer Q/K, and the K = V tying that lets the GPU upload one tensor instead of two on global layers. Until Gemma 4 weights load (see gemma4-architecture-roadmap), these paths are passthrough no-ops on the running Qwen3 path above.

FIG. 03a — Gemma 4 prefill · one decoder block · pre-LN · SwiGLU · GQA(8:1) · hybrid attention (local-SWA + global) · PLE gather in-TEE · rev 2 · 2026-05-18 · 4 GPU GEMMs per block blue arc = residual · solid red = TEE flow · dashed amber = masked PCIe transit (mix → / unmix ←) · green = PLE gather (TEE-local)

FIG. 03b — Gemma 4 decode · one step · n_q = 1 · n_kv = n_cache · same block topology · attention reads from CVM-resident KV cache · rev 2 · 2026-05-18 · 4 GPU GEMMs per block same legend · one fresh Haar A per decode step · π handling decided at §10

What to look for between the two Gemma figures. ① Per-block topology is identical — both phases dispatch 2 matmul_many + 2 matmul + 1 attention per block, repeated × 35 (E2B) or × 42 (E4B). What differs is the input shape of every GEMM: (n_prompt, d_hidden) in prefill, (1, d_hidden) in decode. The protocol does the same work; the wall-clock cost regime is what flips. ② PLE gather happens in both phases, on different shapes: n_prompt rows in prefill, one row per decode step. Both are TEE-only — green border on the box, never crosses PCIe. ③ KV cache is written by prefill (full prompt at once) and appended by decode (one position per step). Lives in encrypted CVM DRAM in both phases. ④ Attention dispatch diverges by layer class, not by phase. Local layers stay in-TEE in both phases (CPU under AOCL-BLIS); global layers cross PCIe under mask in both phases. ⑤ Dispatch overhead dominates decode (every block's four (1 × d) · (d × d) GEMMs and one Q · Kᵀ_cache cache-read sum to a tight per-token budget); attention compute dominates prefill at long context (a single global-layer Q · Kᵀ is the largest GEMM in the whole stack).

Boundary	What crosses	What does not
PCIe (TEE ↔ GPU), prefill	`U = A·H` per masked offload · public weights · public PLE projection matrices	clean activations · mask `A` · prompt tokens · PLE gather indices · PLE table contents · KV cache
PCIe (TEE ↔ GPU), decode step	one-row `U = A·H` per masked offload · public weights · Q · Kᵀ_cache under mask on global layers only	sampled token · KV-cache bytes · logits · sampler RNG state
CVM ↔ Host RAM	encrypted CVM pages · SWIOTLB DMA bounce buffers · masked activation slabs only	plaintext memory · PLE table · KV cache · per-CVM encryption key
Network (TEE → client), streaming	per-token AES-GCM `(nonce, ciphertext)` frame · per-token TLS write timing	token content · token logits · sampled distribution shape
Network (TEE → client), batched	single AES-GCM bundle of the full continuation	per-token timing · token content

Mask cadence in the generation loop

The protocol defaults from embedding and reranking carry over unchanged: per-forward-pass Haar mask sampling with shield rows (k = 8 at energy 4·mean‖h‖), and per-offload shield freshness. The new dimension is generation:

Prefill is one forward pass — one Haar QR, then the same A rides every masked offload in the block stack. Same cost amortisation as the embedder.
Each decode step is one forward pass — one fresh Haar QR per token. At d_hidden = 2048 (Qwen3-1.7B), the per-step QR is small enough to not dominate; at d_hidden = 2560 (Gemma 4 E4B once the decoder integration lands), HKDF-derived mask material (§10 lever) can amortise it across steps for free if the security analysis confirms equivalence.
π in permuted attention is the open decode-phase decision: fresh π per step requires SCX-style encoded-KV (§10); session-persistent π enables in-place cache reuse but trades one weaker decode-step argument. Documented in §10, gated on the security spike when the bench harness lands.

Why per-forward + shield remains the right default. Per-offload Haar sampling would cost ~96–140 QRs per forward at Gemma 4 shapes; at d = 2560 each QR is ~20–30 ms, so per-offload mask sampling alone would dominate every other cost in the system. Per-forward + shield is the paper's §3.2 + §4.2 trade and what the executor ships by default — see reranking §05 for the per-cadence breakdown that motivates it.

§07

Performance & correctness#

Qwen3-4B production state · 2026-05-26

End-to-end measurement of the GELO protocol on real-weight Qwen3-4B generation — fp16 wgpu engine, LM-head GPU-offloaded under the same masked-offload boundary, mask family auto-dispatched by pad ratio (HD₃ at pad ≤ 1.6, DCT-IV cascade above), and the CPU DCT-IV path running as a tile-fused six-stage cascade entirely in L2 per column tile.

Qwen3-4B · current measured throughput across shapes

Five shapes spanning the Auto-family decision space, ordered by pad ratio. Auto picks the mask family that minimises GPU pad penalty: HD₃ when s_pad / s ≤ 1.6 (zero-pad to next pow2 is cheap), DCT-IV cascade otherwise. The DCT-IV path runs all three DCT stages and three diagonal multiplications fused over 16-column tiles so the whole cascade lives in L2 per tile — no inter-stage DDR5 round-trip — which cut prefill wall by 22 % at the production-extraction shape vs the prior 3-stage path. Single-sample cells; long-n run-to-run variance ~7 %.

Shape (B · n)	pad ratio	Auto family	Prefill wall (s)	Prefill agg. tok/s	Decode wall (s)	Decode agg. tok/s	Decode tok/s · seq
8 · 3 500 · long-n HD₃	1.17	HD₃	289.09	96.85	80.64	3.17	0.40
8 · 320 · short-n HD₃	1.56	HD₃	24.22	105.7	26.82	9.55	1.19
1 · 2 561 · single-stream	1.59	HD₃	31.45	81.43	22.28	1.44	1.44
8 · 2 400 · crossover band	1.70	DCT-IV	169.76	113.10	61.42	4.17	0.52
8 · 2 048 · production extraction	1.99	DCT-IV	135.13	121.25	55.44	4.62	0.58

Adapter: AMD Radeon 8060S (Strix Halo, RADV gfx1151, iGPU), Mesa Vulkan, fp16 wgpu engine. Model: Qwen/Qwen3-4B bf16 weights, GPU-resident post-provision (host RSS ~ 2.4 GiB residual = token-embedding + layer norms + config; per-layer weight Arcs released after VRAM upload). Workload: identical-length prompts per batch, greedy, K = 32 decode tokens, R3 LM-head GPU offload on (paper- parity per-forward fresh mask + shield(8, 4.0)). Bench code: crates/gelo-gpu-wgpu/tests/qwen3_m1_12_r1_q1_microbench.rs; driver: scripts/measurement-gaps-sweep.sh.

Per-bucket attribution — production shape

Prefill wall at B = 8 · n = 2 048 (pad 1.99, DCT-IV cascade) decomposes into four buckets. Decode at the same shape is dominated by in-TEE attention (the DDR5-bandwidth-bound per-step GQA kernel over the cached n_kv ≈ 2 056 prefix).

Bucket	Prefill share	Decode share	Reach for the lever
CPU mask (DCT-IV cascade)	20.4 %	~4 %	Cascade landed (was 38.0 % pre-2026-05-26); bf16 inner kernel remains as a follow-up.
GPU matmul (QKV / O / gate / up / down)	~52 %	~38 %	Iso-engine on iGPU — DDR5-shared; ceiling lifts on dGPU (HBM, ~40× kernel-bandwidth headroom).
In-TEE attention (GQA over `n_kv`)	~16 %	53.9 %	Iso-engine on iGPU — moving it to GPU regressed 16× at this shape (memory-bound kernel on shared DDR5). dGPU with persistent K/V on HBM is the path.
Shield + strip + misc (rmsnorm, qk-norm, RoPE, SwiGLU, residuals, kv scatter)	~7 %	~3 %	Each individually below the variance floor; end-to-end bf16 activations compress these incidentally.

compute_logits (R3 LM-head GPU offload) lands at 4.2 % of decode wall under the masked-offload boundary — nesting inside the engine matmul span listed above, not double-counted. Pre-R3 the in-TEE LM-head was 46-58 % of decode wall; the GPU offload cut it 97.6 % at B = 8 · K = 64.

Cascade headline — what shifted the production-shape number

At the production shape the prior 3-stage DCT-IV path paid six full-buffer DDR5 traverses per apply / unapply (one stride-d copy plus a full-tensor diagonal multiply per stage). The tile-fused cascade loads a 16-column tile once, runs all three DCT stages and all three diagonal multiplications inside L2, then writes the tile out — pricing the cascade at one copy-in plus one copy-out per tile.

`B = 8 · n = 2 048` (pad 1.99)	3-stage path (pre-cascade)	Tile-fused cascade	Δ
Prefill wall	174.92 s	135.13 s	−22.7 %
Mask bucket (apply + unapply)	66.5 s · 38.0 %	27.57 s · 20.4 %	−58.5 % bucket
Aggregate prefill tok/s	93.7	121.3	+29.4 %

A second DCT-IV cell at the crossover band (n = 2 400, pad 1.70) shows the same picture (prefill 216.15 s → 169.76 s, −21.5 %; bucket −53.1 %). At long-n HD₃ shapes (n = 3 500, pad 1.17) the cascade is a no-op — the prefill wall stays at 289.09 s (HD₃ shapes don't run the DCT-IV path).

Why decode tok/s is not on the prefill curve

The headline iGPU ceiling is the in-TEE attention bucket at decode. At B = 8 · n = 2 048 it owns 53.9 % of decode wall (rising to 63 % at B = 1 · n = 2 561 and 66 % at B = 8 · n = 3 500) — a memory-bound per-step GQA kernel reading the cached n_kv prefix. Lifting it to iGPU compute regressed 16× at the production shape (the bus that feeds the in-TEE kernel is the same DDR5 the CPU mask cascade already saturates). The 40+ tok/s decode target lives on dGPU substrate: HBM ~3 TB/s kernel-side reads (~75× headroom over shared DDR5) plus persistent K/V on device so each decode step uploads only the new K_t, V_t row (32 KB) instead of the full 256 MB cached prefix. Compressing the decode bucket on iGPU isn't gated on engineering — it's gated on the memory hierarchy.

§08

Optimizations#

mask-family alternatives · long-context measurement · trade-offs

The §07 short-prompt run is cheap on the Haar QR sample and the mask GEMM. Production prefill at n ≥ 2k flips that picture: the QR alone takes seconds. The current executor retires Haar in favour of HD₃ (A = D₃·H·D₂·H·D₁·H) — an exactly-orthogonal cascade of three sign-flips around three in-place Walsh-Hadamard transforms, no QR, mask never materialised, O(s · log s) per column. The Walsh-Hadamard matrix only exists at power-of-two side length, so the executor leans on the shield rows it was going to add anyway: rather than fix k_shield = 8 and pad internally to next_pow2(n+k) with zero rows, it picks k directly such that n + k already lands on a power of two. This unifies the mask path under HD₃ at every shape.

HD₃ — shield-as-padding to the next pow2#

The shield row count k_shield is a free parameter above its paper-mandated floor (k ≥ 8). Excess shield rows are monotonically safer per GELO §4.2's shield-energy argument, so the executor uses them as structural padding: pick k = next_pow2(n + k_base) − n (where k_base = 8). The stacked operand n + k is then exactly a power of two for every n; HD₃ applies with no internal zero-pad; the GPU sees exactly the rows the GELO protocol needs and nothing more.

The cascade itself is three sign-flips D_i ∈ {-1, +1}^s (drawn fresh per forward pass, 3·s bits, sub-microsecond) interleaved with three in-place FWHTs. Kernel: radix-8 AVX-512 fused butterflies (three stages per pass), AVX-2 + scalar fallbacks, rayon-parallel butterflies within each stage.

Shield-padding under batching

The formula generalises directly to batched forwards. At batched decode each step has n = B (one row per sequence) and k = next_pow2(B + 8) − B lands the stacked axis on a power of two at every batch size. k stays ≥ 8 (the paper floor); worst-case k = 2·k_base − 1 = 15. At batched prefill the same formula applies per-sequence block; each block carries its own shield rows sized to the per-sequence n_max + k target.

B (batched decode)	k (shield rows)	stacked axis	HD₃ pow2?
1 (single-stream decode)	15	16	✓
8	8	16	✓
12	20	32	✓
24	8	32	✓
56	8	64	✓

vs the old Haar baseline

Haar samples a fresh dense orthogonal A ∈ O(s) per forward via Householder QR and applies it with a dense GEMM. HD₃ retires the QR bucket entirely and drops the apply / unapply from O(s²·d) to O(s·d·log s), with mask storage shrinking from O(s²) to O(s) (three sign vectors). At Qwen3-4B prefill (n = 2 040, s = 2 048 exact pow2) HD₃ TTFT is 16.29 s vs ~25.6 s Haar at the same shape (−36 %).

DCT-IV cascade — tried, deprecated. An earlier design avoided HD₃'s pow2 requirement with A = D₃·C·D₂·C·D₁·C using DCT-IV (Bluestein chirp-z embedding to handle any s). It cut the non-pow2 prefill TTFT from 32.01 s (HD₃-with-internal-pad) to 22.79 s at n = 2 048, but at the cost of a second mask family, an external DCT planner cache, and a ~3–4× per-call CPU multiplier vs HD₃ at the same s. The shield-as-padding design above subsumes it: one mask family, one kernel path, HD₃ pow2 alignment at every shape by construction. DCT-IV stays in the source tree under MaskKind::Dct4 for parity tests but is no longer the Auto-dispatch target.

Cost vs Haar

op	Haar	HD₃ (current)
sample (per forward)	`O(s³)` Householder QR	`O(s)` sign-bit draws
apply / unapply (per offload)	`O(s²·d)` dense GEMM	`O(s·d·log s)` FWHT
operand size to GPU	`s = n + 8`	`s = next_pow2(n + 8)` via shield-as-padding
mask storage	`O(s²)`	`O(s)` (three sign vectors)

Mask-family TTFT comparison — Qwen3-4B, n = 2 048

The non-pow2 single-stream prefill is the hard case — it's where the four mask strategies diverge most. Numbers below are Qwen3-4B on Strix Halo iGPU, AOCL-BLIS at GELO_BLIS_THREADS = 16, 2 048-token greedy prefill + 4 decode tokens.

Strategy	TTFT (ms)	vs Haar	Mask families in tree
Haar · dense QR	25 873	(base)	1 (Haar)
HD₃ · zero-pad to next pow2	32 012	+24 %	1 (HD₃)
DCT-IV · Bluestein at exact `s`	22 794	−12 %	2 (HD₃ + DCT-IV)
HD₃ + shield-to-pow2 (current)	~ 32 000	+24 %	1 (HD₃)

The Haar / HD₃-zero-pad / DCT-IV rows are direct measurements across the three mask paths that lived in-tree historically. The current shield-to-pow2 row is the active production design but shares the dominant GPU + mask-GEMM cost with HD₃-zero-pad at s = 4 096 — the shield Gaussians replace the zero rows that would otherwise pad the FWHT, paying a small shield-fill CPU overhead on top of an otherwise identical wall. DCT-IV remains in the source tree under MaskKind::Dct4 for parity tests but is no longer dispatched by MaskKind::Auto.

At pow2-aligned shapes (n = 2 040) HD₃ wins outright vs Haar at 16.29 s vs ~25.6 s (−36 % TTFT, see headline stat). The +24 % regression above is the cost the design accepts only at non-pow2 single-stream long prefill — a regime the batched substrate structurally avoids (the variable shield_k formula keeps batched decode and rerank-shape prefill on a pow2 stacked axis by construction).

How we got here. Haar shipped first (paper-parity); HD₃ replaced Haar at pow2 shapes for the −36 % TTFT win; HD₃ regressed +24 % at non-pow2 because the FWHT requires pow2 and zero-padding doubles the GPU operand; DCT-IV was added as a non-pow2 branch (−12 % via Bluestein chirp-z at exact s) which closed the regression but introduced a second mask family and a second kernel path; HD₃ + shield-to-pow2 retires DCT-IV by using the GELO-mandated shield rows themselves as the next-pow2 padding (k = next_pow2(n + k_base) − n), giving back one mask family + one kernel path at the cost of restoring the +24 % non-pow2 regression. Worth it because the batched substrate moves the production regime away from non-pow2 single-stream prefill entirely.

Trade-offs

Mix / unmix kernels are less parallel than Haar's GEMM. AOCL-BLIS scales the dense Haar mask 5.04 × at threads = 16 vs threads = 1 on the s = 2 056 mask GEMM. The FWHT cascade chains log₂ s stages with a serial barrier between them; rayon parallelises butterflies within each stage but coarse-grained core scaling is well below a tuned GEMM. HD₃ still wins at long n because Haar's O(s³) sample cost dominates — but the bench-confirmed mask CPU subtotal gap is only ~2× at our shapes.
Pow2 pad on the stacked-with-shield operand — addressed by shield-as-padding. FWHT requires pow2 s. The current design picks k_shield = next_pow2(n + k_base) − n directly so s = n + k_shield already lands on a power of two for every n; no zero-pad rows enter the FWHT. Excess shield is monotonically safer per GELO §4.2. At single-stream long prefill (n = 2 048) this picks k = 2 048 shield rows; the GPU operand jumps to s = 4 096 and the mask GEMM cost doubles vs the historical DCT-IV branch — accepted as the cost of unifying on one mask family. At batched decode and rerank-shape prefill the formula keeps k in [k_base, 2·k_base − 1] (worst-case k = 15); there's no padding penalty there.
Threat model parity vs Haar is unvalidated for both alternatives. HD₃'s orbit is a discrete 2^{3·s}-element set inside O(s); DCT-IV's orbit is the same size but with a cosine-basis inner transform instead of Walsh-Hadamard. Both are structurally different from the continuous Haar measure that GELO's §3.2 / §4.3 BSS-hardness arguments are written against. QuIP# / QuaRot's incoherence proofs (Tseng et al., ICML '24; Ashkboos et al.) cover the Hadamard cascade for weight quantisation, not BSS-hardness against PCIe observation. Both HD₃ and DCT-IV ship opt-in only (with_hd3_mask() / with_dct4_mask() / with_auto_mask()); the executor default stays Haar until the AloePri + GELO §4.3 attack drivers show parity at the bench's shapes for whichever family is the candidate default. Harness in evals/aloepri-attacks/; gate documented in §10. Design analysis covering the Auto-dispatch design, the block-diagonal HD₃ design (ruled out by multi-anchor attack), and the DCT-IV branch design (accepted as Auto's non-pow2 branch) in docs/research/hd3-non-pow2-fix.md.

Protocol-side optimisations landed (cumulative)

Four optimisations landed between the §07 short-prompt baseline (2026-05-18) and the current §08 measurement (2026-05-20). Each is independent of the others; the bench cells above all reflect the cumulative state.

AOCL-BLIS multi-thread Haar mask GEMM (2026-05-19). The blas feature is now default-on in gelo-protocol; the bench pins GELO_BLIS_THREADS = 16 (the embedder default stays single-thread). Haar mask GEMM at s = 2 056, d = 2 048 measures 5.04 × faster than threads = 1. HD₃ and DCT-IV don't route through BLIS, so this is a Haar-only win.
HD₃ radix-8 FWHT + in-place apply on cached scratch (2026-05-19). Radix-8 fuses three radix-2 butterfly stages per pass (3 × less memory traffic); Hd3Mask::apply_in_place mutates the cached shield-stack scratch rather than allocating a fresh ~32 MB output per call.
HD₃ unapply_take + scratch reuse (2026-05-20). The unapply path previously allocated a fresh (s_pad, p) output and copied the engine output into it before running Aᵀ. MaskFamily::unapply_take now consumes the engine output buffer directly and runs the FWHT cascade in place via Hd3Mask::unapply_in_place / Dct4Mask::unapply_in_place. Measured at Qwen3-4B n = 2 040: gelo:mask_unapply bucket dropped 3 352 → 2 029 ms (−39.5 %), TTFT 18 490 → 16 566 ms (−10.4 %). Cumulative HD₃ + unapply_take vs Haar baseline at the same shape: −36 % TTFT.
DCT-IV cascade — tried, superseded by shield-as-padding (2026-05-20 / superseded 2026-05-21). An earlier non-pow2 branch added a DCT-IV cascade (D₃·C·D₂·C·D₁·C via Bluestein chirp-z) so HD₃ could skip its pow2 pad at long single-stream prefill. It delivered −12 % TTFT vs Haar at n = 2 048, but at the cost of a second mask family, an external DCT planner cache, and a ~3-4× per-call CPU multiplier. The shield-as-padding design above retires it: one mask family, pow2 alignment by construction, accept the long-context cost in exchange for substrate simplicity (and because the batched substrate is now the production regime). Design + multi-anchor security analysis preserved at docs/research/hd3-non-pow2-fix.md §6.

Quantized kernels#

Two GPU-side levers were spike-tested at the Qwen3-4B projection shapes against the same f32 baseline; both came back negative or marginal on AMD Strix Halo's gfx1151 iGPU. Plumbing-side work to wire them in is deferred until either the hardware changes (discrete GPU) or the kernel stack improves.

variant	QKV (2560→4096)	Gate∥Up (2560→9728)	FfnDown (9728→2560)	O (4096→2560)	verdict
Q4 weight quant (Vulkan, cubek-matmul)	0.82 ×	0.81 ×	0.71 ×	0.75 ×	regresses
Q4 + ROCm + rocWMMA (HIP)	0.75 ×	0.79 ×	0.73 ×	0.80 ×	regresses
f16 engine (Vulkan, shader-f16)	0.78 ×	0.94 ×	1.47 ×	0.91 ×	partial (FfnDown only)

Speedup expressed as f32 / variant (higher = variant faster). Q4 rel-err ~11-13 % (expected for naive Q4 without QuIP# hidden-axis rotation); f16 rel-err < 5e-4 across all shapes. Spike code: crates/gelo-gpu-wgpu/tests/q4_kernel_spike.rs, q4_hip_kernel_spike.rs, and f16_kernel_spike.rs. Plan + diagnosis at docs/plans/q4-gpu-weights.md §10. Diagnosis: gfx1151 supports v_wmma_i32_16x16x16_iu4 in hardware, but cubek-matmul 0.9's Q4 path doesn't emit the WMMA intrinsics on either runtime — Vulkan and HIP+rocWMMA produce identical ~0.75× speedups. f16 helps only where weight bandwidth dominates (large d_in like FfnDown's 9 728).

Batched forward + decode#

Single-stream decode at n_q = 1 is kernel-launch-bound on this iGPU: a prior spike confirmed no GPU strategy beats in-TEE attention at B = 1. The protocol restructures to put B sequences in flight at once, so GPU dispatch, GELO mask sampling, and per-block CPU work all amortise across the batch. Three architectural calls drive the design.

1. Mask topology by phase. At batched prefill each sequence carries its own (n_max+k)-row mask A_b — mathematically identical to the prior per-Rayon-worker model, so the per-row security argument carries over unchanged. At batched decode every sequence contributes one row, and the executor offers two topologies: a default per-sequence path (the same B independent masks, just at row count 1) and an opt-in shared dense A of size (B+k, B+k) that mixes all B current-token rows under one mask. The shared path is HD₃-aligned at every B via the variable-k formula k = next_pow2(B + 8) − B (always ≥ 8, the paper minimum). It defaults off until the AloePri c5_batched_decode_shared_a spot-check at B = 8 clears.

2. Per-block parallelism wherever the substrate allows. Mask apply / unapply, shield-row Gaussian fill, and in-TEE causal attention all rayon-iterate across the B sequence blocks. The shield-fill case needed extra work — the parent RNG can only be borrowed once at a time, so the executor pre-derives one Xoshiro256++ sub-stream seed per block before entering the parallel section. Each closure constructs its own local RNG from its seed; per-element distribution is unchanged, only the cross-block correlation structure differs (invariant for the shield-energy argument since shield rows of different sequences are independent by construction).

3. KV cache layout = (B, max_cache_len, kv_dim) per layer. Per-sequence valid lengths grow at their own pace under append_decode (one row per sequence per step) or append_prefill (variable prompt lengths). The attention call site reads each sequence's prefix slice independently, so SWA / global / Gemma-4 K=V-shared layers all carry over without change.

Decode-step result — synthetic Qwen3-4B-shape, B=8

Four-layer Qwen3-4B-shape weights (hidden 2560, 32 / 8 heads, head_dim 128, intermediate 9728), K = 8 decode steps over a random n_kv = 64 prefix, Strix Halo iGPU. Baseline is B serial single-stream decode steps (no concurrency, no Rayon — the acceptance baseline for the batched substrate). Three consecutive optimisations:

Stack	Wall (ms)	Per-step-per-seq (ms)	vs serial
Serial × B (baseline)	3 240	50.6	(base)
+ batched substrate (mask topology + KV layout)	751	11.7	4.31×
+ parallel shield-fill with sub-stream RNGs	680	10.6	4.80×
+ rayon-parallel batched in-TEE attention (final stack)	629	9.8	5.23×

Where the wall goes (post-optimisation)

Bucket	Serial	Batched	Batched share
GPU matmul (single + many)	2 408 ms	444 ms	70.9 %
GELO mask apply + unapply	580 ms	115 ms	18.5 %
Shield-row fill	181 ms	40 ms	6.4 %
In-TEE causal attention	89 ms	17 ms	2.7 %
Other CPU (RMSNorm, RoPE, SwiGLU, residual)	17 ms	11 ms	1.7 %

GPU dispatch count crashes 8× (the structural batched-substrate win); per-call GPU work grew only 1.5×, which confirms that single-stream m = 1 matmuls were heavily launch-overhead-bound. CPU side is fully amortised across the 32-thread Strix Halo — every remaining bucket is now rayon-parallel. With GPU at 70.9 % of batched wall, the next levers all sit outside the batched substrate: Q4 quantised GPU weights (cuts the dominant bucket), bf16 mask GEMM (deferred per the bf16-mask analysis above), and a batched GPU attention kernel (deferred — the in-TEE bucket is only 2.7 % so the kernel work isn't justified until longer-n_kv workloads appear).

LM-head GPU offload#

At decode the bf16 vocab × hidden dot-product that turns each last-layer hidden state into a logit vector ran in-TEE under a single-threaded loop. On Qwen3-4B (vocab 152 064, hidden 2 560) that loop measures 222 ms per token at greedy single-stream — 46 % of total decode wall — and scales linearly with decoded length. The lever is to register the tied-embedding transpose as one more VRAM-stationary weight and route the per-token projection through the existing masked offload path that the QKV / O / gate / up / down projections already use.

Design. Each call opens its own one-row forward-pass bracket (a fresh (1+k, 1+k) per-token mask, HD₃-aligned via shield-as-padding so the stacked operand is exactly 2^⌈log₂(1+k_base)⌉ = 16). The substrate masks the (1, hidden) last hidden, stacks with shield rows, ships to the engine for a (16, hidden) × (hidden, vocab) matmul, unmasks the (16, vocab) result, strips shield rows, and returns the (1, vocab) plaintext logits to the in-TEE sampler. Tied-embedding handling keeps the original (vocab, hidden) token-embedding host-resident for embedding_lookup; the registered LM-head weight is a separate (hidden, vocab) transpose. ~778 MB additional VRAM on Qwen3-4B; transient ~778 MB host RAM during the transpose materialisation, dropped once the wgpu upload consumes the Arc.

Threat-model delta. The GPU now observes a new per-token shape: a (1+k=16, 152 064) masked output under the same per-forward Haar / HD₃ mask. This is roughly 37 × wider on the output axis than any prior offload shape (QKV at (16, 4 096)); the GELO §3.2 BSS-hardness argument's quantitative bounds were written against narrower outputs. The path is the production default and only LM-head implementation in the generate loop (the prior in-TEE bf16 vocab × hidden loop was retired); the c6_lm_head_offload condition in the AloePri attack harness captures snapshots at this shape and runs the recovery-attack drivers against the c2 baseline — the c6 gate retroactively validates the default and remains the only unblocked step before treating this win as security-cleared. See §10 for the gate methodology.

Measurement — Qwen3-4B, single-stream greedy, K = 32 decode

Same fixture as the §08 per-op breakdown above. Both rows ran back-to-back against the same executor (projections + LM-head provisioned) with only the lm_head_via_gpu_offload flag toggled.

Cell	Total wall	tee:compute_logits	engine:matmul	Decode tok/s
In-TEE LM-head (baseline)	16.13 s	7 639 ms / 47.5 %	2 454 ms / 15.2 %	2.0
GPU LM-head offload	8.86 s	1 016 ms / 10.3 %	3 300 ms / 33.5 %	3.6
Δ	−45.1 %	−86.7 %	+846 ms (LM-head GEMM)	1.82×

The residual 1 016 ms in tee:compute_logits under the offload path is the profile::time wrapper measuring the begin / end_forward_pass bracket and the (1, vocab) output-row copy — the bulk math has moved to engine:matmul, which gains exactly the ~846 ms accounted for by the 32 new (16, hidden) × (hidden, vocab) dispatches (~26 ms each). Per-token cost flips from ~239 ms in-TEE to ~60 ms GPU-offloaded. Greedy token sequences match the baseline 32 / 32 on real Qwen3-4B weights through the full horizon — the mask round-trip noise stays below argmax discrimination threshold even with the per-step extra mask sampling.

PCIe bandwidth cost

Per decoded token at B = 8: upload (16, 2 560) f16 per sequence × 8 ≈ 640 KB; download (16, 152 064) f16 per sequence × 8 ≈ 78 MB. Over a 500- token decode that's an additional ~39 GiB of host↔device transfer per forward pass on top of the ~48 GiB the linear projections already move. Acceptable on UMA where transit is memcpy at DDR5 bandwidth; material on production discrete-GPU PCIe where the DMA is the floor. The follow-up lever is to accumulate K decode steps' hidden states and dispatch one wider LM-head matmul; not in scope today.

§09

Attack harness#

three conditions · twelve attacks · empirical evidence for the HD₃ attack-resistance gate

Our threat model is the GELO paper's §3.2 BSS-hardness game on a per-forward orthogonal mask. The PCIe-side attacker observes U = A · H (and, for some attacks, the engine output U · W) for each linear offload of each forward pass. Each forward draws a fresh A from either the dense Haar measure on O(s) or the HD₃ Hadamard cascade (§08); shield rows k_shield = 8 at energy 4·mean‖h‖ are appended to defeat Gram-leak attacks. The attacker does not see token ids on the wire (the embedding lookup is TEE-internal) and does not hold an obfuscated weight table (weights are shipped plain; the per-forward mask is the protocol's only secret). All attacks here recover either individual hidden-state rows H_i or row-Gram structure G_H = H·Hᵀ from U; success means privacy compromise on the prompt of that forward pass. 64-prompt smoke gate run: 2026-05-20.

Naming convention. Twelve attacks across two threat-model families: AloePri row-recovery (NN, IMA, ImaPaperLike, ISA, IA — port of the AloePri reference primitives against GELO's PCIe snapshots) and GELO §4.3 BSS-hardness (anchor_ica, JADE, JD, gram_error — direct probes of the mask's blind-source- separation hardness). Four AloePri families (VMA / TFMA / SDA / IMA-EmbedRow / IMA-paper-like-EmbedRow) emit not_applicable rows because they assume either a static obfuscated weight table or a token-id stream on the wire — GELO ships neither. One driver (isa_attn_score) is gated on the fused permuted-attention path and currently emits not_applicable until attention-score snapshots land in the protocol.

Conditions under test

Condition	Executor configuration	Role
C0 plain	`CapturingPlaintextExecutor` wrapping `PlaintextExecutor` — no mask, no shield	Control. Each attack must succeed against plain inference — if it doesn't, the attack itself is mis-calibrated and any defence reading is meaningless.
C2 default	`InProcessTrustedExecutor::with_seed(…)` — paper-parity defaults: per-forward-pass Haar `A` + `ShieldConfig::new(8, 4.0)`	The released GELO default. Acceptance threshold per paper §3.2 + AloePri-derived gate: TTRSR ≤ 10 % per recovery attack.
C3 HD₃	Same as C2 except `.with_hd3_mask()` — per-forward HD₃ Hadamard cascade + shield(8, 4.0)	The HD₃ Hadamard-cascade alternative. Must match C2 to within ±0.05 on every recovery attack to clear the default-flip gate.

Attacks under test

Twelve attacks across the two threat-model families. Paper expected tracks the AloePri reference's published numbers where the attack family overlaps. C0 / C2 / C3 columns are this harness against Qwen3-1.7B at the 64-prompt corpus (evals/aloepri-attacks/corpora/release-gate-64.txt, max 32 tokens per prompt; the harness also captures a C1 mask-only-no-shield row used for diagnostic acceptance-gate ordering checks, omitted from the per-attack table for readability).

What's covered. Per-condition: 64 prompts × 28 layers × 7 op-kinds = 12 544 snapshots. AloePri-family drivers (NN / IMA / ImaPaperLike / ISA / IA) target individual row recovery from masked operands. GELO §4.3 drivers attack the BSS-hardness directly:

anchor_ica — ridge LS + FastICA variants with adversary-known anchor rows. Skipped in this smoke run (single-snapshot FastICA wall-clock exceeds the 20 min budget; vectorised batching is the open follow-up).
jade — Cardoso–Souloumiac 1993 JADE port on the masked operand, Hungarian-aligned p95 cosine vs plaintext. Now batched + numba-JIT (§08 perf wing); 5.7 min for the full 3-condition matrix.
jd — Belouchrani 1997 SOBI adapted to multi-observation joint diagonalisation across T per-forward masks. Skipped in the smoke run (same per-stack iteration cost as serial JADE; batching is open).
gram_error — row-Gram side-channel probe, cos-normalised: ‖G_U/‖G_U‖_F − G_H/‖G_H‖_F‖_F. Range [0, √2], lower = more fingerprintable. Pad-invariant.

Attack	Scope	Reference value	C0 plain	C2 default (Haar)	C3 HD₃	Notes
NN cosine top-K against the plain embedding table; surface = layer-0 attn_norm operand	Recovers token id by nearest-neighbour search against the plain `W_e` table on a captured row. Standard embedding-inversion baseline (Morris et al., EMNLP 2023 "vec2text"); AloePri §F.1.	0.0 % (Q2.5-14B, T1)	100.0 %	4.5 %	4.2 %	C0 100 % (control ✓). C2/C3 well under the 10 % threshold; HD₃ marginally stronger than Haar. Pass.
IMA multi-α ridge inverter; layer 0; vocab-disjoint splits	Ridge regression mapping captured hidden-state rows → plain embedding rows, vocab-disjoint train/test. Linear-inverter family (Song & Raghunathan, CCS 2020); AloePri §F.1 baseline.	98.0 % (Q2.5-14B, T1)	96.4 %	1.5 %	1.5 %	C0 96.4 % (control ✓ at 64 prompts; smoke-corpus version at 8 prompts read 35.7 % — sample-size starvation, fixed by 64-prompt corpus). C2/C3 within 0.1 pt of each other and well under 10 %. Pass.
ImaPaperLike trained 2-layer 8-head transformer inverter; layer 0	Same surface as IMA but inverter is a trained transformer (port of AloePri `run_ima_paper_like`). Trained-inverter family (Morris et al., EMNLP 2023).	49–73 % (paper-scale corpus)	0.4 %	0.0 %	0.4 %	Driver broken — fails plain identity control. Confirmed root cause in the AloePri parallel investigation: 4 architecture variants × paper-default 2-epoch training budget all fail to push a 2048×2048 weight to ridge's closed-form solution. Not interpretable as defence; cite IMA-ridge instead. See `docs/handoffs/2026-05-20-ima-embedrow-transformer-investigation.md`.
ISA multi-α ridge at layer 23; vocab-disjoint splits	Same ridge primitive as IMA, but at layer 23 — deep residual stream after 22 layers of attention + FFN. Internal-state-attack family (AloePri §F.1).	0.0 % (Q2.5-14B, T1)	18.8 %	0.0 %	4.7 %	C0 18.8 % (control ✓ — deep ISA is naturally noisy). C2 effectively zero; C3 4.7 % vs C2 0.0 % is the largest within-family delta but stays well under the 10 % threshold. Pass.
anchor_ica FastICA + ridge LS with adversary-known anchor rows; max_features=256	Adversary-known anchor rows fix FastICA's permutation/sign ambiguity. Recovers non-anchor rows of `H` from `U`; p95 absolute cosine on recovered rows. FastICA primitive: Hyvärinen, IEEE TNN 1999; BSS framing: Cardoso, Proc. IEEE 1998.	≥ 0.95 (synthetic non-Gaussian mixtures, Hyvärinen 1999)	0.671	0.497	0.484	Values from the 8-prompt smoke run (64-prompt run skipped: per-snapshot sklearn FastICA × 3 K-values × 12 544 snapshots is single-core-bound). C2/C3 Δ = 0.013 (parity within ±0.05 band ✓). Both ~0.49 — well below the published ≥ 0.95 ICA-on-synthetic-mixtures baseline; FastICA's permutation/sign-recovery is disrupted by the per-forward mask. Pass at smoke scale. Batched FastICA is the follow-up for a 64-prompt reading.
JADE batched + numba-JIT joint-diag; max_features=256	Direct ICA on `U` via fourth-order cumulant joint-diagonalisation; Hungarian-aligned p95 cosine vs plaintext rows. Cardoso & Souloumiac, IEE Proc.-F 1993 "Blind beamforming for non-Gaussian signals" — the foundational JADE algorithm.	≥ 0.95 (8-source kurtotic mixtures, Cardoso 1993 Tbl. I)	0.704	0.575	0.574	C2/C3 Δ = 0.001 (parity well within ±0.05 band). Both ~0.575 cosine — well below Cardoso's ≥ 0.95 baseline on cleanly-separable non-Gaussian sources; mask + shield disrupts the cumulant signature ICA relies on. Pass.
JD Belouchrani SOBI adapted to multi-observation JD	Multiple `U_t = A_t·H` observations stacked under different per-forward masks; tests whether per-forward freshness keeps mask realisations independent. Flat `p95(T)` curve = freshness defence holding. Belouchrani et al., IEEE TSP 1997 "A blind source separation technique using second-order statistics" — the SOBI / joint-diag-of-time-lagged-covariances algorithm.	≥ 0.92 (correlated sources w/ T=20 lags, Belouchrani 1997 §V)	NaN	NaN	NaN	8-prompt smoke run returned NaN for all conditions — driver's whitening guard `d ≥ 2·s` fires at short prompts (n_data ≈ 10-13 stripped to data rows), so JD can't produce a meaningful demixing. 64-prompt run skipped on the same per-stack joint-diag bottleneck as anchor_ica. Batched JIT-JADE fix ports directly; the metric becomes meaningful at longer prompts where the whitening guard clears.
gram_error cos-normalised row-Gram distance: `‖G_U/‖G_U‖_F − G_H/‖G_H‖_F‖_F`	Row-Gram side-channel probe. Tests prompt-fingerprinting via similarity-structure leakage. Range `[0, √2]`, lower = more fingerprintable. Pad-invariant (Haar-vs-HD₃ agreement to bench noise verified in 50-seed controlled experiment, §08). Related literature: Song & Raghunathan, CCS 2020 on embedding-leakage information theory and Morris et al., EMNLP 2023 ("vec2text") on hidden-state-as-side-channel.	0 → √2 (0 = perfect fingerprint; √2 ≈ 1.41 = orthogonal structure; asymptotic ideal-mask read ≈ √(2 − 2/n) ≈ 1.34 at n→∞)	0.000	1.104	1.111	C0 = 0 (identity; control ✓). C2/C3 Δ = 0.007 (parity within ±0.05 band ✓), both at ~78 % of the theoretical max √2 — well clear of the 0.5 fingerprint-floor. Pass.
IA Gate-IA + Attn-IA invariants (AloePri §F.1)	Static weight invariants `Avg(e·W_gate) = Avg(ẽ·W̃_gate)`; recovers τ from `(W_plain, W_obfuscated)`.	5.95 % (Q2.5-14B, T1)	n/a	n/a	n/a	Not applicable — GELO ships no obfuscated weight table; there's no τ for the invariants to recover.
isa_attn_score ridge ISA on attention-score tensors	Ridge inversion on per-head attention scores `(n_heads, n_q, n_kv)`.	2.9 % (Q2.5-14B, T3)	n/a	n/a	n/a	Not applicable — attention compute stays in-TEE by design; attention scores never cross PCIe. Becomes runnable when fused permuted-attention lands.
VMA RowSort weight-pair recovery (AloePri §F.1)	Row-sort + neighbour-match recovery of τ from `(W_plain, W_obfuscated)`.	13.5 % (Q2.5-14B, T1)	n/a	n/a	n/a	Not applicable — GELO ships no obfuscated weight table.
TFMA / SDA token-frequency / substitution-cipher decipherment	Recovers τ from the wire-side response stream of obfuscated token ids.	0.1–3.2 % (Q2.5-14B, T4)	n/a	n/a	n/a	Not applicable — token ids never cross the trust boundary; embedding lookup + LM head both TEE-internal.

Acceptance gate — HD₃ default-flip — click to expand

Each row of the gate pairs an absolute defence claim with a C3-vs-C2 parity check: an HD₃ default-flip requires both (a) C2 (the released Haar default) defends, and (b) C3 matches C2 within ±0.05 in cosine-distance space or its equivalent. C0 control fields confirm the attack itself is not broken.

Gate field	Required	Measured	Verdict
`ima_c2_below_10pct`	C2 IMA < 10 %	1.5 %	✓ PASS
`isa_c2_below_10pct`	C2 ISA < 10 %	0.0 %	✓ PASS
`ima_paper_like_c2_below_10pct`	C2 IMA-paper-like < 10 %	0.0 %	✓ PASS
`c0_ima_at_least_95pct`	C0 control IMA ≥ 95 %	96.4 %	✓ PASS (was 35.7 % at 8-prompt smoke — fixed by the 64-prompt release-gate corpus from the AloePri harness)
`c0_ima_paper_like_at_least_50pct`	C0 control IMA-paper-like ≥ 50 %	0.4 %	✗ FAIL (driver broken — see AloePri investigation; not a protocol issue)
`c0_nn_at_least_50pct`	C0 control NN ≥ 50 %	100.0 %	✓ PASS
`per_offload_at_most_per_forward_plus_shield_on_isa`	per-offload-mode ISA ≤ per-forward-with-shield ISA (internal diagnostic — uses the harness's C1 capture not displayed in the table above)	0.0 ≤ 0.0	✓ PASS
`ima_c3_within_haar_band`	\|C3 − C2\| ≤ 0.05 on IMA	Δ 0.000	✓ PASS
`ima_paper_like_c3_within_haar_band`	\|C3 − C2\| ≤ 0.05 on IMA-paper-like	Δ 0.004	✓ PASS
`ima_c3_below_10pct`	C3 IMA < 10 %	1.5 %	✓ PASS
`isa_c3_below_10pct`	C3 ISA < 10 %	4.7 %	✓ PASS
`anchor_ica_c3_within_haar_band`	\|C3 − C2\| ≤ 0.05 on anchor_ica	Δ 0.013 (8-prompt smoke)	✓ PASS (at smoke scale; 64-prompt re-measurement pending batching follow-up)
`jade_c3_within_haar_band`	\|C3 − C2\| ≤ 0.05 on JADE	Δ 0.001	✓ PASS
`gram_error_c3_within_haar_band`	\|C3 − C2\| ≤ 0.05 on gram_error	Δ 0.007	✓ PASS
`gram_error_c3_above_fingerprint_floor`	C3 gram_error ≥ 0.5 absolute	1.111	✓ PASS

Summary. 14 of 15 measurable gates pass. The one failure is the known-broken ImaPaperLike driver — a measurement-side issue, not a protocol issue. Every C2 absolute-defence threshold holds. Every C3-vs-C2 parity check (including anchor_ica at smoke scale) holds within ±0.05. The strongest empirical evidence to date that HD₃ achieves Haar parity on the GELO threat model.

Open follow-ups

anchor_ica + jd batching. Both attacks share the same per-snapshot single-core grinding pattern that JADE had pre-batching. The JADE numba-JIT + batched joint-diag pattern from §08 ports directly: bucket snapshots by (layer, kind), batch the inner sweeps via numba.prange. Drops anchor_ica from O(hours) to ≤ 5 min and unblocks the JD sweep. ~1 day of work.
ImaPaperLike training budget. The AloePri investigation showed the paper's 2-epoch default cannot reach ridge's closed-form on a 2048×2048 weight. Options: (a) extend epochs to ~100 + matching paper-like inverter sweep (the sweep already supports this), (b) drop the driver from the gate and rely on IMA-ridge alone (load-bearing trio still: NN / IMA / ISA). Decision deferred to release-gate run.
isa_attn_score capture. Driver in tree, gated on the fused permuted-attention path. Adding a WeightKind::AttnScore variant in gelo-protocol + per-prompt attention-score snapshot would activate the driver. ~half day once the fused kernel lands.
512-prompt release-gate run. The 64-prompt smoke clears every reachable gate; the 512-prompt run (release-gate-512.txt in tree) is the final calibration before HD₃ default-flip — tightens statistics on JADE, jd, anchor_ica, gram_error, and ImaPaperLike if its training budget is fixed.

§10

Status & gaps#

what's landed · what's next · what's deferred

Qwen3-1.7B greedy generation runs end-to-end under GELO, with the HD₃ Hadamard-cascade mask (§08) opt-in and the AloePri / GELO §4.3 attack harness (§09) cleared at smoke scale. Today's measurement: +33 % TPOT vs unprotected GPU baseline, bit-identical token output. Gemma 4 substrate is landed but real-weight generation gated on the Gemma decoder refactor decoder refactor (~4-5 weeks; details in §04 collapsed block).

Status snapshot

Track	Status	Where
Qwen3 generation harness	landed · greedy-only · HF-transformers parity passing	`decoder/{generation, forward, kv_cache, rms_norm}.rs`
Performance bench (3-cell)	landed	`tests/qwen3_generation_bench.rs` · §07 numbers
HD₃ mask (Opt 2: radix-8 + scratch reuse)	landed · opt-in	§08 + `crates/gelo-protocol/src/hd3.rs`
Attack harness (4-condition matrix)	landed · 14 of 15 gates pass at 64-prompt smoke	§09 + `evals/aloepri-attacks/`
HTTP `/generate` route	planned · reranker route as template	`gelo-snp-runner :: main`
Sampler — top-p / top-k / temperature	open · greedy-only today	`decoder/generation.rs`
Gemma 4 decoder refactor	open · ~4-5 weeks · protocol substrate already landed	`gemma4-architecture-roadmap`

Open protocol gaps

U-Verify cost at ≥ 1B parameters. Linear-in-weight Freivalds gives 150× decode slowdown at k=2; k=8 production probes scale worse. Candidate fixes: randomised-block Freivalds, batched-across-decode verification, or checkpoint-only attestation. Mandatory before integrity rides production; not a blocker for the current Qwen3 demonstrator path.
SCX-style KV-cache encoding for decode. Decode KV-cache bandwidth at long context is the currently unmeasured cost on the demonstrator path. SCX (Yuan et al., SIGCOMM 2025) is the candidate primitive; gated on a security spike against Hidden-No-More / KV-Cloak adversaries.
HD₃ default-flip — attack-harness re-run at 512 prompts with anchor_ica / JD batched. 64-prompt smoke clears every reachable gate; release-gate run is the final calibration before flipping InProcessTrustedExecutor::new default from Haar to HD₃.
HKDF-derived mask material for decode-step amortisation at d=2560 (Gemma E4B). Lever, not critical path; landing condition is the E4B bench showing mask-sample > ~10 % TPOT.

Deferred

Speculative decoding — protocol implications under per-batch fresh A are unexplored.
MoE generation (Qwen3-MoE / DeepSeek-V2 / Gemma 4 26B-A4B) — router-histogram side channel needs CryptoMoE-class balanced dispatch; separate ~3-6 month workstream.
Score-DP / token-DP accountant — only relevant when a deployment needs per-token probability export.
ECDH-bound session-key handshake — reranker's swap-in applies verbatim when needed.
Gemma 4 31B dense — same protocol primitives, but the ~31 GB weight footprint forces a 64 GB SEV-SNP SKU.

Decisions locked (2026-05-20)

Today's demonstrator — Qwen3-1.7B; Qwen3-4B as same-code-path stretch. Qwen3.5 family rejected.
Gemma 4 is the next architecture target; protocol substrate (PLE / hybrid dispatch / p-RoPE / K=V) landed ahead of decoder refactor.
Accuracy reference — HF transformers.AutoModelForCausalLM at temperature = 0 is the accept gate (llama.cpp / vLLM informational only).
Sampler — greedy only; top-p / top-k / temperature lands with the HTTP route.
HD₃ default-flip gated on 512-prompt attack-harness re-run + anchor_ica/JD batching follow-up.