Zero TVM Documentation

Quick start

No install. Just open the chat page. Weights load from your browser cache (if you've used WebLLM before) or download fresh from HuggingFace (~2.1 GB).

How it works

The engine has three phases per generated token:

1. Prefill — process each prompt token sequentially, building up the KV cache
2. First decode — the last prefill step produces the first generated token
3. Decode loop — each step takes the previous token as input, runs the full forward pass with the KV cache providing attention context, produces the next token

Each forward pass runs 10 kernel roles through 32 transformer layers — 228 dispatches per token in the default f16-KV path (260 with the ?int8kv=1 flag) — then reads one i32 token ID back from the GPU. For comparison, WebLLM's TVM-generated decode path fires 342 dispatches per token using 85 distinct shaders.

Architecture overview

Four source files make up the engine:

Weight loader

Weights are stored in MLC's ndarray-cache.json format — an index file listing every parameter, which shard binary it lives in, its byte offset, and byte size. The loader reads this index then fetches each referenced shard.

Fetch priority

1. Browser Cache API — if a prior WebLLM session cached the shards, they load instantly
2. HuggingFace — direct HTTPS fetch from huggingface.co/mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC

Parameter naming (MLC format)

MLC uses non-standard parameter names. The actual names in the cache:

Tokenizer

A hand-written BPE tokenizer in TypeScript. No SentencePiece WASM, no HuggingFace tokenizers bundle. Reads tokenizer.json directly.

Key steps

• Pre-tokenization — Metaspace: spaces become ▁, words are split on whitespace
• BPE encoding — merge pairs by rank from the merge table in tokenizer.json
• Special tokens — <|system|>, <|user|>, <|assistant|>, <|end|>
• Chat template — Phi-3 format applied by buildChatPrompt()

Phi-3 chat template

<|system|>
You are a helpful assistant.<|end|>
<|user|>
What is the capital of Australia?<|end|>
<|assistant|>

Stop tokens: 2 (EOS), 32000 (<|end|>), 32007 (<|endoftext|>).

KV cache

Uses a paged KV cache — the same approach as vLLM and PagedAttention. Memory is divided into fixed-size pages (16 slots each). A page table maps logical positions to physical pages.

Each layer has its own GPUBuffer for KV pages. The page table is a simple identity mapping (page i → physical page i) for single-sequence inference.

Decode loop

Each call to decodeToken(tokenId, position) submits one command encoder with the full forward pass:

// Per-token GPU state written via writeBuffer
B.inputIds   ← [tokenId]          // i32
B.posMap     ← [position]         // i32
B.pageIndptr ← [0, nnzPages]      // page range
B.lengthInfo ← [position+1, 0, 0] // seq length

// Forward pass (one command encoder)
embedding(B.residual)             // token → hidden state
rmsNorm(B.hidden1, B.residual)    // initial norm

for L in 0..32:
  // QKV matmul + RoPE + KV-append in ONE dispatch (M4 fusion)
  qkvFused(B.qOut, kvPages[L], B.hidden1)
  attention(B.attnOut, B.qOut, kvPages[L])
  int4Matmul(B.hidden2, B.attnOut)               // O projection
  addNorm(B.hidden2, resIn → B.hidden1, resOut)  // residual + RMSNorm, ping-pong
  // Gate + Up + SiLU + mul + Down in ONE dispatch
  fusedFfn(B.hidden2, B.hidden1)
  addNorm(B.hidden2, resIn → B.hidden1, resOut)  // residual + RMSNorm, ping-pong

int4Matmul(B.logits, B.hidden1)   // LM head
argmax(B.tokenOut, B.logits)      // → next token ID

Ping-pong residual buffers

WebGPU's validation rules forbid binding the same buffer as both read and read_write in the same dispatch. The add_norm shader needs to read the old residual and write the new one.

Solution: two residual buffers that alternate each dispatch.

let resIn  = B.residual   // ping (starts with embedding)
let resOut = B.residual2  // pong (uninitialized)

// Each add_norm:
dispatch(addNorm, [delta, resIn, gamma, hidden1, resOut])
[resIn, resOut] = [resOut, resIn]  // swap — O(1), no GPU copy

WGSL Kernel Roles

All shaders live in src/compiler/shaders/ — 27 .wgsl files implementing 10 distinct kernel roles. The rest are tiled and subgroup variants of the same role, selectable at runtime via URL flags. The compiler compiles them all at startup into GPUComputePipeline objects.

Binding convention: @group(0) always. Binding indices are zero-based and match the order you pass buffers to bg(device, pipeline, [...bufs]).

Phi-3 model constants

export const PHI3 = {
  D:        3072,   // hidden dimension
  HEADS:    32,     // attention heads
  HEAD_DIM: 96,     // D / HEADS
  LAYERS:   32,     // transformer layers
  FFN:      8192,   // FFN intermediate dimension
  VOCAB:    32064,  // vocabulary size
  PAGE_SIZE:16,     // KV cache slots per page
  MAX_PAGES:257,    // max pages (≈ 4096 context)
}

Q4F16 quantization format

MLC's Q4F16_1 format packs 8 int4 values into each uint32. Scales are stored as float16 with group_size=32 — one scale per 32 weights.

// Extract nibble for element i within a u32
let nibble = (packed_u32 >> (i * 4)) & 0xF;

// Dequantize: center around 0, multiply by scale
let value = f16(i32(nibble) - 7) * scale;

Weight shapes in Q4F16 (for Phi-3-mini):

Qwen3-4B (?model=qwen3)

The engine is parameterized over a ModelSpec, and a v1 Qwen3-4B (q4f16_1) port ships alongside Phi-3. Append ?model=qwen3 to zero-tvm.html or validate.html — it works on the live site (weights stream from HuggingFace, ~2.3 GB), and node scripts/download-weights.mjs --model qwen3 primes the local dev mirror. Phi-3 stays the default; all existing URLs keep their exact behavior.

What the port exercises that Phi-3 doesn't:

Measured 2026-07-28 on an Apple M2 Max (same session, identical local weight bytes): Zero-TVM 25.43 tok/s decode vs WebLLM's prebuilt Qwen3-4B at 14.15 tok/s — +80% on that pair. Unlike the Phi-3 headline, these figures are static dated text, not synced from bench/results.json.

Qwen3.5-4B hybrid (?model=qwen35)

The third model, and the first hybrid architecture on the engine: 24 gated-DeltaNet (linear-attention) layers interleaved with 8 gated full-attention layers (attention on every 4th layer). To our knowledge this is the first hand-written-kernel int4 implementation of a gated-DeltaNet hybrid running in a browser. Append ?model=qwen35 to zero-tvm.html or validate.html; node scripts/download-weights.mjs --model qwen35 primes the local dev mirror (~2.6 GB).

What the hybrid adds over Qwen3-4B:

Measured 2026-07-28 on an Apple M2 Max (same session, identical local weight bytes): Zero-TVM 47.99 tok/s decode vs WebLLM 0.2.84's prebuilt Qwen3.5-4B at 31.99 tok/s — +50% on that pair. Static dated text, not synced from bench/results.json.

Port to Phi-4-mini or Qwen3

Both are available as MLC Q4F16 packages. The Qwen3 port has since landed (see Qwen3-4B above) — these are the steps it followed, and the recipe for the next model:

1. Update PHI3 constants in compiler.ts — D, HEADS, HEAD_DIM, LAYERS, FFN, VOCAB
2. Check parameter names — fetch ndarray-cache.json and log all keys. Update weight-loader.ts candidates to match
3. Check for GQA — if KV heads ≠ Q heads (grouped-query attention), the attention shader needs a small change to repeat KV heads
4. Update chat template — each model has its own special tokens and prompt format
5. Update HuggingFace base URL — change PHI3_MODEL_BASE in weight-loader.ts

Local weight serving

Run the download script once to save all shards locally. Subsequent page loads are instant (served from localhost, no network).

node scripts/download-weights.mjs

# Downloads to: public/weights/Phi-3-mini-4k-instruct-q4f16_1-MLC/
# Served at:    /weights/Phi-3-mini-4k-instruct-q4f16_1-MLC/
# Size:         ~2.1 GB

Debugging tips

All output is <unk>

• Weight version mismatch — ndarray-cache.json offsets don't match shard content
• Buffer aliasing — add_norm dispatched with same buffer as both @1 and @4
• Wrong rope binding order — check @0=q_out, @1=k_out, @2=v_out, @3=qkv, @4=posMap

Garbage / repetitive output

• Rope bindings are in the wrong order (this was our bug — garbage like -,unlintzegesenma)
• Wrong uniform values for a shader (K_groups, N, etc.)

WebGPU validation error about aliasing

• Same buffer bound as read_write and read in one dispatch
• Fix: use ping-pong buffers. Never bind B.residual as both @1 and @4 to add_norm

Model not loading (Weight not found)

• Log all available keys: the weight loader prints them to console on load
• MLC names differ from HuggingFace standard names (transformer.h.N.mixer.* not model.layers.N.self_attn.*)

Bugs we fixed (and how)

vs WebLLM

Head-to-head on Phi-3-mini-4k-instruct Q4F16_1, same weights, same run, same browser (Chrome 150 with WebGPU), Apple M2 Max — npm run bench, 128 decode tokens × 5 runs, median (latest run recorded in bench/results.json):

How the result flipped (22% behind → 28–31% ahead)

An earlier head-to-head (M2 Pro, 2026-06) read 42.14 vs ~51.5 tok/s — Zero-TVM 22% behind. Both the hardware and the engine changed since, so the two Zero-TVM numbers are not a pure optimization delta; the same-day, same-machine WebLLM number is the valid comparator for the current Zero-TVM median (both in the table above). What changed in the engine:

• Correctness fixes. f32 accumulation in the fused FFN, a workgroup-barrier fix in attention, and a decode off-by-one fix — the old number understates even the old hardware.
• vec4 loads, now default. Re-declaring weight + activation buffers as array<vec4<u32>> in the int4 matmuls and qkv_fused measured +7.1% on M2 Max; opt out with ?vec4=0 / ?vec4qkv=0.
• Tiled + subgroup kernels. The 4-row tiled subgroup FFN and the _sg matmul/attention/argmax variants are the default path on Apple sg32 hardware.

Still default since 2026-07-27 (`?splitk=0` to disable): split-K attention (?splitk=N, ~+3% at short context, needs a long-context A/B); the best measured config ?vec4=1&vec4qkv=1&splitk=8 reaches 68.36 tok/s. Falsified and not shipped: FFN prologue fusion (?fuseprologue=1, −13.7% on M2 Max). Dispatch overhead was never the gap — Zero-TVM already submitted fewer dispatches per token (228 vs 342) when it was slower. Full A/B tables in BENCH.md.

The memory-bandwidth ceiling

Phi-3-mini Q4F16_1 touches ~1.8 GB of weights per decode token. On M2 Max's 400 GB/s memory bus that's ~4.5 ms/token, or ~222 tok/s theoretical max. Zero-TVM's measured decode median sits at roughly 30% of that ceiling; WebLLM's at roughly 23% (exact medians in the table above). Neither engine can exceed the ceiling without changing weight layout or quantization (int8 KV shaves a bit, which is why it's exposed behind a flag).