Latent-cache arithmetic and the embedded implementation mismatch

Read the equation

The arithmetic (512+64)×2=1152 bytes is correct for a cache storing one compressed key/value latent and one rotary key vector per token and layer at two bytes per element. The dossier’s embedded eager code instead caches reconstructed per-head keys and values:128×(192+128)×2=81920 bytes. Its FlashAttention path pads values to 192 before caching, giving 98304 bytes. The official optimized absorbed path provides the latent representation; it is a separate implementation choice.

Terms & definitions

How it works

Read the definition from its right-hand expression to the quantity on the left. Its surrounding source records the intended input, state, or parameter context. Where the source leaves an operation unspecified, this page preserves that gap.

Assumptions & limits

Source recovery establishes that the expression appears in this record. Its mathematical validity, implementation behavior, and interpretation must be assessed separately. See the linked source and research notes for the checks available for this version.

Original context

A conventional 128-head KV cache would store K and V per head:
          128 heads x (128 + 128) x 2 bytes = 65,536 B per token per layer.
      MLA caches the LATENT instead:
          compressed_kv (512) + k_rope (64) = 576 elements x 2 B = 1,152 B.
      A 56.9x reduction.
      k_nope and v are RECONSTRUCTED from the 512-dim latent through kv_b_proj on
      every decode step, so the memory saving is paid for in compute: the KV read
      path and the kv_b_proj matmul are coupled.

compressed_kv (512) + k_rope (64) = 576 elements x 2 B = 1,152 B.

Related builds