Normalize each query’s attention scores

Read the equation

Apply softmax across key positions for each query and head. The resulting nonnegative weights sum to one when a row has at least one finite eligible score. The source computes this normalization in float32 and then converts it to the query dtype; its eager implementation incorporates a mask beforehand.

Terms & definitions

How it works

Read the definition from its right-hand expression to the quantity on the left. Its surrounding source records the intended input, state, or parameter context. Where the source leaves an operation unspecified, this page preserves that gap.

Assumptions & limits

Source recovery establishes that the expression appears in this record. Its mathematical validity, implementation behavior, and interpretation must be assessed separately. See the linked source and research notes for the checks available for this version.

Original context

q = concat(q_nope, q_rope)  -> 192 per head
      k = concat(k_nope, k_rope)  -> 192 per head  (k_rope broadcast to all heads)
      scores = q @ k^T * 0.135237
      probs  = softmax(scores, dim=-1) computed in float32, cast back to q.dtype
      out    = probs @ v                            -> 128 per head
      out    = o_proj(out.reshape(T, 16384))        -> 7168  fp8 (7168, 16384)

  WHAT IS CACHED - THE POINT OF MLA

probs  = softmax(scores, dim=-1) computed in float32, cast back to q.dtype

Related builds