What the archive contains
The dossier describes a 61-layer language model with 64 attention heads, 384 routed experts and eight selected experts in each of 60 MoE layers, plus one shared expert. A 27-block MoonViT vision tower and patch-merging projector feed the language residual stream. The actual recovered artifact is the 282,011-byte source dossier; the stated 64 safetensors shards and hundreds of gigabytes of weight payload are historical file locators, not recovered model weights in this website.
Quantization and source boundaries
The overview calls the donor BF16 and says no quantization_config block exists, but its own nested text configuration includes one. The parameter patterns describe packed int4 routed-expert weights with BF16 scales, alongside BF16 dense/attention/vision tensors and F32 router correction bias. These distinctions affect loaders, storage and numerical behavior.
A modified reference, not purely upstream source
The document labels Part 5 as verbatim official code, yet its DeepSeek MLP contains use_exact_generators hooks, PackedInt4Linear imports, to_exact and convert_kimi_to_exact additions. These are separately documented in the exact-generator adaptation build. The source should therefore be used with its modification boundaries visible.
Tensor-pattern discrepancy
The projector pattern table lists two 4,608×4,608 layers, while the embedded PatchMergerMLP second layer targets text_hidden_size, configured as 7,168. The table is an attributed census claim, not a substitute for checking the actual module definition and checkpoint header.
What this recovery establishes
All ten embedded Python files were parsed as source text and their numerical expressions catalogued. The audit did not load the donor weights, run inference, reproduce the listed benchmarks, or independently rescan the remote M4 directory.
Equations in this build
209 linked mathematical records. Each opens its notation, interpretation and source.
Scaled dot-product attention logits
For each query, compute its dot product with every key and divide by the square root of the query/key feature width. The transposed key axes put corresponding features into the…
Float32 normalization of vision-attention logits
Softmax across the final key-position axis turns each query’s masked logits into relative nonnegative weights. The source calculates the normalization in float32 and casts its…
Aggregate value vectors with vision-attention weights
For each query, sum the value vectors using its attention weights. The matrix product contracts the key-position axis, leaving the value-feature axis as the output. When weights…
get_rope_shape: return
In get_rope_shape, this statement returns the displayed result. This statement evaluates the named projection, normalization or numeric transform with the source arguments. Its…
apply_rope: xq out
In apply_rope, this statement defines or updates xq_out. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…
apply_rope: xk out
In apply_rope, this statement defines or updates xk_out. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…
get_1d_sincos_pos_embed_from_grid: omega
In get_1d_sincos_pos_embed_from_grid, this statement defines or updates omega. The expression forms a quotient or integer division. The denominator, integer rounding and tensor…
get_1d_sincos_pos_embed_from_grid: omega
In get_1d_sincos_pos_embed_from_grid, this statement defines or updates omega. The expression forms a quotient or integer division. The denominator, integer rounding and tensor…
get_1d_sincos_pos_embed_from_grid: omega
In get_1d_sincos_pos_embed_from_grid, this statement defines or updates omega. The expression forms a quotient or integer division. The denominator, integer rounding and tensor…
get_1d_sincos_pos_embed_from_grid: out
In get_1d_sincos_pos_embed_from_grid, this statement defines or updates out. The matrix or indexed contraction combines the stated axes. Compatible tensor dimensions and the…
get_1d_sincos_pos_embed_from_grid: emb sin
In get_1d_sincos_pos_embed_from_grid, this statement defines or updates emb_sin. This is part of the positional-phase calculation. Frequency, position, sine/cosine and coordinate…
get_1d_sincos_pos_embed_from_grid: emb cos
In get_1d_sincos_pos_embed_from_grid, this statement defines or updates emb_cos. This is part of the positional-phase calculation. Frequency, position, sine/cosine and coordinate…
get_1d_sincos_pos_embed: grid t
In get_1d_sincos_pos_embed, this statement defines or updates grid_t. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…
tpool_patch_merger: pre sum
In tpool_patch_merger, this statement defines or updates pre_sum. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…
Learnable2DInterpPosEmbDivided_fixed.forward: out
In Learnable2DInterpPosEmbDivided_fixed.forward, this statement defines or updates out. The addition or subtraction updates the named state or combines residual contributions. The…
Rope2DPosEmbRepeated._precompute_freqs_cis: N
In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates N. The displayed multiplication combines the source factors. For tensors this is elementwise…
Rope2DPosEmbRepeated._precompute_freqs_cis: flat pos
In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates flat_pos. This statement evaluates the named projection, normalization or numeric transform with…
Rope2DPosEmbRepeated._precompute_freqs_cis: x pos
In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates x_pos. This statement evaluates the named projection, normalization or numeric transform with the…
Rope2DPosEmbRepeated._precompute_freqs_cis: y pos
In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates y_pos. The expression forms a quotient or integer division. The denominator, integer rounding and…
Rope2DPosEmbRepeated._precompute_freqs_cis: dim range
In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates dim_range. The expression forms a quotient or integer division. The denominator, integer rounding…
Rope2DPosEmbRepeated._precompute_freqs_cis: freqs
In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates freqs. The expression forms a quotient or integer division. The denominator, integer rounding and…
Rope2DPosEmbRepeated._precompute_freqs_cis: x freqs
In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates x_freqs. This statement evaluates the named projection, normalization or numeric transform with…
Rope2DPosEmbRepeated._precompute_freqs_cis: y freqs
In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates y_freqs. This statement evaluates the named projection, normalization or numeric transform with…
Rope2DPosEmbRepeated.get_freqs_cis: freqs cis
In Rope2DPosEmbRepeated.get_freqs_cis, this statement defines or updates freqs_cis. The expression forms a quotient or integer division. The denominator, integer rounding and…
MoonViTEncoderLayer.__init__: hidden size per attention head
In MoonViTEncoderLayer.__init__, this statement defines or updates self.hidden_size_per_attention_head. The expression forms a quotient or integer division. The denominator,…
MoonViTEncoderLayer.__init__: wqkv
In MoonViTEncoderLayer.__init__, this statement defines or updates self.wqkv. The displayed multiplication combines the source factors. For tensors this is elementwise unless the…
MoonViTEncoderLayer.attention_qkvpacked: qkv shape
In MoonViTEncoderLayer.attention_qkvpacked, this statement defines or updates qkv_shape. The addition or subtraction updates the named state or combines residual contributions.…
Residual addition around a transformed hidden state
Add the block input saved in residual to the output of the intervening transformation. Both operands must have the same residual-stream shape. The same expression appears around…
MoonViT3dEncoder.__init__: rope 2d
In MoonViT3dEncoder.__init__, this statement defines or updates self.rope_2d. The expression forms a quotient or integer division. The denominator, integer rounding and tensor…
MoonViT3dEncoder.forward: lengths
In MoonViT3dEncoder.forward, this statement defines or updates lengths. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…
MoonViT3dEncoder.forward: max seqlen
In MoonViT3dEncoder.forward, this statement defines or updates max_seqlen. This statement evaluates the named projection, normalization or numeric transform with the source…
MoonViT3dEncoder.forward: cu seqlens
In MoonViT3dEncoder.forward, this statement defines or updates cu_seqlens. This statement evaluates the named projection, normalization or numeric transform with the source…
tpool_patch_merger: seq
In tpool_patch_merger, this statement defines or updates seq. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…
tpool_patch_merger: (new height, new width)
In tpool_patch_merger, this statement defines or updates (new_height, new_width). The expression forms a quotient or integer division. The denominator, integer rounding and tensor…
tpool_patch_merger: reshaped seq
In tpool_patch_merger, this statement defines or updates reshaped_seq. The reduction aggregates values along the explicit source axis; preserving the axis and keep-dimension…
tpool_patch_merger: padded seq
In tpool_patch_merger, this statement defines or updates padded_seq. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…
tpool_patch_merger: pre sum
In tpool_patch_merger, this statement defines or updates pre_sum. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…
MLP.forward: x
In MLP.forward, this statement defines or updates x. This statement evaluates the named projection, normalization or numeric transform with the source arguments. Its learned…
Width of a concatenated Kimi patch group
The feature width of one patch is multiplied by the number of patches in a merge group. For 1,152 features and a 2×2 group, the intermediate width is 4,608. The later constructor…
KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: left padding
In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates left_padding. The reduction aggregates values along the explicit source…
KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: max embed dim
In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates max_embed_dim. The reduction aggregates values along the explicit source…
KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: new token positions
In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates new_token_positions. The addition or subtraction updates the named state…
KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: nb image pad
In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates nb_image_pad. The addition or subtraction updates the named state or…
KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: image to overwrite
In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates image_to_overwrite. The addition or subtraction updates the named state…
KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: position ids
In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates position_ids. The addition or subtraction updates the named state or…
KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: final embedding[batch indices, indices to mask]
In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates final_embedding[batch_indices, indices_to_mask]. This statement fixes a…
Apply the patch-merger projection to each variable-sized group
For each input item, normalize its per-patch feature vectors, flatten each patch group while retaining its leading group count, and apply the merger projection. The list form…
Normalize, concatenate, and project Kimi patch features
Per-patch LayerNorm is applied before adjacent patch features are concatenated by the reshape. The projection network then maps each merged group into the language representation.…
KimiK25ForConditionalGeneration.prepare_inputs_for_generation: position ids
In KimiK25ForConditionalGeneration.prepare_inputs_for_generation, this statement defines or updates position_ids. The addition or subtraction updates the named state or combines…
Learnable2DInterpPosEmbDivided_fixed.forward: pos emb 3d
In Learnable2DInterpPosEmbDivided_fixed.forward, this statement defines or updates pos_emb_3d. The addition or subtraction updates the named state or combines residual…
KimiK25ForConditionalGeneration.prepare_inputs_for_generation: input ids
In KimiK25ForConditionalGeneration.prepare_inputs_for_generation, this statement defines or updates input_ids. The addition or subtraction updates the named state or combines…
KimiK25ForConditionalGeneration.prepare_inputs_for_generation: attention mask
In KimiK25ForConditionalGeneration.prepare_inputs_for_generation, this statement defines or updates attention_mask. The addition or subtraction updates the named state or combines…
KimiK25ForConditionalGeneration.forward: (batch index, non attended tokens)
In KimiK25ForConditionalGeneration.forward, this statement defines or updates (batch_index, non_attended_tokens). The reduction aggregates values along the explicit source axis;…
KimiK25ForConditionalGeneration.forward: extended attention mask[new batch index, new non attended tokens]
In KimiK25ForConditionalGeneration.forward, this statement defines or updates extended_attention_mask[new_batch_index, new_non_attended_tokens]. This statement fixes a numeric…
KimiK25ForConditionalGeneration.forward: position ids
In KimiK25ForConditionalGeneration.forward, this statement defines or updates position_ids. The reduction aggregates values along the explicit source axis; preserving the axis and…
KimiK25ForConditionalGeneration.prepare_inputs_for_generation: input ids
In KimiK25ForConditionalGeneration.prepare_inputs_for_generation, this statement defines or updates input_ids. The addition or subtraction updates the named state or combines…
_get_unpad_data: seqlens in batch
In _get_unpad_data, this statement defines or updates seqlens_in_batch. The reduction aggregates values along the explicit source axis; preserving the axis and keep-dimension…
_get_unpad_data: max seqlen in batch
In _get_unpad_data, this statement defines or updates max_seqlen_in_batch. This statement evaluates the named projection, normalization or numeric transform with the source…
_get_unpad_data: cu seqlens
In _get_unpad_data, this statement defines or updates cu_seqlens. This statement evaluates the named projection, normalization or numeric transform with the source arguments. Its…
Solve for the rotary frequency index at a chosen rotation count
At index i, the unscaled angular frequency is base^(−2 i/dim). Over orig_max positions it completes orig_max*base^(−2 i/dim)/(2π) rotations. Setting that count equal to rot and…
yarn_find_correction_range: low
In yarn_find_correction_range, this statement defines or updates low. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…
yarn_find_correction_range: high
In yarn_find_correction_range, this statement defines or updates high. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…
yarn_find_correction_range: return
In yarn_find_correction_range, this statement returns the displayed result. The addition or subtraction updates the named state or combines residual contributions. The input…
Logarithmic magnitude adjustment for extended context
For scale>1, the adjustment adds 0.1*mscale*ln(scale) to one. The code returns one for scale≤1. One parameter specifies the context extension, while the other controls adjustment…
yarn_linear_ramp_mask: linear func
In yarn_linear_ramp_mask, this statement defines or updates linear_func. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape…
yarn_linear_ramp_mask: ramp func
In yarn_linear_ramp_mask, this statement defines or updates ramp_func. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…
rotate_half: x1
In rotate_half, this statement defines or updates x1. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention determine…
rotate_half: x2
In rotate_half, this statement defines or updates x2. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention determine…
apply_rotary_pos_emb: q
In apply_rotary_pos_emb, this statement defines or updates q. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…
apply_rotary_pos_emb: k
In apply_rotary_pos_emb, this statement defines or updates k. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…
Rotate paired query coordinates using a position-dependent angle
rotate_half maps a pair of coordinate blocks(x 1,x 2) to(−x 2,x 1). Multiplication by cosine and addition of the sine-weighted rotated vector implements a two-dimensional rotation…
Rotate paired key coordinates using a position-dependent angle
rotate_half maps a pair of coordinate blocks(x 1,x 2) to(−x 2,x 1). Multiplication by cosine and addition of the sine-weighted rotated vector implements a two-dimensional rotation…
convert_kimi_to_exact: n
In convert_kimi_to_exact, this statement defines or updates n. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…
repeat_kv: return
In repeat_kv, this statement returns the displayed result. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…
get_usable_length: return
In get_usable_length, this statement returns the displayed result. The addition or subtraction updates the named state or combines residual contributions. The input shapes and the…
RMSNorm mean square, despite the local variable name
This expression averages squared components along the final feature axis and preserves that axis for broadcasting. It computes the second raw moment, not centered variance: the…
Normalize by the root mean square
Multiply each feature by 1/√(mean(x²)+epsilon). Positive epsilon prevents division by zero and changes exact unit-RMS normalization near the zero vector. Unlike LayerNorm, this…
Apply learned per-feature RMSNorm gains
After RMS scaling, multiply each component by its learned gain and restore the earlier input dtype. The gain allows the model to choose different output scales by feature; it does…
DeepSeek RotaryEmbedding.__init__: inv freq
In DeepseekV3RotaryEmbedding.__init__, this statement defines or updates inv_freq. This is part of the positional-phase calculation. Frequency, position, sine/cosine and…
DeepSeek RotaryEmbedding._set_cos_sin_cache: t
In DeepseekV3RotaryEmbedding._set_cos_sin_cache, this statement defines or updates t. This is part of the positional-phase calculation. Frequency, position, sine/cosine and…
DeepSeek RotaryEmbedding._set_cos_sin_cache: freqs
In DeepseekV3RotaryEmbedding._set_cos_sin_cache, this statement defines or updates freqs. This is part of the positional-phase calculation. Frequency, position, sine/cosine and…
DeepSeek LinearScalingRotaryEmbedding._set_cos_sin_cache: t
In DeepseekV3LinearScalingRotaryEmbedding._set_cos_sin_cache, this statement defines or updates t. The expression forms a quotient or integer division. The denominator, integer…
DeepSeek LinearScalingRotaryEmbedding._set_cos_sin_cache: freqs
In DeepseekV3LinearScalingRotaryEmbedding._set_cos_sin_cache, this statement defines or updates freqs. This is part of the positional-phase calculation. Frequency, position,…
yarn_get_mscale: return
In yarn_get_mscale, this statement returns the displayed result. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…
yarn_linear_ramp_mask: max
In yarn_linear_ramp_mask, this statement defines or updates max. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…
DeepSeek YarnRotaryEmbedding._set_cos_sin_cache: freq extra
In DeepseekV3YarnRotaryEmbedding._set_cos_sin_cache, this statement defines or updates freq_extra. The expression forms a quotient or integer division. The denominator, integer…
DeepSeek YarnRotaryEmbedding._set_cos_sin_cache: freq inter
In DeepseekV3YarnRotaryEmbedding._set_cos_sin_cache, this statement defines or updates freq_inter. The expression forms a quotient or integer division. The denominator, integer…
Complement the increasing YaRN frequency ramp
The helper produces a ramp increasing from zero to one across frequency-pair indices. Subtracting it from one makes inv_freq_mask select original frequencies at the small-index…
Blend original and divided YaRN frequency schedules
The source uses inv_freq_mask as the weight on the original schedule and its complement as the weight on the divided schedule. Since inv_freq_mask=1−ramp, this is…
DeepSeek YarnRotaryEmbedding._set_cos_sin_cache: t
In DeepseekV3YarnRotaryEmbedding._set_cos_sin_cache, this statement defines or updates t. This statement evaluates the named projection, normalization or numeric transform with…
DeepSeek YarnRotaryEmbedding._set_cos_sin_cache: freqs
In DeepseekV3YarnRotaryEmbedding._set_cos_sin_cache, this statement defines or updates freqs. This is part of the positional-phase calculation. Frequency, position, sine/cosine…
Ratio controlling the rotary cache magnitude
This coefficient divides the adjustment computed with mscale by the adjustment computed with mscale_all_dim. Equal strengths give a ratio of one. The attention initializer can…
SwiGLU-style gated feed-forward projection
Two learned maps project x into the intermediate width. The configured activation, SiLU in these dossiers, is applied to the gate projection; its result multiplies the…
MoEGate.forward: logits
In MoEGate.forward, this statement defines or updates logits. The matrix or indexed contraction combines the stated axes. Compatible tensor dimensions and the transposes in this…
MoEGate.forward: topk weight
In MoEGate.forward, this statement defines or updates topk_weight. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive…
DeepSeek MoE.moe_infer: tokens per expert
In DeepseekV3MoE.moe_infer, this statement defines or updates tokens_per_expert. The reduction aggregates values along the explicit source axis; preserving the axis and…
DeepSeek MoE.moe_infer: sorted tokens
In DeepseekV3MoE.moe_infer, this statement defines or updates sorted_tokens. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries…
DeepSeek MoE.moe_infer: start idx
In DeepseekV3MoE.moe_infer, this statement defines or updates start_idx. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is…
Sum the selected expert outputs with their routing weights
Each chosen expert evaluates the same input representation h. Its output vector is multiplied by the associated routing weight, and the selected vectors are added. The…
DeepSeek Attention.__init__: q head dim
In DeepseekV3Attention.__init__, this statement defines or updates self.q_head_dim. The addition or subtraction updates the named state or combines residual contributions. The…
DeepSeek Attention.__init__: kv a proj with mqa
In DeepseekV3Attention.__init__, this statement defines or updates self.kv_a_proj_with_mqa. The addition or subtraction updates the named state or combines residual contributions.…
DeepSeek Attention.__init__: kv b proj
In DeepseekV3Attention.__init__, this statement defines or updates self.kv_b_proj. The displayed multiplication combines the source factors. For tensors this is elementwise unless…
DeepSeek Attention.__init__: o proj
In DeepseekV3Attention.__init__, this statement defines or updates self.o_proj. The displayed multiplication combines the source factors. For tensors this is elementwise unless…
Baseline inverse-square-root attention coefficient
Initialize the score multiplier to 1/√d, where d is the total rotary plus non-rotary query-head width. This is the starting coefficient; the following YaRN branch can multiply it…
Expand the compressed latent into per-head keys and values
Normalize the compressed key/value latent, apply the learned up-projection, and reshape into heads. Each head receives its non-rotary key and value components. This eagerly…
DeepSeek Attention.forward: attn weights
In DeepseekV3Attention.forward, this statement defines or updates attn_weights. Softmax exponentiates and normalizes scores along the chosen axis, producing nonnegative weights…
Normalize each query’s attention scores
Apply softmax across key positions for each query and head. The resulting nonnegative weights sum to one when a row has at least one finite eligible score. The source computes…
DeepSeek Attention.forward: attn output
In DeepseekV3Attention.forward, this statement defines or updates attn_output. The matrix or indexed contraction combines the stated axes. Compatible tensor dimensions and the…
DeepSeek Attention.forward: attn output
In DeepseekV3Attention.forward, this statement defines or updates attn_output. The displayed multiplication combines the source factors. For tensors this is elementwise unless the…
DeepSeek Attention.forward: attn output
In DeepseekV3Attention.forward, this statement defines or updates attn_output. This statement evaluates the named projection, normalization or numeric transform with the source…
DeepSeek FlashAttention2.forward: attn output
In DeepseekV3FlashAttention2.forward, this statement defines or updates attn_output. The displayed multiplication combines the source factors. For tensors this is elementwise…
DeepSeek FlashAttention2._upad_input: key layer
In DeepseekV3FlashAttention2._upad_input, this statement defines or updates key_layer. The displayed multiplication combines the source factors. For tensors this is elementwise…
DeepSeek FlashAttention2._upad_input: value layer
In DeepseekV3FlashAttention2._upad_input, this statement defines or updates value_layer. The displayed multiplication combines the source factors. For tensors this is elementwise…
DeepSeek DecoderLayer.__init__: mlp
In DeepseekV3DecoderLayer.__init__, this statement defines or updates self.mlp. This statement evaluates the named projection, normalization or numeric transform with the source…
DeepSeek Model.forward: past key values length
In DeepseekV3Model.forward, this statement defines or updates past_key_values_length. This statement fixes a numeric parameter or initializes a numerical state in this source…
DeepSeek Model.forward: hidden states
In DeepseekV3Model.forward, this statement defines or updates hidden_states. This statement evaluates the named projection, normalization or numeric transform with the source…
DeepSeek ForSequenceClassification.forward: pooled logits
In DeepseekV3ForSequenceClassification.forward, this statement defines or updates pooled_logits. This statement evaluates the named projection, normalization or numeric transform…
DeepSeek DynamicNTKScalingRotaryEmbedding._set_cos_sin_cache: base
In DeepseekV3DynamicNTKScalingRotaryEmbedding._set_cos_sin_cache, this statement defines or updates base. The expression forms a quotient or integer division. The denominator,…
DeepSeek DynamicNTKScalingRotaryEmbedding._set_cos_sin_cache: inv freq
In DeepseekV3DynamicNTKScalingRotaryEmbedding._set_cos_sin_cache, this statement defines or updates inv_freq. This is part of the positional-phase calculation. Frequency,…
convert_kimi_to_exact: n
In convert_kimi_to_exact, this statement defines or updates n. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…
Independent sigmoid scores for the router
The sigmoid 1/(1+exp(−z)) is applied to each expert logit independently. It compresses each finite real logit into (0,1), without normalizing across experts. These values are…
Bias the choice of experts while retaining original affinities
An expert-specific correction vector is added to the sigmoid affinities for ranking. This addition can change which groups and experts are selected. The later mixture weights are…
MoEGate.forward: group scores
In MoEGate.forward, this statement defines or updates group_scores. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive…
MoEGate.forward: group idx
In MoEGate.forward, this statement defines or updates group_idx. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive the…
MoEGate.forward: score mask
In MoEGate.forward, this statement defines or updates score_mask. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape…
MoEGate.forward: ( , topk idx)
In MoEGate.forward, this statement defines or updates (_, topk_idx). The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive…
MoEGate.forward: denominator
In MoEGate.forward, this statement defines or updates denominator. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive…
Normalize the selected routing affinities
Each gathered affinity is divided by the sum across the selected experts plus 10⁻²⁰. Ignoring that stabilizer, the selected weights sum to one. With the stabilizer retained, the…
DeepSeek MoE.__init__: experts per rank
In DeepseekV3MoE.__init__, this statement defines or updates self.experts_per_rank. The expression forms a quotient or integer division. The denominator, integer rounding and…
DeepSeek MoE.__init__: experts
In DeepseekV3MoE.__init__, this statement defines or updates self.experts. The displayed multiplication combines the source factors. For tensors this is elementwise unless the…
DeepSeek MoE.__init__: ep size
In DeepseekV3MoE.__init__, this statement defines or updates self.ep_size. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is…
DeepSeek MoE.__init__: ep rank
In DeepseekV3MoE.__init__, this statement defines or updates self.ep_rank. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is…
DeepSeek MoE.__init__: intermediate size
In DeepseekV3MoE.__init__, this statement defines or updates intermediate_size. The displayed multiplication combines the source factors. For tensors this is elementwise unless…
Add an always-active shared expert
The shared expert processes identity, the input representation saved before the routed expert computation. Its output is added to the weighted routed sum. It therefore contributes…
DeepSeek MoE.moe_infer: tokens per ep rank
In DeepseekV3MoE.moe_infer, this statement defines or updates tokens_per_ep_rank. The reduction aggregates values along the explicit source axis; preserving the axis and…
DeepSeek MoE.moe_infer: output splits
In DeepseekV3MoE.moe_infer, this statement defines or updates output_splits. The reduction aggregates values along the explicit source axis; preserving the axis and keep-dimension…
DeepSeek MoE.moe_infer: gathered tokens
In DeepseekV3MoE.moe_infer, this statement defines or updates gathered_tokens. The reduction aggregates values along the explicit source axis; preserving the axis and…
DeepSeek MoE.moe_infer: tokens per expert post gather
In DeepseekV3MoE.moe_infer, this statement defines or updates tokens_per_expert_post_gather. The reduction aggregates values along the explicit source axis; preserving the axis…
DeepSeek MoE.moe_infer: s
In DeepseekV3MoE.moe_infer, this statement defines or updates s. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…
DeepSeek MoE.moe_infer: end idx
In DeepseekV3MoE.moe_infer, this statement defines or updates end_idx. The addition or subtraction updates the named state or combines residual contributions. The input shapes and…
DeepSeek MoE.moe_infer: expert
In DeepseekV3MoE.moe_infer, this statement defines or updates expert. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…
DeepSeek Attention.__init__: q proj
In DeepseekV3Attention.__init__, this statement defines or updates self.q_proj. The displayed multiplication combines the source factors. For tensors this is elementwise unless…
DeepSeek Attention.__init__: q b proj
In DeepseekV3Attention.__init__, this statement defines or updates self.q_b_proj. The displayed multiplication combines the source factors. For tensors this is elementwise unless…
DeepSeek Attention.forward: q
In DeepseekV3Attention.forward, this statement defines or updates q. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…
DeepSeek Attention.forward: q
In DeepseekV3Attention.forward, this statement defines or updates q. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…
DeepSeek Attention.forward: attn weights
In DeepseekV3Attention.forward, this statement defines or updates attn_weights. The addition or subtraction updates the named state or combines residual contributions. The input…
Pad value vectors to the query-head width
The FlashAttention path appends zero-valued components to V when the value width is smaller than the query width. With the displayed 192 and 128 widths, it appends 64 zeros. The…
DeepSeek FlashAttention2._upad_input: query layer
In DeepseekV3FlashAttention2._upad_input, this statement defines or updates query_layer. The displayed multiplication combines the source factors. For tensors this is elementwise…
DeepSeek Model.forward: position ids
In DeepseekV3Model.forward, this statement defines or updates position_ids. The addition or subtraction updates the named state or combines residual contributions. The input…
DeepSeek ForSequenceClassification.forward: output
In DeepseekV3ForSequenceClassification.forward, this statement defines or updates output. The addition or subtraction updates the named state or combines residual contributions.…
DeepSeek ForSequenceClassification.forward: return
In DeepseekV3ForSequenceClassification.forward, this statement returns the displayed result. The addition or subtraction updates the named state or combines residual…
DeepSeek MoE.moe_infer: gatherd idxs[s:s + k]
In DeepseekV3MoE.moe_infer, this statement defines or updates gatherd_idxs[s:s + k]. This statement evaluates the named projection, normalization or numeric transform with the…
Apply the additional YaRN score adjustment
Multiply the existing inverse-square-root attention coefficient by mscale². Here mscale is an evaluated logarithmic adjustment, not an independent second query dimension. The…
DeepSeek FlashAttention2._upad_input: max seqlen in batch q
In DeepseekV3FlashAttention2._upad_input, this statement defines or updates max_seqlen_in_batch_q. This statement fixes a numeric parameter or initializes a numerical state in…
DeepSeek FlashAttention2._upad_input: cu seqlens q
In DeepseekV3FlashAttention2._upad_input, this statement defines or updates cu_seqlens_q. The addition or subtraction updates the named state or combines residual contributions.…
DeepSeek ForSequenceClassification.forward: sequence lengths
In DeepseekV3ForSequenceClassification.forward, this statement defines or updates sequence_lengths. The addition or subtraction updates the named state or combines residual…
TikTokenTokenizer: num reserved special tokens
In TikTokenTokenizer, this statement defines or updates num_reserved_special_tokens. This statement fixes a numeric parameter or initializes a numerical state in this source…
TikTokenTokenizer.__init__: special tokens
In TikTokenTokenizer.__init__, this statement defines or updates self.special_tokens. The addition or subtraction updates the named state or combines residual contributions. The…
TikTokenTokenizer.__init__: cache max size
In TikTokenTokenizer.__init__, this statement defines or updates self._cache_max_size. This statement fixes a numeric parameter or initializes a numerical state in this source…
TikTokenTokenizer.encode: TIKTOKEN MAX ENCODE CHARS
In TikTokenTokenizer.encode, this statement defines or updates TIKTOKEN_MAX_ENCODE_CHARS. This statement fixes a numeric parameter or initializes a numerical state in this source…
TikTokenTokenizer.encode: MAX NO WHITESPACES CHARS
In TikTokenTokenizer.encode, this statement defines or updates MAX_NO_WHITESPACES_CHARS. This statement fixes a numeric parameter or initializes a numerical state in this source…
TikTokenTokenizer._split_whitespaces_or_nonwhitespaces: current slice len
In TikTokenTokenizer._split_whitespaces_or_nonwhitespaces, this statement defines or updates current_slice_len. This statement fixes a numeric parameter or initializes a numerical…
TikTokenTokenizer._split_whitespaces_or_nonwhitespaces: slice start
In TikTokenTokenizer._split_whitespaces_or_nonwhitespaces, this statement defines or updates slice_start. This statement fixes a numeric parameter or initializes a numerical state…
TikTokenTokenizer.save_vocabulary: out vocab file
In TikTokenTokenizer.save_vocabulary, this statement defines or updates out_vocab_file. The addition or subtraction updates the named state or combines residual contributions. The…
TikTokenTokenizer.encode: substrs
In TikTokenTokenizer.encode, this statement defines or updates substrs. The addition or subtraction updates the named state or combines residual contributions. The input shapes…
TikTokenTokenizer._split_whitespaces_or_nonwhitespaces: current slice len
In TikTokenTokenizer._split_whitespaces_or_nonwhitespaces, this statement defines or updates current_slice_len. This statement fixes a numeric parameter or initializes a numerical…
TikTokenTokenizer._split_whitespaces_or_nonwhitespaces: current slice len
In TikTokenTokenizer._split_whitespaces_or_nonwhitespaces, this statement defines or updates current_slice_len. This statement fixes a numeric parameter or initializes a numerical…
KimiK25Processor.update_raw_text: text
In KimiK25Processor.update_raw_text, this statement defines or updates text. The addition or subtraction updates the named state or combines residual contributions. The input…
KimiK25VisionProcessor.split_video_chunks: sample fps
In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates sample_fps. This statement evaluates the named projection, normalization or numeric transform with…
KimiK25VisionProcessor.split_video_chunks: sampled nframes
In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates sampled_nframes. The expression forms a quotient or integer division. The denominator, integer…
KimiK25VisionProcessor.split_video_chunks: frame inds
In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates frame_inds. The addition or subtraction updates the named state or combines residual contributions.…
KimiK25VisionProcessor.split_video_chunks: num chunks
In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates num_chunks. This statement fixes a numeric parameter or initializes a numerical state in this…
KimiK25VisionProcessor.to_json_string: return
In KimiK25VisionProcessor.to_json_string, this statement returns the displayed result. The addition or subtraction updates the named state or combines residual contributions. The…
KimiK25VisionProcessor.split_video_chunks: start time
In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates start_time. The expression forms a quotient or integer division. The denominator, integer rounding…
KimiK25VisionProcessor.split_video_chunks: num chunks
In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates num_chunks. This statement fixes a numeric parameter or initializes a numerical state in this…
KimiK25VisionProcessor.split_video_chunks: chunk
In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates chunk. The displayed multiplication combines the source factors. For tensors this is elementwise…
KimiK25VisionProcessor.preprocess: image std inv
In KimiK25VisionProcessor.preprocess, this statement defines or updates image_std_inv. The expression forms a quotient or integer division. The denominator, integer rounding and…
KimiK25VisionProcessor.get_resize_config: fps
In KimiK25VisionProcessor.get_resize_config, this statement defines or updates fps. This statement fixes a numeric parameter or initializes a numerical state in this source…
media_utils.py: MediaInput
In media_utils.py, this statement defines or updates MediaInput. This statement evaluates the named projection, normalization or numeric transform with the source arguments. Its…
navit_resize_image: s1
In navit_resize_image, this statement defines or updates s1. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…
navit_resize_image: s2
In navit_resize_image, this statement defines or updates s2. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…
navit_resize_image: s3
In navit_resize_image, this statement defines or updates s3. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…
navit_resize_image: scale
In navit_resize_image, this statement defines or updates scale. This statement evaluates the named projection, normalization or numeric transform with the source arguments. Its…
navit_resize_image: (new w, new h)
In navit_resize_image, this statement defines or updates (new_w, new_h). The displayed multiplication combines the source factors. For tensors this is elementwise unless the…
navit_resize_image: new w
In navit_resize_image, this statement defines or updates new_w. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…
navit_resize_image: new h
In navit_resize_image, this statement defines or updates new_h. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…
navit_resize_image: factor
In navit_resize_image, this statement defines or updates factor. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…
navit_resize_image: pad height
In navit_resize_image, this statement defines or updates pad_height. The addition or subtraction updates the named state or combines residual contributions. The input shapes and…
navit_resize_image: pad width
In navit_resize_image, this statement defines or updates pad_width. The addition or subtraction updates the named state or combines residual contributions. The input shapes and…
navit_resize_video: sample fps
In navit_resize_video, this statement defines or updates sample_fps. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…
navit_resize_video: sampled nframes
In navit_resize_video, this statement defines or updates sampled_nframes. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape…
navit_patchify: patches
In navit_patchify, this statement defines or updates patches. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…
navit_patchify: grid thw
In navit_patchify, this statement defines or updates grid_thw. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…
normalize: x
In normalize, this statement defines or updates x. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention determine its…
navit_resize_image: token height
In navit_resize_image, this statement defines or updates token_height. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape…
navit_resize_image: token width
In navit_resize_image, this statement defines or updates token_width. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape…
navit_resize_image: num tokens
In navit_resize_image, this statement defines or updates num_tokens. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…
navit_resize_video: sampled nframes
In navit_resize_video, this statement defines or updates sampled_nframes. This statement evaluates the named projection, normalization or numeric transform with the source…
navit_resize_video: in patch limit each frame
In navit_resize_video, this statement defines or updates in_patch_limit_each_frame. The expression forms a quotient or integer division. The denominator, integer rounding and…
image_to_np: scale
In image_to_np, this statement defines or updates scale. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…
image_to_np: new width
In image_to_np, this statement defines or updates new_width. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…
image_to_np: new height
In image_to_np, this statement defines or updates new_height. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…
image_to_np: padding left
In image_to_np, this statement defines or updates padding_left. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…
image_to_np: padding right
In image_to_np, this statement defines or updates padding_right. The addition or subtraction updates the named state or combines residual contributions. The input shapes and the…
image_to_np: padding top
In image_to_np, this statement defines or updates padding_top. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…
image_to_np: padding bottom
In image_to_np, this statement defines or updates padding_bottom. The addition or subtraction updates the named state or combines residual contributions. The input shapes and the…
image_to_np: padding right
In image_to_np, this statement defines or updates padding_right. The addition or subtraction updates the named state or combines residual contributions. The input shapes and the…
image_to_np: padding bottom
In image_to_np, this statement defines or updates padding_bottom. The addition or subtraction updates the named state or combines residual contributions. The input shapes and the…
tool_declaration_ts.py: ParameterType
In tool_declaration_ts.py, this statement defines or updates _ParameterType. This statement evaluates the named projection, normalization or numeric transform with the source…
Sources & provenance
================================================================================ KIMI K2.5 — COMPLETE SOURCE TREE The official model as shipped, pulled from /Volumes/MEGAMIND/kimi_k25/model (M4). Everything: file tree, configs, full source code, and the complete parameter tree. Pulled: 2026-08-05. This is the reference for THE REAL FRONTIER (comparing our runtime engine to the OFFICIAL BF16 model). ================================================================================ This document is assembled from the actual files. It contains: PART 1 file tree (every file, size, role) — OFFICIAL vs OUR additions PART 2 architecture spec (from config.json) PART 3 the COMPLETE parameter / tensor tree (every tensor, dtype, shape, count) PART 4 ALL config files, verbatim PART 5 ALL source code, verbatim (configuration, modeling, tokenizer, processors) PART 6 docs / license, verbatim (Parts 3-6 are the raw shipped files, appended below this header.) ================================================================================ PART 1 — FILE TREE (/Volumes/MEGAMIND/kimi_k25/model, 595,148,192,736 bytes on disk)…
# ROUTED EXPERTS (the int4 intelligence bulk): when exact generators are on, the cells ARE # PackedInt4Linear whose buffers (weight_packed, weight_scale) match the shipped checkpoint keys — so the # native int4 loads straight in with zero conversion, never expands to bf16, and computes the MAC exactly. if is_routed_expert and getattr(config, "use_exact_generators", False): from exact_generators import PackedInt4Linear ab = getattr(config, "exact_act_bits", 16) self.gate_proj = PackedInt4Linear(self.hidden_size, self.intermediate_size, act_bits=ab) self.up_proj = PackedInt4Linear(self.hidden_size, self.intermediate_size, act_bits=ab) self.down_proj = PackedInt4Linear(self.intermediate_size, self.hidden_size, act_bits=ab) else: # shared expert + dense layer stay bf16 nn.Linear self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False) self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False) self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size,…
self.hidden_size = config.mm_hidden_size * ( config.merge_kernel_size[0] * config.merge_kernel_size[1]) self.pre_norm = nn.LayerNorm(config.mm_hidden_size, eps=eps) self.proj = nn.Sequential( nn.Linear(self.hidden_size, self.hidden_size), nn.GELU(), nn.Linear(self.hidden_size, config.hidden_size),
class ProjectorConfig: def __init__(self, config: KimiK25Config): self.mm_projector_type = config.mm_projector_type self.mm_hidden_size = config.mm_hidden_size self.hidden_size = config.text_hidden_size self.merge_kernel_size = config.merge_kernel_size