kimi source tree.txt · Consciousness Atlas

Source identity

Field	Recorded value
Source type	donor model source and installation dossier
Original record	kimi source tree.txt
Date	Aug 5, 2026
Path or archive member	kimi source tree.txt
Original SHA-256	fc220361f1964691f7eab7c00ae3b669ea6f2affe33daf493aaa420488483fa8
Reading-copy SHA-256	da4fa167beba964a275402e6fd9ca8bd53de9e6a31d7e4af3fd02414dfb006e5

Recovered source transcript

Download complete source text ↓6,675 lines · 275.4 KB

================================================================================
KIMI K2.5 — COMPLETE SOURCE TREE
The official model as shipped, pulled from /Volumes/MEGAMIND/kimi_k25/model (M4).
Everything: file tree, configs, full source code, and the complete parameter tree.
Pulled: 2026-08-05.  This is the reference for THE REAL FRONTIER (comparing our
runtime engine to the OFFICIAL BF16 model).
================================================================================

This document is assembled from the actual files. It contains:
  PART 1  file tree (every file, size, role) — OFFICIAL vs OUR additions
  PART 2  architecture spec (from config.json)
  PART 3  the COMPLETE parameter / tensor tree (every tensor, dtype, shape, count)
  PART 4  ALL config files, verbatim
  PART 5  ALL source code, verbatim (configuration, modeling, tokenizer, processors)
  PART 6  docs / license, verbatim
(Parts 3-6 are the raw shipped files, appended below this header.)


================================================================================
PART 1 — FILE TREE  (/Volumes/MEGAMIND/kimi_k25/model, 595,148,192,736 bytes on disk)
================================================================================
OFFICIAL SHIPPED FILES
  config.json                       5351   top model config (KimiK25 + text + vision)
  configuration_kimi_k25.py         5435   KimiK25Config
  configuration_deepseek.py        10615   DeepseekV3Config (the language model)
  modeling_kimi_k25.py             50912   KimiK25ForConditionalGeneration (wrapper+vision)
  modeling_deepseek.py             80663   DeepseekV3ForCausalLM (the 61-layer LM)
  generation_config.json              52   eos 163586, max_length 262144
  preprocessor_config.json           742   MoonViT media processor config
  tokenizer_config.json             4862   tokenizer settings + special tokens
  tokenization_kimi.py             13353   the tokenizer class
  tiktoken.model                 2795286   the BPE merges (tiktoken)
  kimi_k25_processor.py             6906   KimiK25Processor (text+media)
  kimi_k25_vision_processing.py    10019   KimiK25VisionProcessor (MoonViT)
  media_utils.py                   13179   media (image/video) helpers
  tool_declaration_ts.py           16517   tool-calling / TS declarations
  __init__.py                          0   package marker
  .gitattributes                    1643   LFS attributes
  README.md                        40082   model card
  docs/deploy_guidance.md           3233   deployment guide
  figures/kimi-logo.png            87988 | figures/demo_video.mp4  270100
  LICENSE                           1465 | THIRD_PARTY_NOTICES.md   1662
  model.safetensors.index.json             weight_map (208,550 tensors -> 64 shards)
  model-00001..00064-of-000064.safetensors   the 64 weight shards (~595 GB)

OUR ADDITIONS (kimi-blanket work, dropped into the model dir — NOT official)
  exact_generators.py   ExactLinear + PackedInt4Linear (int64 exact int4 MAC)
  load_exact.py         convert_kimi_to_exact loader
  test_exact_expert.py / test_exact_native.py   our int4-exactness tests
  activation_geometry.py  measure_compaction.py   our analysis scripts
  __pycache__/modeling_kimi_k25.cpython-314.pyc  (byte-compiled cache)


================================================================================
PART 2 — ARCHITECTURE SPEC  (from config.json)
================================================================================
TOP (KimiK25ForConditionalGeneration, model_type "kimi_k25", dtype bfloat16)
  bos 163584 · eos 163585 (generation eos 163586) · pad 163839
  media_placeholder_token_id 163605
  auto_map -> configuration_kimi_k25.KimiK25Config / modeling_kimi_k25.KimiK25ForConditionalGeneration

TEXT (DeepseekV3ForCausalLM)
  hidden_size 7168 · num_hidden_layers 61 · vocab_size 163840 · hidden_act silu
  intermediate_size 18432 (dense MLP)  · rms_norm_eps 1e-5
  MLA attention: num_attention_heads 64 · num_key_value_heads 64
     q_lora_rank 1536 · kv_lora_rank 512
     qk_nope_head_dim 128 · qk_rope_head_dim 64 · v_head_dim 128
  MoE: first_k_dense_replace 1 (layer 0 dense; layers 1-60 MoE)
     n_routed_experts 384 · num_experts_per_tok 8 · n_shared_experts 1
     moe_intermediate_size 2048 · moe_layer_freq 1 · routed_scaling_factor 2.827
     scoring_func sigmoid · topk_method noaux_tc · n_group 1 · topk_group 1
     norm_topk_prob true · aux_loss_alpha 0.001
  RoPE: rope_theta 50000 · max_position_embeddings 262144
     rope_scaling = yarn, factor 64, beta_fast 32, beta_slow 1, mscale 1,
     mscale_all_dim 1, original_max_position_embeddings 4096

VISION (MoonViT-400M, vision_tower + mm_projector)
  patch_size 14 · vt_hidden_size 1152 · vt_intermediate_size 4304
  vt_num_attention_heads 16 · vt_num_hidden_layers 27
  pos_emb_type divided_fixed · init_pos_emb 64x64 (h x w), time 4
  merge_kernel_size [2,2] · merge_type sd2_tpool · video_attn spatial_temporal
  mm_hidden_size 1152 · mm_projector_type patchmerger · projector_hidden_act gelu
  text_hidden_size 7168 (projects vision -> LM residual)

QUANTIZATION
  No quantization_config block. Routed experts are shipped int4 in compressed-tensors
  form (weight_packed I32 + weight_scale BF16 + weight_shape I32). Everything else bf16.
  int4 pack: 8 nibbles per int32; per-group scale, GROUP SIZE 32 (scale cols = in/32).


================================================================================
PART 3 — COMPLETE PARAMETER / TENSOR TREE  (208,550 tensors)
================================================================================
Structure below is the exact per-pattern dtype/shape/count from the shard headers.
Per-layer patterns repeat across the 61 layers (MoE-only patterns across 60 layers,
384 experts each). The full machine-generated list follows.

  LANGUAGE MODEL (DeepseekV3)
    embed_tokens.weight ............... BF16 [163840, 7168]     (x1)
    per layer (x61):
      input_layernorm.weight .......... BF16 [7168]
      post_attention_layernorm.weight . BF16 [7168]
      self_attn.q_a_proj.weight ....... BF16 [1536, 7168]      (down to q_lora)
      self_attn.q_a_layernorm.weight .. BF16 [1536]
      self_attn.q_b_proj.weight ....... BF16 [12288, 1536]     (64 heads x (128+64))
      self_attn.kv_a_proj_with_mqa .... BF16 [576, 7168]       (512 kv_lora + 64 rope)
      self_attn.kv_a_layernorm.weight . BF16 [512]
      self_attn.kv_b_proj.weight ...... BF16 [16384, 512]      (64 heads x (128+128))
      self_attn.o_proj.weight ......... BF16 [7168, 8192]      (64 heads x v128)
    layer 0 ONLY (dense MLP, x1 each):
      mlp.gate_proj.weight ............ BF16 [18432, 7168]
      mlp.up_proj.weight .............. BF16 [18432, 7168]
      mlp.down_proj.weight ............ BF16 [7168, 18432]
    layers 1-60 (MoE, x60 each):
      mlp.gate.weight ................. BF16 [384, 7168]       (the router)
      mlp.gate.e_score_correction_bias  F32  [384]            (** F32, not bf16 -> our bug **)
      mlp.shared_experts.gate_proj .... BF16 [2048, 7168]
      mlp.shared_experts.up_proj ...... BF16 [2048, 7168]
      mlp.shared_experts.down_proj .... BF16 [7168, 2048]
      mlp.experts.{0..383}. (x384 per layer = 23040 total each):
        gate_proj.weight_packed ....... I32  [2048, 896]      (int4: 896*8 = 7168 in)
        gate_proj.weight_scale ........ BF16 [2048, 224]      (group 32: 7168/32 = 224)
        gate_proj.weight_shape ........ I32  [2]
        up_proj.weight_packed ......... I32  [2048, 896]
        up_proj.weight_scale .......... BF16 [2048, 224]
        up_proj.weight_shape .......... I32  [2]
        down_proj.weight_packed ....... I32  [7168, 256]      (int4: 256*8 = 2048 in)
        down_proj.weight_scale ........ BF16 [7168, 64]       (group 32: 2048/32 = 64)
        down_proj.weight_shape ........ I32  [2]
    norm.weight ....................... BF16 [7168]            (final RMSNorm, x1)
    lm_head.weight .................... BF16 [163840, 7168]    (x1)

  VISION TOWER (MoonViT-400M)
    patch_embed.proj.weight ........... BF16 [1152, 3, 14, 14] (conv patchify)
    patch_embed.proj.bias ............. BF16 [1152]
    patch_embed.pos_emb.weight ........ BF16 [64, 64, 1152]
    per block (x27):
      norm0.weight/bias ............... BF16 [1152] / [1152]
      norm1.weight/bias ............... BF16 [1152] / [1152]
      wqkv.weight/bias ................ BF16 [3456, 1152] / [3456]   (fused QKV = 3x1152)
      wo.weight/bias .................. BF16 [1152, 1152] / [1152]
      mlp.fc0.weight/bias ............. BF16 [4304, 1152] / [4304]
      mlp.fc1.weight/bias ............. BF16 [1152, 4304] / [1152]
    encoder.final_layernorm.weight/bias BF16 [1152] / [1152]
  MM PROJECTOR (patchmerger, vision -> LM)
    mm_projector.pre_norm.weight/bias . BF16 [1152] / [1152]
    mm_projector.proj.{0,1}.weight/bias (2 linear layers -> 7168)

--- machine-generated full pattern table (dtype, shape, count) below ---
total tensors: 208550   |   distinct patterns: 50   |   on-disk: 595148192736 bytes
(pattern = tensor name with layer/expert indices replaced by N; xCOUNT = how many)

  x1      BF16     [163840, 7168]           language_model.lm_head.weight
  x1      BF16     [163840, 7168]           language_model.model.embed_tokens.weight
  x61     BF16     [7168]                   language_model.model.layers.N.input_layernorm.weight
  x1      BF16     [7168, 18432]            language_model.model.layers.N.mlp.down_proj.weight
  x23040  I32      [7168, 256]              language_model.model.layers.N.mlp.experts.N.down_proj.weight_packed
  x23040  BF16     [7168, 64]               language_model.model.layers.N.mlp.experts.N.down_proj.weight_scale
  x23040  I32      [2]                      language_model.model.layers.N.mlp.experts.N.down_proj.weight_shape
  x23040  I32      [2048, 896]              language_model.model.layers.N.mlp.experts.N.gate_proj.weight_packed
  x23040  BF16     [2048, 224]              language_model.model.layers.N.mlp.experts.N.gate_proj.weight_scale
  x23040  I32      [2]                      language_model.model.layers.N.mlp.experts.N.gate_proj.weight_shape
  x23040  I32      [2048, 896]              language_model.model.layers.N.mlp.experts.N.up_proj.weight_packed
  x23040  BF16     [2048, 224]              language_model.model.layers.N.mlp.experts.N.up_proj.weight_scale
  x23040  I32      [2]                      language_model.model.layers.N.mlp.experts.N.up_proj.weight_shape
  x60     F32      [384]                    language_model.model.layers.N.mlp.gate.e_score_correction_bias
  x60     BF16     [384, 7168]              language_model.model.layers.N.mlp.gate.weight
  x1      BF16     [18432, 7168]            language_model.model.layers.N.mlp.gate_proj.weight
  x60     BF16     [7168, 2048]             language_model.model.layers.N.mlp.shared_experts.down_proj.weight
  x60     BF16     [2048, 7168]             language_model.model.layers.N.mlp.shared_experts.gate_proj.weight
  x60     BF16     [2048, 7168]             language_model.model.layers.N.mlp.shared_experts.up_proj.weight
  x1      BF16     [18432, 7168]            language_model.model.layers.N.mlp.up_proj.weight
  x61     BF16     [7168]                   language_model.model.layers.N.post_attention_layernorm.weight
  x61     BF16     [512]                    language_model.model.layers.N.self_attn.kv_a_layernorm.weight
  x61     BF16     [576, 7168]              language_model.model.layers.N.self_attn.kv_a_proj_with_mqa.weight
  x61     BF16     [16384, 512]             language_model.model.layers.N.self_attn.kv_b_proj.weight
  x61     BF16     [7168, 8192]             language_model.model.layers.N.self_attn.o_proj.weight
  x61     BF16     [1536]                   language_model.model.layers.N.self_attn.q_a_layernorm.weight
  x61     BF16     [1536, 7168]             language_model.model.layers.N.self_attn.q_a_proj.weight
  x61     BF16     [12288, 1536]            language_model.model.layers.N.self_attn.q_b_proj.weight
  x1      BF16     [7168]                   language_model.model.norm.weight
  x1      BF16     [1152]                   mm_projector.pre_norm.bias
  x1      BF16     [1152]                   mm_projector.pre_norm.weight
  x2      BF16     [4608]                   mm_projector.proj.N.bias
  x2      BF16     [4608, 4608]             mm_projector.proj.N.weight
  x27     BF16     [4304]                   vision_tower.encoder.blocks.N.mlp.fc0.bias
  x27     BF16     [4304, 1152]             vision_tower.encoder.blocks.N.mlp.fc0.weight
  x27     BF16     [1152]                   vision_tower.encoder.blocks.N.mlp.fc1.bias
  x27     BF16     [1152, 4304]             vision_tower.encoder.blocks.N.mlp.fc1.weight
  x27     BF16     [1152]                   vision_tower.encoder.blocks.N.norm0.bias
  x27     BF16     [1152]                   vision_tower.encoder.blocks.N.norm0.weight
  x27     BF16     [1152]                   vision_tower.encoder.blocks.N.norm1.bias
  x27     BF16     [1152]                   vision_tower.encoder.blocks.N.norm1.weight
  x27     BF16     [1152]                   vision_tower.encoder.blocks.N.wo.bias
  x27     BF16     [1152, 1152]             vision_tower.encoder.blocks.N.wo.weight
  x27     BF16     [3456]                   vision_tower.encoder.blocks.N.wqkv.bias
  x27     BF16     [3456, 1152]             vision_tower.encoder.blocks.N.wqkv.weight
  x1      BF16     [1152]                   vision_tower.encoder.final_layernorm.bias
  x1      BF16     [1152]                   vision_tower.encoder.final_layernorm.weight
  x1      BF16     [64, 64, 1152]           vision_tower.patch_embed.pos_emb.weight
  x1      BF16     [1152]                   vision_tower.patch_embed.proj.bias
  x1      BF16     [1152, 3, 14, 14]        vision_tower.patch_embed.proj.weight

================================================================================
PART 4 - CONFIG FILES (verbatim)
================================================================================

>>>>> FILE: config.json <<<<<
{
  "architectures": [
    "KimiK25ForConditionalGeneration"
  ],
  "auto_map": {
    "AutoConfig": "configuration_kimi_k25.KimiK25Config",
    "AutoModel": "modeling_kimi_k25.KimiK25ForConditionalGeneration",
    "AutoModelForCausalLM": "modeling_kimi_k25.KimiK25ForConditionalGeneration"
  },
  "bos_token_id": 163584,
  "dtype": "bfloat16",
  "eos_token_id": 163585,
  "ignore_index": -100,
  "media_placeholder_token_id": 163605,
  "model_type": "kimi_k25",
  "pad_token_id": 163839,
  "text_config": {
    "_name_or_path": "",
    "add_cross_attention": false,
    "architectures": [
      "DeepseekV3ForCausalLM"
    ],
    "attention_bias": false,
    "attention_dropout": 0.0,
    "auto_map": {
      "AutoConfig": "configuration_deepseek.DeepseekV3Config",
      "AutoModel": "modeling_deepseek.DeepseekV3Model",
      "AutoModelForCausalLM": "modeling_deepseek.DeepseekV3ForCausalLM"
    },
    "aux_loss_alpha": 0.001,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": 163584,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "dtype": "bfloat16",
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 163585,
    "ep_size": 1,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "first_k_dense_replace": 1,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 7168,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 18432,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "kv_lora_rank": 512,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 262144,
    "min_length": 0,
    "model_type": "kimi_k2",
    "moe_intermediate_size": 2048,
    "moe_layer_freq": 1,
    "n_group": 1,
    "n_routed_experts": 384,
    "n_shared_experts": 1,
    "no_repeat_ngram_size": 0,
    "norm_topk_prob": true,
    "num_attention_heads": 64,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_experts_per_tok": 8,
    "num_hidden_layers": 61,
    "num_key_value_heads": 64,
    "num_nextn_predict_layers": 0,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": 163839,
    "prefix": null,
    "pretraining_tp": 1,
    "problem_type": null,
    "pruned_heads": {},
    "q_lora_rank": 1536,
    "qk_nope_head_dim": 128,
    "qk_rope_head_dim": 64,
    "quantization_config": {
      "config_groups": {
        "group_0": {
          "input_activations": null,
          "output_activations": null,
          "targets": [
            "Linear"
          ],
          "weights": {
            "actorder": null,
            "block_structure": null,
            "dynamic": false,
            "group_size": 32,
            "num_bits": 4,
            "observer": "minmax",
            "observer_kwargs": {},
            "strategy": "group",
            "symmetric": true,
            "type": "int"
          }
        }
      },
      "format": "pack-quantized",
      "ignore": [
        "lm_head",
        "re:.*self_attn.*",
        "re:.*shared_experts.*",
        "re:.*mlp\\.(gate|up|gate_up|down)_proj.*"
      ],
      "kv_cache_scheme": null,
      "quant_method": "compressed-tensors",
      "quantization_status": "compressed"
    },
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-05,
    "rope_scaling": {
      "beta_fast": 32.0,
      "beta_slow": 1.0,
      "factor": 64.0,
      "mscale": 1.0,
      "mscale_all_dim": 1.0,
      "original_max_position_embeddings": 4096,
      "type": "yarn"
    },
    "rope_theta": 50000.0,
    "routed_scaling_factor": 2.827,
    "scoring_func": "sigmoid",
    "sep_token_id": null,
    "seq_aux": true,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "topk_group": 1,
    "topk_method": "noaux_tc",
    "torchscript": false,
    "transformers_version": "4.56.2",
    "typical_p": 1.0,
    "use_bfloat16": false,
    "use_cache": true,
    "v_head_dim": 128,
    "vocab_size": 163840
  },
  "tie_word_embeddings": false,
  "use_unified_vision_chunk": true,
  "video_placeholder": "<|kimi_k25_video_placeholder|>",
  "vision_config": {
    "_attn_implementation": "flash_attention_2",
    "init_pos_emb_height": 64,
    "init_pos_emb_time": 4,
    "init_pos_emb_width": 64,
    "merge_kernel_size": [
      2,
      2
    ],
    "merge_type": "sd2_tpool",
    "mm_hidden_size": 1152,
    "mm_projector_type": "patchmerger",
    "patch_size": 14,
    "pos_emb_type": "divided_fixed",
    "projector_hidden_act": "gelu",
    "projector_ln_eps": 1e-05,
    "text_hidden_size": 7168,
    "video_attn_type": "spatial_temporal",
    "vt_hidden_size": 1152,
    "vt_intermediate_size": 4304,
    "vt_num_attention_heads": 16,
    "vt_num_hidden_layers": 27
  }
}

>>>>> FILE: generation_config.json <<<<<
{
  "max_length": 262144,
  "eos_token_id": 163586
}

>>>>> FILE: preprocessor_config.json <<<<<
{
  "auto_map": {
    "AutoProcessor": "kimi_k25_processor.KimiK25Processor",
    "AutoImageProcessor": "kimi_k25_vision_processing.KimiK25VisionProcessor"
  },
  "media_proc_cfg": {
    "in_patch_limit": 16384,
    "patch_size": 14,
    "image_mean": [
      0.5,
      0.5,
      0.5
    ],
    "image_std": [
      0.5,
      0.5,
      0.5
    ],
    "merge_kernel_size": 2,
    "fixed_output_tokens": null,
    "patch_limit_on_one_side": 512,
    "in_patch_limit_each_frame": 4096,
    "in_patch_limit_video": null,
    "sample_fps": 2.0,
    "max_num_frames_each_video": null,
    "temporal_merge_kernel_size": 4,
    "timestamp_mode": "hh:mm:ss.fff",
    "config_type": "media_proc.processors.moonvit.MoonViTMediaProcessorConfig"
  }
}

>>>>> FILE: tokenizer_config.json <<<<<
{
  "added_tokens_decoder": {
    "163584": {
      "content": "[BOS]",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163585": {
      "content": "[EOS]",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163586": {
      "content": "<|im_end|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163587": {
      "content": "<|im_user|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163588": {
      "content": "<|im_assistant|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163590": {
      "content": "<|start_header_id|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163591": {
      "content": "<|end_header_id|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163593": {
      "content": "[EOT]",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163594": {
      "content": "<|im_system|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163595": {
      "content": "<|tool_calls_section_begin|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": false
    },
    "163596": {
      "content": "<|tool_calls_section_end|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": false
    },
    "163597": {
      "content": "<|tool_call_begin|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": false
    },
    "163598": {
      "content": "<|tool_call_argument_begin|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": false
    },
    "163599": {
      "content": "<|tool_call_end|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": false
    },
    "163601": {
      "content": "<|im_middle|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163602": {
      "content": "<|media_begin|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163603": {
      "content": "<|media_content|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163604": {
      "content": "<|media_end|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163605": {
      "content": "<|media_pad|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163606": {
      "content": "<think>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": false
    },
    "163607": {
      "content": "</think>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": false
    },
    "163838": {
      "content": "[UNK]",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "163839": {
      "content": "[PAD]",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    }
  },
  "additional_special_tokens": [
    "<|im_end|>",
    "<|im_user|>",
    "<|im_assistant|>",
    "<|start_header_id|>",
    "<|end_header_id|>",
    "[EOT]",
    "<|im_system|>",
    "<|im_middle|>",
    "<|media_begin|>",
    "<|media_content|>",
    "<|media_end|>",
    "<|media_pad|>"
  ],
  "bos_token": "[BOS]",
  "clean_up_tokenization_spaces": false,
  "eos_token": "[EOS]",
  "extra_special_tokens": {},
  "model_max_length": 1000000000000000019884624838656,
  "pad_token": "[PAD]",
  "tokenizer_class": "TikTokenTokenizer",
  "unk_token": "[UNK]",
  "auto_map": {
    "AutoTokenizer": [
      "tokenization_kimi.TikTokenTokenizer",
      null
    ]
  }
}

>>>>> FILE: .gitattributes <<<<<
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text
figures/demo_video.mp4 filter=lfs diff=lfs merge=lfs -text


================================================================================
PART 5 - SOURCE CODE (verbatim)
================================================================================

>>>>> FILE: __init__.py (0 bytes) <<<<<


>>>>> FILE: configuration_kimi_k25.py (5435 bytes) <<<<<
from transformers.configuration_utils import PretrainedConfig

try:
    from configuration_deepseek import DeepseekV3Config
except ImportError:
    from .configuration_deepseek import DeepseekV3Config


class KimiK25VisionConfig(PretrainedConfig):

    def __init__(
            self,
            patch_size: int = 14,
            init_pos_emb_height: int = 64,
            init_pos_emb_width: int = 64,
            init_pos_emb_time: int = 4,
            pos_emb_type: str = 'divided_fixed',
            vt_num_attention_heads: int = 16,
            vt_num_hidden_layers: int = 27,
            vt_hidden_size: int = 1152,
            vt_intermediate_size: int = 4304,
            merge_kernel_size: tuple = (2, 2),
            video_attn_type: str = 'spatial_temporal',
            merge_type: str = 'sd2_tpool',
            _attn_implementation: str = 'flash_attention_2',
            # MM Projector parameters
            mm_projector_type: str = 'patchmerger',
            mm_hidden_size: int | None = None,
            projector_hidden_act: str = "gelu",
            projector_ln_eps: float = 1e-5,
            # Other parameters
            ignore_index: int = -100,
            media_placeholder_token_id: int = 163605,
            pad_token_id: int = 0,
            use_unified_vision_chunk: bool = True,
            video_placeholder="<|kimi_k25_video_placeholder|>",
            text_hidden_size=7168,
            **vision_config_kwargs):

        self.patch_size = patch_size
        self.init_pos_emb_height = init_pos_emb_height
        self.init_pos_emb_width = init_pos_emb_width
        self.init_pos_emb_time = init_pos_emb_time
        self.pos_emb_type = pos_emb_type
        self.vt_num_attention_heads = vt_num_attention_heads
        self.vt_num_hidden_layers = vt_num_hidden_layers
        self.vt_hidden_size = vt_hidden_size
        self.vt_intermediate_size = vt_intermediate_size
        self.merge_kernel_size = merge_kernel_size
        self.video_attn_type = video_attn_type
        self.merge_type = merge_type
        self._attn_implementation = _attn_implementation

        # MM Projector config
        self.mm_projector_type = mm_projector_type
        self.mm_hidden_size = mm_hidden_size if mm_hidden_size is not None else vt_hidden_size
        self.projector_hidden_act = projector_hidden_act
        self.projector_ln_eps = projector_ln_eps
        self.text_hidden_size = text_hidden_size


class KimiK25Config(PretrainedConfig):
    """Kimi-K2.5 model configuration.

    Args:
        text_config (dict | DeepseekV3Config): Configuration for the text model.
        
        Vision Tower Parameters (from MoonViT3dConfig):
            patch_size (int): Patch size for vision tower.
            init_pos_emb_height (int): Initial position embedding height.
            init_pos_emb_width (int): Initial position embedding width.
            init_pos_emb_time (int): Initial position embedding time dimension.
            pos_emb_type (str): Type of position embedding.
            vt_num_attention_heads (int): Number of attention heads in vision tower.
            vt_num_hidden_layers (int): Number of hidden layers in vision tower.
            vt_hidden_size (int): Hidden size of vision tower.
            vt_intermediate_size (int): Intermediate size in vision tower FFN.
            merge_kernel_size (tuple): Kernel size for patch merging.
            video_attn_type (str): Type of video attention.
            merge_type (str): Type of merge operation.
            _attn_implementation (str): Attention implementation type.
        
        MM Projector Parameters (from MultiModalProjectorConfig):
            mm_projector_type (str): Type of multimodal projector.
            mm_hidden_size (int): Hidden size from vision tower (should match vt_hidden_size).
            projector_hidden_act (str): Activation function for projector.
            projector_ln_eps (float): Layer norm epsilon for projector.
        
        Other Parameters:
            ignore_index (int): The ignore index for the loss function.
            media_placeholder_token_id (int): The token ID to use for media placeholders.
            pad_token_id (int): The token ID to use for padding.
    """

    model_type = "kimi_k25"

    def __init__(
        self,
        text_config: dict | DeepseekV3Config = None,
        vision_config: dict | KimiK25VisionConfig = None,
        # Other parameters
        ignore_index: int = -100,
        media_placeholder_token_id: int = 163605,
        pad_token_id: int = 0,
        use_unified_vision_chunk: bool = True,
        video_placeholder="<|kimi_k25_video_placeholder|>",
        **kwargs,
    ):
        if isinstance(text_config, dict):
            text_config = DeepseekV3Config(**text_config)
        if isinstance(vision_config, dict):
            vision_config = KimiK25VisionConfig(**vision_config)
        self.text_config = text_config
        self.vision_config = vision_config
        # Other config
        self.ignore_index = ignore_index
        self.media_placeholder_token_id = media_placeholder_token_id
        self.use_unified_vision_chunk = use_unified_vision_chunk
        self.video_placeholder = video_placeholder
        if getattr(self.text_config, "quantization_config", None) is not None:
            self.quantization_config = self.text_config.quantization_config

        super().__init__(pad_token_id=pad_token_id, **kwargs)


>>>>> FILE: configuration_deepseek.py (10615 bytes) <<<<<
# Copy from https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/configuration_deepseek.py

from transformers.configuration_utils import PretrainedConfig
from transformers.utils import logging

logger = logging.get_logger(__name__)

DEEPSEEK_PRETRAINED_CONFIG_ARCHIVE_MAP = {}


class DeepseekV3Config(PretrainedConfig):
    r"""
    This is the configuration class to store the configuration of a [`DeepseekV3Model`]. It is used to instantiate an DeepSeek
    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
    defaults will yield a similar configuration to that of the DeepSeek-V3.

    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
    documentation from [`PretrainedConfig`] for more information.


    Args:
        vocab_size (`int`, *optional*, defaults to 129280):
            Vocabulary size of the Deep model. Defines the number of different tokens that can be represented by the
            `inputs_ids` passed when calling [`DeepseekV3Model`]
        hidden_size (`int`, *optional*, defaults to 4096):
            Dimension of the hidden representations.
        intermediate_size (`int`, *optional*, defaults to 11008):
            Dimension of the MLP representations.
        moe_intermediate_size (`int`, *optional*, defaults to 1407):
            Dimension of the MoE representations.
        num_hidden_layers (`int`, *optional*, defaults to 32):
            Number of hidden layers in the Transformer decoder.
        num_nextn_predict_layers (`int`, *optional*, defaults to 1):
            Number of nextn predict layers in the DeepSeekV3 Model.
        num_attention_heads (`int`, *optional*, defaults to 32):
            Number of attention heads for each attention layer in the Transformer decoder.
        n_shared_experts (`int`, *optional*, defaults to None):
            Number of shared experts, None means dense model.
        n_routed_experts (`int`, *optional*, defaults to None):
            Number of routed experts, None means dense model.
        routed_scaling_factor (`float`, *optional*, defaults to 1.0):
            Scaling factor or routed experts.
        topk_method (`str`, *optional*, defaults to `gready`):
            Topk method used in routed gate.
        n_group (`int`, *optional*, defaults to None):
            Number of groups for routed experts.
        topk_group (`int`, *optional*, defaults to None):
            Number of selected groups for each token(for each token, ensuring the selected experts is only within `topk_group` groups).
        num_experts_per_tok (`int`, *optional*, defaults to None):
            Number of selected experts, None means dense model.
        moe_layer_freq (`int`, *optional*, defaults to 1):
            The frequency of the MoE layer: one expert layer for every `moe_layer_freq - 1` dense layers.
        first_k_dense_replace (`int`, *optional*, defaults to 0):
            Number of dense layers in shallow layers(embed->dense->dense->...->dense->moe->moe...->lm_head).
                                                            \--k dense layers--/
        norm_topk_prob (`bool`, *optional*, defaults to False):
            Whether to normalize the weights of the routed experts.
        scoring_func (`str`, *optional*, defaults to 'softmax'):
            Method of computing expert weights.
        aux_loss_alpha (`float`, *optional*, defaults to 0.001):
            Auxiliary loss weight coefficient.
        seq_aux = (`bool`, *optional*, defaults to True):
            Whether to compute the auxiliary loss for each individual sample.
        num_key_value_heads (`int`, *optional*):
            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
            `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
            by meanpooling all the original heads within that group. For more details checkout [this
            paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
            `num_attention_heads`.
        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
            The non-linear activation function (function or string) in the decoder.
        max_position_embeddings (`int`, *optional*, defaults to 2048):
            The maximum sequence length that this model might ever be used with.
        initializer_range (`float`, *optional*, defaults to 0.02):
            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
        rms_norm_eps (`float`, *optional*, defaults to 1e-06):
            The epsilon used by the rms normalization layers.
        use_cache (`bool`, *optional*, defaults to `True`):
            Whether or not the model should return the last key/values attentions (not used by all models). Only
            relevant if `config.is_decoder=True`.
        pad_token_id (`int`, *optional*):
            Padding token id.
        bos_token_id (`int`, *optional*, defaults to 1):
            Beginning of stream token id.
        eos_token_id (`int`, *optional*, defaults to 2):
            End of stream token id.
        pretraining_tp (`int`, *optional*, defaults to 1):
            Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this
            document](https://huggingface.co/docs/transformers/parallelism) to understand more about it. This value is
            necessary to ensure exact reproducibility of the pretraining results. Please refer to [this
            issue](https://github.com/pytorch/pytorch/issues/76232).
        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
            Whether to tie weight embeddings
        rope_theta (`float`, *optional*, defaults to 10000.0):
            The base period of the RoPE embeddings.
        rope_scaling (`Dict`, *optional*):
            Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
            strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
            `{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update
            `max_position_embeddings` to the expected new maximum.
        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
            Whether to use a bias in the query, key, value and output projection layers during self-attention.
        attention_dropout (`float`, *optional*, defaults to 0.0):
            The dropout ratio for the attention probabilities.

    ```python
    >>> from transformers import DeepseekV3Model, DeepseekV3Config

    >>> # Initializing a Deepseek-V3 style configuration
    >>> configuration = DeepseekV3Config()

    >>> # Accessing the model configuration
    >>> configuration = model.config
    ```"""

    model_type = "deepseek_v3"
    keys_to_ignore_at_inference = ["past_key_values"]

    def __init__(
        self,
        vocab_size=129280,
        hidden_size=7168,
        intermediate_size=18432,
        moe_intermediate_size=2048,
        num_hidden_layers=61,
        num_nextn_predict_layers=1,
        num_attention_heads=128,
        num_key_value_heads=128,
        n_shared_experts=1,
        n_routed_experts=256,
        ep_size=1,
        routed_scaling_factor=2.5,
        kv_lora_rank=512,
        q_lora_rank=1536,
        qk_rope_head_dim=64,
        v_head_dim=128,
        qk_nope_head_dim=128,
        topk_method='noaux_tc',
        n_group=8,
        topk_group=4,
        num_experts_per_tok=8,
        moe_layer_freq=1,
        first_k_dense_replace=3,
        norm_topk_prob=True,
        scoring_func='sigmoid',
        aux_loss_alpha=0.001,
        seq_aux=True,
        hidden_act="silu",
        max_position_embeddings=4096,
        initializer_range=0.02,
        rms_norm_eps=1e-6,
        use_cache=True,
        pad_token_id=None,
        bos_token_id=0,
        eos_token_id=1,
        pretraining_tp=1,
        tie_word_embeddings=False,
        rope_theta=10000.0,
        rope_scaling=None,
        attention_bias=False,
        attention_dropout=0.0,
        **kwargs,
    ):
        self.vocab_size = vocab_size
        self.max_position_embeddings = max_position_embeddings
        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.moe_intermediate_size = moe_intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_nextn_predict_layers = num_nextn_predict_layers
        self.num_attention_heads = num_attention_heads
        self.n_shared_experts = n_shared_experts
        self.n_routed_experts = n_routed_experts
        self.ep_size = ep_size
        self.routed_scaling_factor = routed_scaling_factor
        self.kv_lora_rank = kv_lora_rank
        self.q_lora_rank = q_lora_rank
        self.qk_rope_head_dim = qk_rope_head_dim
        self.v_head_dim = v_head_dim
        self.qk_nope_head_dim = qk_nope_head_dim
        self.topk_method = topk_method
        self.n_group = n_group
        self.topk_group = topk_group
        self.num_experts_per_tok = num_experts_per_tok
        self.moe_layer_freq = moe_layer_freq
        self.first_k_dense_replace = first_k_dense_replace
        self.norm_topk_prob = norm_topk_prob
        self.scoring_func = scoring_func
        self.aux_loss_alpha = aux_loss_alpha
        self.seq_aux = seq_aux
        # for backward compatibility
        if num_key_value_heads is None:
            num_key_value_heads = num_attention_heads

        self.num_key_value_heads = num_key_value_heads
        self.hidden_act = hidden_act
        self.initializer_range = initializer_range
        self.rms_norm_eps = rms_norm_eps
        self.pretraining_tp = pretraining_tp
        self.use_cache = use_cache
        self.rope_theta = rope_theta
        self.rope_scaling = rope_scaling
        self.attention_bias = attention_bias
        self.attention_dropout = attention_dropout

        super().__init__(
            pad_token_id=pad_token_id,
            bos_token_id=bos_token_id,
            eos_token_id=eos_token_id,
            tie_word_embeddings=tie_word_embeddings,
            **kwargs,
        )


>>>>> FILE: modeling_kimi_k25.py (50912 bytes) <<<<<
# coding=utf-8
# Copyright 2025-2026 The Moonshot AI Team, DeepSeek-AI, and HuggingFace Inc. team. All rights reserved.
#
# The code is based on llava (llava/modeling_llava.py) and DeepSeek-V3 (DeepSeek-V3/modeling_deepseek.py), but modified for Kimi-K2.5.
#
# Licensing Information:
# - Code derived from llava (llava/modeling_llava.py) and DeepSeek-V3 (DeepSeek-V3/modeling_deepseek.py) is licensed under the Apache License, Version 2.0.
# - Other parts of the code are licensed under the MIT License.
#
# Apache License, Version 2.0:
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# MIT License:
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
import math
from collections.abc import Sequence
from copy import deepcopy
from typing import Optional

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import activations

try:
    from transformers.activations import PytorchGELUTanh
except ImportError:
    from transformers.activations import GELUTanh
    activations.PytorchGELUTanh = GELUTanh
    PytorchGELUTanh = GELUTanh
from transformers.activations import PytorchGELUTanh
from transformers.cache_utils import Cache
from transformers.configuration_utils import PretrainedConfig
from transformers.modeling_utils import PreTrainedModel
from transformers.models.llava.modeling_llava import \
    LlavaCausalLMOutputWithPast
from transformers.utils import is_flash_attn_2_available

from .configuration_kimi_k25 import KimiK25Config
from .modeling_deepseek import DeepseekV3ForCausalLM

# Flash attention imports
if is_flash_attn_2_available():
    from flash_attn import flash_attn_varlen_func
else:
    flash_attn_varlen_func = None


def multihead_attention(
    q: torch.Tensor,
    k: torch.Tensor,
    v: torch.Tensor,
    q_cu_seqlens: torch.Tensor | None = None,
    k_cu_seqlens: torch.Tensor | None = None,
    max_seqlen_q: int | None = None,
    max_seqlen_k: int | None = None,
    deterministic: bool = False,
):
    """Multi-head attention using flash attention 2.

    Args:
        q, k, v: tensor of shape (batch_size, seqlen, num_heads, head_dim),
            or (tot_seqlens, num_heads, head_dim) if packing.
        q_cu_seqlens (torch.Tensor): cumulative sequence lengths of q.
            The first element should be 0 and the last element should be q.shape[0].
        k_cu_seqlens (torch.Tensor): cumulative sequence lengths of k.
            The first element should be 0 and the last element should be k.shape[0].

    Returns:
        output: shape (batch_size, seqlen, dim) or (tot_seqlens, dim) if packing,
            where dim = num_heads * head_dim
    """
    attn_out = flash_attn_varlen_func(
        q,
        k,
        v,
        q_cu_seqlens,
        k_cu_seqlens,
        max_seqlen_q,
        max_seqlen_k,
        causal=False,
        deterministic=deterministic,
    )
    if isinstance(attn_out, tuple):
        attn_out = attn_out[0]

    attn_out = attn_out.flatten(start_dim=-2)

    return attn_out


def eager_attention(
    q: torch.Tensor,
    k: torch.Tensor,
    v: torch.Tensor,
    q_cu_seqlens: Optional[torch.Tensor] = None,
    k_cu_seqlens: Optional[torch.Tensor] = None,
    **kwargs,
) -> torch.Tensor:
    seq_length = q.shape[0]
    attention_mask = torch.zeros([1, seq_length, seq_length],
                                 device=q.device,
                                 dtype=torch.bool)
    for i in range(1, len(q_cu_seqlens)):
        attention_mask[
            ...,
            q_cu_seqlens[i - 1]:q_cu_seqlens[i],
            q_cu_seqlens[i - 1]:q_cu_seqlens[i],
        ] = True
    q = q.transpose(0, 1)
    k = k.transpose(0, 1)
    v = v.transpose(0, 1)

    attn_weight = q @ k.transpose(-2, -1) / math.sqrt(q.shape[-1])
    attn_weight += attention_mask
    attn_weight = torch.softmax(attn_weight, dim=-1,
                                dtype=torch.float32).to(q.dtype)

    attn_output = attn_weight @ v
    attn_output = attn_output.transpose(0, 1)
    attn_output = attn_output.reshape(seq_length, -1)
    return attn_output


VL_VISION_ATTENTION_FUNCTIONS = {
    "flash_attention_2": multihead_attention,
    "eager": eager_attention,
}


def _apply_rope_input_validation(x, freqs_cis):
    assert x.ndim == freqs_cis.ndim + 1, (x.shape, freqs_cis.shape)
    assert x.shape[:-2] == freqs_cis.shape[:-1], (x.shape, freqs_cis.shape)
    assert x.shape[-1] == 2 * freqs_cis.shape[-1], (x.shape, freqs_cis.shape)
    assert freqs_cis.dtype == torch.complex64, freqs_cis.dtype


def get_rope_shape_decorate(func):
    _get_rope_shape_first_call_flag = set()

    def wrapper(org, interpolation_mode, shape):
        key = (org.requires_grad, torch.is_grad_enabled(), interpolation_mode)
        if key not in _get_rope_shape_first_call_flag:
            _get_rope_shape_first_call_flag.add(key)
            _ = func(org, interpolation_mode, shape=(64, 64))
        return func(org, interpolation_mode, shape)

    return wrapper


@get_rope_shape_decorate
@torch.compile(dynamic=True)
def get_rope_shape(org, interpolation_mode, shape):
    return (F.interpolate(
        org.permute((2, 0, 1)).unsqueeze(0),
        size=shape,
        mode=interpolation_mode,
    ).squeeze(0).permute((1, 2, 0)).flatten(end_dim=1))


def apply_rope(xq: torch.Tensor, xk: torch.Tensor,
               freqs_cis: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
    """
    Args: (The leading dimensions of all inputs should be the same)
        xq: query, tensor of shape (..., num_heads, head_dim)
        xk: key, tensor of shape (..., num_heads, head_dim)
        freqs_cis: tensor of shape (..., head_dim/2), dtype=torch.complex64. It contains the precomputed cis(freqs) for each position in the 2D grid.
    Returns:
        xq_out, xk_out: tensors of shape (..., num_heads, head_dim)
    """
    _apply_rope_input_validation(xq, freqs_cis)
    _apply_rope_input_validation(xk, freqs_cis)

    freqs_cis = freqs_cis.unsqueeze(-2)  # ..., 1, head_dim/2
    # ..., num_heads, head_dim/2
    xq_ = torch.view_as_complex(xq.float().view(*xq.shape[:-1], -1, 2))
    xk_ = torch.view_as_complex(xk.float().view(*xq.shape[:-1], -1, 2))
    xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(
        -2)  # ..., num_heads, head_dim
    xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(
        -2)  # ..., num_heads, head_dim
    return xq_out.type_as(xq), xk_out.type_as(xk)


def get_1d_sincos_pos_embed_from_grid(embed_dim, pos):
    """
    From:
    https://github.com/OpenGVLab/InternVideo/blob/421f6d2361fc8f61a3394244571f2601a4e99e29/InternVideo2/multi_modality/models/backbones/internvideo2/pos_embed.py#L86
    embed_dim: output dimension for each position
    pos: a list of positions to be encoded: size (M,)
    out: (M, D)
    """
    assert embed_dim % 2 == 0
    omega = np.arange(embed_dim // 2, dtype=np.float32)
    omega /= embed_dim / 2.0
    omega = 1.0 / 10000**omega  # (D/2,)

    pos = pos.reshape(-1)  # (M,)
    out = np.einsum('m,d->md', pos, omega)  # (M, D/2), outer product

    emb_sin = np.sin(out)  # (M, D/2)
    emb_cos = np.cos(out)  # (M, D/2)

    emb = np.concatenate([emb_sin, emb_cos], axis=1)  # (M, D)
    return emb


def get_1d_sincos_pos_embed(embed_dim, t_size, cls_token=False):
    """
    t_size: int of the temporal size
    return:
    pos_embed: [t_size, embed_dim] or [1+t_size, embed_dim] (w/ or w/o cls_token)
    """
    grid_t = np.arange(t_size, dtype=np.float32)
    pos_embed = get_1d_sincos_pos_embed_from_grid(embed_dim, grid_t)
    if cls_token:
        pos_embed = np.concatenate([np.zeros([1, embed_dim]), pos_embed],
                                   axis=0)
    return pos_embed


class Learnable2DInterpPosEmbDivided_fixed(nn.Module):

    def __init__(self,
                 height: int,
                 width: int,
                 num_frames: int,
                 dim: int,
                 interpolation_mode: str = 'bicubic') -> None:
        super().__init__()
        self.height = height
        self.width = width
        self.num_frames = num_frames
        self.dim = dim
        self.interpolation_mode = interpolation_mode
        self.weight = nn.Parameter(torch.empty(height, width, dim))
        self.register_buffer('time_weight',
                             torch.from_numpy(
                                 get_1d_sincos_pos_embed(
                                     self.dim,
                                     self.num_frames)).float().unsqueeze(1),
                             persistent=False)

        self.reset_parameters()

    def reset_parameters(self):
        nn.init.normal_(self.weight)

    def forward(self, x: torch.Tensor,
                grid_thws: torch.Tensor) -> torch.Tensor:
        pos_embs = []
        for t, h, w in grid_thws.tolist():
            assert t <= self.num_frames, f't:{t} > self.num_frames:{self.num_frames}'
            if (h, w) == self.weight.shape[:-1]:
                pos_emb_2d = self.weight.flatten(end_dim=1)
            else:
                pos_emb_2d = get_rope_shape(
                    self.weight,
                    interpolation_mode=self.interpolation_mode,
                    shape=(h, w),
                )

            if t == 1:
                pos_emb_3d = pos_emb_2d
            else:
                pos_emb_3d = pos_emb_2d.unsqueeze(0).repeat(
                    t, 1, 1) + self.time_weight[0:t]

            pos_embs.append(pos_emb_3d.reshape(-1, pos_emb_3d.shape[-1]))

        out = x + torch.cat(pos_embs)
        return out


class MoonVision3dPatchEmbed(nn.Module):

    def __init__(self,
                 out_dim: int,
                 in_dim: int = 3,
                 patch_size: int | tuple[int, int] = (14, 14),
                 pos_emb_height: int = 14,
                 pos_emb_width: int = 14,
                 pos_emb_time: int = 4,
                 pos_emb_type: str = 'divided_fixed'):
        super().__init__()
        assert isinstance(
            patch_size,
            int | Sequence), f'Invalid patch_size type: {type(patch_size)}'
        if isinstance(patch_size, int):
            patch_size = (patch_size, patch_size)
        assert (len(patch_size) == 2
                ), f'Expected patch_size to be a tuple of 2, got {patch_size}'
        self.patch_size = patch_size

        self.proj = nn.Conv2d(in_dim,
                              out_dim,
                              kernel_size=patch_size,
                              stride=patch_size)

        if pos_emb_type == 'divided_fixed':
            self.pos_emb = Learnable2DInterpPosEmbDivided_fixed(
                height=pos_emb_height,
                width=pos_emb_width,
                num_frames=pos_emb_time,
                dim=out_dim)
        else:
            raise NotImplementedError(
                f'Not support pos_emb_type: {pos_emb_type}')

    def forward(self, x: torch.Tensor,
                grid_thws: torch.Tensor) -> torch.Tensor:
        """
        Args:
            x (L, Channels): input tensor
            grid_hws (N, 3): temporal, height and width

        Returns:
            (L, Cout) tensor
        """
        x = self.proj(x).view(x.size(0), -1)
        # apply positional embedding
        x = self.pos_emb(x, grid_thws)
        return x


class Rope2DPosEmbRepeated(nn.Module):
    """2D rotary position embedding with multi-resolution support.

    This class is intended to be used in the following way:
    1. Before training, create an instance of Rope2DPosEmb. This instance will hold the precomputed cis.
    2. Before each forward pass, call `get_freqs_cis_by_*` to get the `freqs_cis` tensor for this iteration.
    3. During the forward pass, pass the `freqs_cis` tensor to each attention layer, and call `apply` just before each attention operation.
        The rope is shared across all attention layers and all heads.

    Refs:
    - RoFormer: https://arxiv.org/abs/2104.09864
    - VisionLLaMA: https://arxiv.org/abs/2403.00522
    - https://github.com/Meituan-AutoML/VisionLLaMA/blob/main/dit/models.py

    Args:
        dim (int): usually the multi-head attention dimension, should be divisible by 4 (TODO: relax this constraint if needed)
        max_height (int): the maximum height of the 2D grid
        max_width (int): the maximum width of the 2D grid
        theta_base (float): the base of the theta
        device (str): the device to store the precomputed cis
    """

    def __init__(self,
                 dim: int,
                 max_height: int,
                 max_width: int,
                 theta_base=10000):
        super().__init__()
        self.dim = dim
        assert self.dim % 4 == 0, 'dim must be divisible by 4'
        self.max_height = max_height
        self.max_width = max_width
        self.theta_base = theta_base

    def extra_repr(self):
        return f'dim={self.dim}, max_height={self.max_height}, max_width={self.max_width}, theta_base={self.theta_base}'

    def _precompute_freqs_cis(self, device: torch.device) -> torch.Tensor:
        """Calculate the cis(freqs) for each position in the 2D grid.

        Return: complex tensor of shape (max_height, max_width, dim//2) and value:
            height axis: ret[h, w, 2*i] = cis(h * theta_base**(-4*i/dim))
            weight axis: ret[h, w, 2*i+1] = cis(w * theta_base**(-4*i/dim))   with (i in [0, dim//4))
            note: `cis` is a mathematical notation defined by cis x = cos x + i sin x,
        """
        N = self.max_height * self.max_width
        flat_pos = torch.arange(0, N).float().to(device)
        x_pos = flat_pos % self.max_width
        y_pos = flat_pos // self.max_width
        dim_range = (torch.arange(0, self.dim,
                                  4)[:(self.dim // 4)].float().to(device)
                     )  # C/4
        freqs = 1.0 / (self.theta_base**(dim_range / self.dim))
        x_freqs = torch.outer(x_pos, freqs).float()  # N, C/4
        y_freqs = torch.outer(y_pos, freqs).float()  # N, C/4
        x_cis = torch.polar(torch.ones_like(x_freqs), x_freqs)  # N, C/4
        y_cis = torch.polar(torch.ones_like(y_freqs), y_freqs)  # N, C/4
        # N, C/4, 2
        freqs_cis = torch.cat(
            [x_cis.unsqueeze(dim=-1),
             y_cis.unsqueeze(dim=-1)], dim=-1)
        # max_height, max_width, C/2
        freqs_cis = freqs_cis.reshape(self.max_height, self.max_width, -1)
        return freqs_cis

    def get_freqs_cis(self, grid_thws: torch.Tensor,
                      device: torch.device) -> torch.Tensor:
        """
        Args:
            grid_thws (torch.Tensor): grid time, height and width

        Returns:
            freqs_cis: tensor of shape (sum(t * height * width), dim//2)
        """
        if not hasattr(self, 'freqs_cis'):
            self.register_buffer('freqs_cis',
                                 self._precompute_freqs_cis(device),
                                 persistent=False)

        shapes = grid_thws.tolist()
        assert all(1 <= h <= self.max_height and 1 <= w <= self.max_width
                   for t, h, w in shapes), (
                       shapes,
                       self.max_height,
                       self.max_width,
                   )
        freqs_cis = torch.cat(
            [
                self.freqs_cis[:h, :w].reshape(-1, self.dim // 2).repeat(t, 1)
                for t, h, w in shapes
            ],
            dim=0,
        )
        return freqs_cis


class MLP2(nn.Module):
    """
    Args:
        dims: [in_dim, hidden_dim, out_dim]

Mathematical records from this source

Scaled dot-product attention logits

For each query, compute its dot product with every key and divide by the square root of the query/key feature width. The transposed key axes put corresponding features into the…

Float32 normalization of vision-attention logits

Softmax across the final key-position axis turns each query’s masked logits into relative nonnegative weights. The source calculates the normalization in float32 and casts its…

Aggregate value vectors with vision-attention weights

For each query, sum the value vectors using its attention weights. The matrix product contracts the key-position axis, leaving the value-feature axis as the output. When weights…

get_rope_shape: return

In get_rope_shape, this statement returns the displayed result. This statement evaluates the named projection, normalization or numeric transform with the source arguments. Its…

apply_rope: xq out

In apply_rope, this statement defines or updates xq_out. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…

apply_rope: xk out

In apply_rope, this statement defines or updates xk_out. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…

get_1d_sincos_pos_embed_from_grid: omega

In get_1d_sincos_pos_embed_from_grid, this statement defines or updates omega. The expression forms a quotient or integer division. The denominator, integer rounding and tensor…

get_1d_sincos_pos_embed_from_grid: omega

In get_1d_sincos_pos_embed_from_grid, this statement defines or updates omega. The expression forms a quotient or integer division. The denominator, integer rounding and tensor…

get_1d_sincos_pos_embed_from_grid: omega

In get_1d_sincos_pos_embed_from_grid, this statement defines or updates omega. The expression forms a quotient or integer division. The denominator, integer rounding and tensor…

get_1d_sincos_pos_embed_from_grid: out

In get_1d_sincos_pos_embed_from_grid, this statement defines or updates out. The matrix or indexed contraction combines the stated axes. Compatible tensor dimensions and the…

get_1d_sincos_pos_embed_from_grid: emb sin

In get_1d_sincos_pos_embed_from_grid, this statement defines or updates emb_sin. This is part of the positional-phase calculation. Frequency, position, sine/cosine and coordinate…

get_1d_sincos_pos_embed_from_grid: emb cos

In get_1d_sincos_pos_embed_from_grid, this statement defines or updates emb_cos. This is part of the positional-phase calculation. Frequency, position, sine/cosine and coordinate…

get_1d_sincos_pos_embed: grid t

In get_1d_sincos_pos_embed, this statement defines or updates grid_t. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…

tpool_patch_merger: pre sum

In tpool_patch_merger, this statement defines or updates pre_sum. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…

Learnable2DInterpPosEmbDivided_fixed.forward: out

In Learnable2DInterpPosEmbDivided_fixed.forward, this statement defines or updates out. The addition or subtraction updates the named state or combines residual contributions. The…

Rope2DPosEmbRepeated._precompute_freqs_cis: N

In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates N. The displayed multiplication combines the source factors. For tensors this is elementwise…

Rope2DPosEmbRepeated._precompute_freqs_cis: flat pos

In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates flat_pos. This statement evaluates the named projection, normalization or numeric transform with…

Rope2DPosEmbRepeated._precompute_freqs_cis: x pos

In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates x_pos. This statement evaluates the named projection, normalization or numeric transform with the…

Rope2DPosEmbRepeated._precompute_freqs_cis: y pos

In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates y_pos. The expression forms a quotient or integer division. The denominator, integer rounding and…

Rope2DPosEmbRepeated._precompute_freqs_cis: dim range

In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates dim_range. The expression forms a quotient or integer division. The denominator, integer rounding…

Rope2DPosEmbRepeated._precompute_freqs_cis: freqs

In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates freqs. The expression forms a quotient or integer division. The denominator, integer rounding and…

Rope2DPosEmbRepeated._precompute_freqs_cis: x freqs

In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates x_freqs. This statement evaluates the named projection, normalization or numeric transform with…

Rope2DPosEmbRepeated._precompute_freqs_cis: y freqs

In Rope2DPosEmbRepeated._precompute_freqs_cis, this statement defines or updates y_freqs. This statement evaluates the named projection, normalization or numeric transform with…

Rope2DPosEmbRepeated.get_freqs_cis: freqs cis

In Rope2DPosEmbRepeated.get_freqs_cis, this statement defines or updates freqs_cis. The expression forms a quotient or integer division. The denominator, integer rounding and…

MoonViTEncoderLayer.init: hidden size per attention head

In MoonViTEncoderLayer.__init__, this statement defines or updates self.hidden_size_per_attention_head. The expression forms a quotient or integer division. The denominator,…

MoonViTEncoderLayer.init: wqkv

In MoonViTEncoderLayer.__init__, this statement defines or updates self.wqkv. The displayed multiplication combines the source factors. For tensors this is elementwise unless the…

MoonViTEncoderLayer.attention_qkvpacked: qkv shape

In MoonViTEncoderLayer.attention_qkvpacked, this statement defines or updates qkv_shape. The addition or subtraction updates the named state or combines residual contributions.…

Residual addition around a transformed hidden state

Add the block input saved in residual to the output of the intervening transformation. Both operands must have the same residual-stream shape. The same expression appears around…

MoonViT3dEncoder.init: rope 2d

In MoonViT3dEncoder.__init__, this statement defines or updates self.rope_2d. The expression forms a quotient or integer division. The denominator, integer rounding and tensor…

MoonViT3dEncoder.forward: lengths

In MoonViT3dEncoder.forward, this statement defines or updates lengths. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…

MoonViT3dEncoder.forward: max seqlen

In MoonViT3dEncoder.forward, this statement defines or updates max_seqlen. This statement evaluates the named projection, normalization or numeric transform with the source…

MoonViT3dEncoder.forward: cu seqlens

In MoonViT3dEncoder.forward, this statement defines or updates cu_seqlens. This statement evaluates the named projection, normalization or numeric transform with the source…

tpool_patch_merger: seq

In tpool_patch_merger, this statement defines or updates seq. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…

tpool_patch_merger: (new height, new width)

In tpool_patch_merger, this statement defines or updates (new_height, new_width). The expression forms a quotient or integer division. The denominator, integer rounding and tensor…

tpool_patch_merger: reshaped seq

In tpool_patch_merger, this statement defines or updates reshaped_seq. The reduction aggregates values along the explicit source axis; preserving the axis and keep-dimension…

tpool_patch_merger: padded seq

In tpool_patch_merger, this statement defines or updates padded_seq. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…

tpool_patch_merger: pre sum

In tpool_patch_merger, this statement defines or updates pre_sum. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…

MLP.forward: x

In MLP.forward, this statement defines or updates x. This statement evaluates the named projection, normalization or numeric transform with the source arguments. Its learned…

Width of a concatenated Kimi patch group

The feature width of one patch is multiplied by the number of patches in a merge group. For 1,152 features and a 2×2 group, the intermediate width is 4,608. The later constructor…

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: left padding

In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates left_padding. The reduction aggregates values along the explicit source…

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: max embed dim

In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates max_embed_dim. The reduction aggregates values along the explicit source…

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: new token positions

In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates new_token_positions. The addition or subtraction updates the named state…

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: nb image pad

In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates nb_image_pad. The addition or subtraction updates the named state or…

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: image to overwrite

In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates image_to_overwrite. The addition or subtraction updates the named state…

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: position ids

In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates position_ids. The addition or subtraction updates the named state or…

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: final embedding[batch indices, indices to mask]

In KimiK25ForConditionalGeneration._merge_input_ids_with_image_features, this statement defines or updates final_embedding[batch_indices, indices_to_mask]. This statement fixes a…

Apply the patch-merger projection to each variable-sized group

For each input item, normalize its per-patch feature vectors, flatten each patch group while retaining its leading group count, and apply the merger projection. The list form…

Normalize, concatenate, and project Kimi patch features

Per-patch LayerNorm is applied before adjacent patch features are concatenated by the reshape. The projection network then maps each merged group into the language representation.…

KimiK25ForConditionalGeneration.prepare_inputs_for_generation: position ids

In KimiK25ForConditionalGeneration.prepare_inputs_for_generation, this statement defines or updates position_ids. The addition or subtraction updates the named state or combines…

Learnable2DInterpPosEmbDivided_fixed.forward: pos emb 3d

In Learnable2DInterpPosEmbDivided_fixed.forward, this statement defines or updates pos_emb_3d. The addition or subtraction updates the named state or combines residual…

KimiK25ForConditionalGeneration.prepare_inputs_for_generation: input ids

In KimiK25ForConditionalGeneration.prepare_inputs_for_generation, this statement defines or updates input_ids. The addition or subtraction updates the named state or combines…

KimiK25ForConditionalGeneration.prepare_inputs_for_generation: attention mask

In KimiK25ForConditionalGeneration.prepare_inputs_for_generation, this statement defines or updates attention_mask. The addition or subtraction updates the named state or combines…

KimiK25ForConditionalGeneration.forward: (batch index, non attended tokens)

In KimiK25ForConditionalGeneration.forward, this statement defines or updates (batch_index, non_attended_tokens). The reduction aggregates values along the explicit source axis;…

KimiK25ForConditionalGeneration.forward: extended attention mask[new batch index, new non attended tokens]

In KimiK25ForConditionalGeneration.forward, this statement defines or updates extended_attention_mask[new_batch_index, new_non_attended_tokens]. This statement fixes a numeric…

KimiK25ForConditionalGeneration.forward: position ids

In KimiK25ForConditionalGeneration.forward, this statement defines or updates position_ids. The reduction aggregates values along the explicit source axis; preserving the axis and…

KimiK25ForConditionalGeneration.prepare_inputs_for_generation: input ids

In KimiK25ForConditionalGeneration.prepare_inputs_for_generation, this statement defines or updates input_ids. The addition or subtraction updates the named state or combines…

_get_unpad_data: seqlens in batch

In _get_unpad_data, this statement defines or updates seqlens_in_batch. The reduction aggregates values along the explicit source axis; preserving the axis and keep-dimension…

_get_unpad_data: max seqlen in batch

In _get_unpad_data, this statement defines or updates max_seqlen_in_batch. This statement evaluates the named projection, normalization or numeric transform with the source…

_get_unpad_data: cu seqlens

In _get_unpad_data, this statement defines or updates cu_seqlens. This statement evaluates the named projection, normalization or numeric transform with the source arguments. Its…

Solve for the rotary frequency index at a chosen rotation count

At index i, the unscaled angular frequency is base^(−2 i/dim). Over orig_max positions it completes orig_max*base^(−2 i/dim)/(2π) rotations. Setting that count equal to rot and…

yarn_find_correction_range: low

In yarn_find_correction_range, this statement defines or updates low. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…

yarn_find_correction_range: high

In yarn_find_correction_range, this statement defines or updates high. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…

yarn_find_correction_range: return

In yarn_find_correction_range, this statement returns the displayed result. The addition or subtraction updates the named state or combines residual contributions. The input…

Logarithmic magnitude adjustment for extended context

For scale>1, the adjustment adds 0.1*mscale*ln(scale) to one. The code returns one for scale≤1. One parameter specifies the context extension, while the other controls adjustment…

yarn_linear_ramp_mask: linear func

In yarn_linear_ramp_mask, this statement defines or updates linear_func. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape…

yarn_linear_ramp_mask: ramp func

In yarn_linear_ramp_mask, this statement defines or updates ramp_func. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…

rotate_half: x1

In rotate_half, this statement defines or updates x1. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention determine…

rotate_half: x2

In rotate_half, this statement defines or updates x2. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention determine…

apply_rotary_pos_emb: q

In apply_rotary_pos_emb, this statement defines or updates q. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…

apply_rotary_pos_emb: k

In apply_rotary_pos_emb, this statement defines or updates k. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…

Rotate paired query coordinates using a position-dependent angle

rotate_half maps a pair of coordinate blocks(x 1,x 2) to(−x 2,x 1). Multiplication by cosine and addition of the sine-weighted rotated vector implements a two-dimensional rotation…

Rotate paired key coordinates using a position-dependent angle

rotate_half maps a pair of coordinate blocks(x 1,x 2) to(−x 2,x 1). Multiplication by cosine and addition of the sine-weighted rotated vector implements a two-dimensional rotation…

convert_kimi_to_exact: n

In convert_kimi_to_exact, this statement defines or updates n. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…

repeat_kv: return

In repeat_kv, this statement returns the displayed result. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…

get_usable_length: return

In get_usable_length, this statement returns the displayed result. The addition or subtraction updates the named state or combines residual contributions. The input shapes and the…

RMSNorm mean square, despite the local variable name

This expression averages squared components along the final feature axis and preserves that axis for broadcasting. It computes the second raw moment, not centered variance: the…

Normalize by the root mean square

Multiply each feature by 1/√(mean(x²)+epsilon). Positive epsilon prevents division by zero and changes exact unit-RMS normalization near the zero vector. Unlike LayerNorm, this…

Apply learned per-feature RMSNorm gains

After RMS scaling, multiply each component by its learned gain and restore the earlier input dtype. The gain allows the model to choose different output scales by feature; it does…

DeepSeek RotaryEmbedding.init: inv freq

In DeepseekV3RotaryEmbedding.__init__, this statement defines or updates inv_freq. This is part of the positional-phase calculation. Frequency, position, sine/cosine and…

DeepSeek RotaryEmbedding._set_cos_sin_cache: t

In DeepseekV3RotaryEmbedding._set_cos_sin_cache, this statement defines or updates t. This is part of the positional-phase calculation. Frequency, position, sine/cosine and…

DeepSeek RotaryEmbedding._set_cos_sin_cache: freqs

In DeepseekV3RotaryEmbedding._set_cos_sin_cache, this statement defines or updates freqs. This is part of the positional-phase calculation. Frequency, position, sine/cosine and…

DeepSeek LinearScalingRotaryEmbedding._set_cos_sin_cache: t

In DeepseekV3LinearScalingRotaryEmbedding._set_cos_sin_cache, this statement defines or updates t. The expression forms a quotient or integer division. The denominator, integer…

DeepSeek LinearScalingRotaryEmbedding._set_cos_sin_cache: freqs

In DeepseekV3LinearScalingRotaryEmbedding._set_cos_sin_cache, this statement defines or updates freqs. This is part of the positional-phase calculation. Frequency, position,…

yarn_get_mscale: return

In yarn_get_mscale, this statement returns the displayed result. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…

yarn_linear_ramp_mask: max

In yarn_linear_ramp_mask, this statement defines or updates max. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…

DeepSeek YarnRotaryEmbedding._set_cos_sin_cache: freq extra

In DeepseekV3YarnRotaryEmbedding._set_cos_sin_cache, this statement defines or updates freq_extra. The expression forms a quotient or integer division. The denominator, integer…

DeepSeek YarnRotaryEmbedding._set_cos_sin_cache: freq inter

In DeepseekV3YarnRotaryEmbedding._set_cos_sin_cache, this statement defines or updates freq_inter. The expression forms a quotient or integer division. The denominator, integer…

Complement the increasing YaRN frequency ramp

The helper produces a ramp increasing from zero to one across frequency-pair indices. Subtracting it from one makes inv_freq_mask select original frequencies at the small-index…

Blend original and divided YaRN frequency schedules

The source uses inv_freq_mask as the weight on the original schedule and its complement as the weight on the divided schedule. Since inv_freq_mask=1−ramp, this is…

DeepSeek YarnRotaryEmbedding._set_cos_sin_cache: t

In DeepseekV3YarnRotaryEmbedding._set_cos_sin_cache, this statement defines or updates t. This statement evaluates the named projection, normalization or numeric transform with…

DeepSeek YarnRotaryEmbedding._set_cos_sin_cache: freqs

In DeepseekV3YarnRotaryEmbedding._set_cos_sin_cache, this statement defines or updates freqs. This is part of the positional-phase calculation. Frequency, position, sine/cosine…

Ratio controlling the rotary cache magnitude

This coefficient divides the adjustment computed with mscale by the adjustment computed with mscale_all_dim. Equal strengths give a ratio of one. The attention initializer can…

SwiGLU-style gated feed-forward projection

Two learned maps project x into the intermediate width. The configured activation, SiLU in these dossiers, is applied to the gate projection; its result multiplies the…

MoEGate.forward: logits

In MoEGate.forward, this statement defines or updates logits. The matrix or indexed contraction combines the stated axes. Compatible tensor dimensions and the transposes in this…

MoEGate.forward: topk weight

In MoEGate.forward, this statement defines or updates topk_weight. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive…

DeepSeek MoE.moe_infer: tokens per expert

In DeepseekV3MoE.moe_infer, this statement defines or updates tokens_per_expert. The reduction aggregates values along the explicit source axis; preserving the axis and…

DeepSeek MoE.moe_infer: sorted tokens

In DeepseekV3MoE.moe_infer, this statement defines or updates sorted_tokens. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries…

DeepSeek MoE.moe_infer: start idx

In DeepseekV3MoE.moe_infer, this statement defines or updates start_idx. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is…

Sum the selected expert outputs with their routing weights

Each chosen expert evaluates the same input representation h. Its output vector is multiplied by the associated routing weight, and the selected vectors are added. The…

100

DeepSeek Attention.init: q head dim

In DeepseekV3Attention.__init__, this statement defines or updates self.q_head_dim. The addition or subtraction updates the named state or combines residual contributions. The…

101

DeepSeek Attention.init: kv a proj with mqa

In DeepseekV3Attention.__init__, this statement defines or updates self.kv_a_proj_with_mqa. The addition or subtraction updates the named state or combines residual contributions.…

102

DeepSeek Attention.init: kv b proj

In DeepseekV3Attention.__init__, this statement defines or updates self.kv_b_proj. The displayed multiplication combines the source factors. For tensors this is elementwise unless…

103

DeepSeek Attention.init: o proj

In DeepseekV3Attention.__init__, this statement defines or updates self.o_proj. The displayed multiplication combines the source factors. For tensors this is elementwise unless…

104

Baseline inverse-square-root attention coefficient

Initialize the score multiplier to 1/√d, where d is the total rotary plus non-rotary query-head width. This is the starting coefficient; the following YaRN branch can multiply it…

105

Expand the compressed latent into per-head keys and values

Normalize the compressed key/value latent, apply the learned up-projection, and reshape into heads. Each head receives its non-rotary key and value components. This eagerly…

106

DeepSeek Attention.forward: attn weights

In DeepseekV3Attention.forward, this statement defines or updates attn_weights. Softmax exponentiates and normalizes scores along the chosen axis, producing nonnegative weights…

107

Normalize each query’s attention scores

Apply softmax across key positions for each query and head. The resulting nonnegative weights sum to one when a row has at least one finite eligible score. The source computes…

108

DeepSeek Attention.forward: attn output

In DeepseekV3Attention.forward, this statement defines or updates attn_output. The matrix or indexed contraction combines the stated axes. Compatible tensor dimensions and the…

109

DeepSeek Attention.forward: attn output

In DeepseekV3Attention.forward, this statement defines or updates attn_output. The displayed multiplication combines the source factors. For tensors this is elementwise unless the…

110

DeepSeek Attention.forward: attn output

In DeepseekV3Attention.forward, this statement defines or updates attn_output. This statement evaluates the named projection, normalization or numeric transform with the source…

111

DeepSeek FlashAttention2.forward: attn output

In DeepseekV3FlashAttention2.forward, this statement defines or updates attn_output. The displayed multiplication combines the source factors. For tensors this is elementwise…

112

DeepSeek FlashAttention2._upad_input: key layer

In DeepseekV3FlashAttention2._upad_input, this statement defines or updates key_layer. The displayed multiplication combines the source factors. For tensors this is elementwise…

113

DeepSeek FlashAttention2._upad_input: value layer

In DeepseekV3FlashAttention2._upad_input, this statement defines or updates value_layer. The displayed multiplication combines the source factors. For tensors this is elementwise…

114

DeepSeek DecoderLayer.init: mlp

In DeepseekV3DecoderLayer.__init__, this statement defines or updates self.mlp. This statement evaluates the named projection, normalization or numeric transform with the source…

115

DeepSeek Model.forward: past key values length

In DeepseekV3Model.forward, this statement defines or updates past_key_values_length. This statement fixes a numeric parameter or initializes a numerical state in this source…

116

DeepSeek Model.forward: hidden states

In DeepseekV3Model.forward, this statement defines or updates hidden_states. This statement evaluates the named projection, normalization or numeric transform with the source…

117

DeepSeek ForSequenceClassification.forward: pooled logits

In DeepseekV3ForSequenceClassification.forward, this statement defines or updates pooled_logits. This statement evaluates the named projection, normalization or numeric transform…

118

DeepSeek DynamicNTKScalingRotaryEmbedding._set_cos_sin_cache: base

In DeepseekV3DynamicNTKScalingRotaryEmbedding._set_cos_sin_cache, this statement defines or updates base. The expression forms a quotient or integer division. The denominator,…

119

DeepSeek DynamicNTKScalingRotaryEmbedding._set_cos_sin_cache: inv freq

In DeepseekV3DynamicNTKScalingRotaryEmbedding._set_cos_sin_cache, this statement defines or updates inv_freq. This is part of the positional-phase calculation. Frequency,…

120

convert_kimi_to_exact: n

In convert_kimi_to_exact, this statement defines or updates n. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…

121

Independent sigmoid scores for the router

The sigmoid 1/(1+exp(−z)) is applied to each expert logit independently. It compresses each finite real logit into (0,1), without normalizing across experts. These values are…

122

Bias the choice of experts while retaining original affinities

An expert-specific correction vector is added to the sigmoid affinities for ranking. This addition can change which groups and experts are selected. The later mixture weights are…

123

MoEGate.forward: group scores

In MoEGate.forward, this statement defines or updates group_scores. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive…

124

MoEGate.forward: group idx

In MoEGate.forward, this statement defines or updates group_idx. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive the…

125

MoEGate.forward: score mask

In MoEGate.forward, this statement defines or updates score_mask. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape…

126

MoEGate.forward: ( , topk idx)

In MoEGate.forward, this statement defines or updates (_, topk_idx). The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive…

127

MoEGate.forward: denominator

In MoEGate.forward, this statement defines or updates denominator. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive…

128

Normalize the selected routing affinities

Each gathered affinity is divided by the sum across the selected experts plus 10⁻²⁰. Ignoring that stabilizer, the selected weights sum to one. With the stabilizer retained, the…

129

DeepSeek MoE.init: experts per rank

In DeepseekV3MoE.__init__, this statement defines or updates self.experts_per_rank. The expression forms a quotient or integer division. The denominator, integer rounding and…

130

DeepSeek MoE.init: experts

In DeepseekV3MoE.__init__, this statement defines or updates self.experts. The displayed multiplication combines the source factors. For tensors this is elementwise unless the…

131

DeepSeek MoE.init: ep size

In DeepseekV3MoE.__init__, this statement defines or updates self.ep_size. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is…

132

DeepSeek MoE.init: ep rank

In DeepseekV3MoE.__init__, this statement defines or updates self.ep_rank. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is…

133

DeepSeek MoE.init: intermediate size

In DeepseekV3MoE.__init__, this statement defines or updates intermediate_size. The displayed multiplication combines the source factors. For tensors this is elementwise unless…

134

Add an always-active shared expert

The shared expert processes identity, the input representation saved before the routed expert computation. Its output is added to the weighted routed sum. It therefore contributes…

135

DeepSeek MoE.moe_infer: tokens per ep rank

In DeepseekV3MoE.moe_infer, this statement defines or updates tokens_per_ep_rank. The reduction aggregates values along the explicit source axis; preserving the axis and…

136

DeepSeek MoE.moe_infer: output splits

In DeepseekV3MoE.moe_infer, this statement defines or updates output_splits. The reduction aggregates values along the explicit source axis; preserving the axis and keep-dimension…

137

DeepSeek MoE.moe_infer: gathered tokens

In DeepseekV3MoE.moe_infer, this statement defines or updates gathered_tokens. The reduction aggregates values along the explicit source axis; preserving the axis and…

138

DeepSeek MoE.moe_infer: tokens per expert post gather

In DeepseekV3MoE.moe_infer, this statement defines or updates tokens_per_expert_post_gather. The reduction aggregates values along the explicit source axis; preserving the axis…

139

DeepSeek MoE.moe_infer: s

In DeepseekV3MoE.moe_infer, this statement defines or updates s. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…

140

DeepSeek MoE.moe_infer: end idx

In DeepseekV3MoE.moe_infer, this statement defines or updates end_idx. The addition or subtraction updates the named state or combines residual contributions. The input shapes and…

141

DeepSeek MoE.moe_infer: expert

In DeepseekV3MoE.moe_infer, this statement defines or updates expert. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…

142

DeepSeek Attention.init: q proj

In DeepseekV3Attention.__init__, this statement defines or updates self.q_proj. The displayed multiplication combines the source factors. For tensors this is elementwise unless…

143

DeepSeek Attention.init: q b proj

In DeepseekV3Attention.__init__, this statement defines or updates self.q_b_proj. The displayed multiplication combines the source factors. For tensors this is elementwise unless…

144

DeepSeek Attention.forward: q

In DeepseekV3Attention.forward, this statement defines or updates q. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…

145

DeepSeek Attention.forward: q

In DeepseekV3Attention.forward, this statement defines or updates q. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…

146

DeepSeek Attention.forward: attn weights

In DeepseekV3Attention.forward, this statement defines or updates attn_weights. The addition or subtraction updates the named state or combines residual contributions. The input…

147

Pad value vectors to the query-head width

The FlashAttention path appends zero-valued components to V when the value width is smaller than the query width. With the displayed 192 and 128 widths, it appends 64 zeros. The…

148

DeepSeek FlashAttention2._upad_input: query layer

In DeepseekV3FlashAttention2._upad_input, this statement defines or updates query_layer. The displayed multiplication combines the source factors. For tensors this is elementwise…

149

DeepSeek Model.forward: position ids

In DeepseekV3Model.forward, this statement defines or updates position_ids. The addition or subtraction updates the named state or combines residual contributions. The input…

150

DeepSeek ForSequenceClassification.forward: output

In DeepseekV3ForSequenceClassification.forward, this statement defines or updates output. The addition or subtraction updates the named state or combines residual contributions.…

151

DeepSeek ForSequenceClassification.forward: return

In DeepseekV3ForSequenceClassification.forward, this statement returns the displayed result. The addition or subtraction updates the named state or combines residual…

152

DeepSeek MoE.moe_infer: gatherd idxs[s:s + k]

In DeepseekV3MoE.moe_infer, this statement defines or updates gatherd_idxs[s:s + k]. This statement evaluates the named projection, normalization or numeric transform with the…

153

Apply the additional YaRN score adjustment

Multiply the existing inverse-square-root attention coefficient by mscale². Here mscale is an evaluated logarithmic adjustment, not an independent second query dimension. The…

154

DeepSeek FlashAttention2._upad_input: max seqlen in batch q

In DeepseekV3FlashAttention2._upad_input, this statement defines or updates max_seqlen_in_batch_q. This statement fixes a numeric parameter or initializes a numerical state in…

155

DeepSeek FlashAttention2._upad_input: cu seqlens q

In DeepseekV3FlashAttention2._upad_input, this statement defines or updates cu_seqlens_q. The addition or subtraction updates the named state or combines residual contributions.…

156

DeepSeek ForSequenceClassification.forward: sequence lengths

In DeepseekV3ForSequenceClassification.forward, this statement defines or updates sequence_lengths. The addition or subtraction updates the named state or combines residual…

157

TikTokenTokenizer: num reserved special tokens

In TikTokenTokenizer, this statement defines or updates num_reserved_special_tokens. This statement fixes a numeric parameter or initializes a numerical state in this source…

158

TikTokenTokenizer.init: special tokens

In TikTokenTokenizer.__init__, this statement defines or updates self.special_tokens. The addition or subtraction updates the named state or combines residual contributions. The…

159

TikTokenTokenizer.init: cache max size

In TikTokenTokenizer.__init__, this statement defines or updates self._cache_max_size. This statement fixes a numeric parameter or initializes a numerical state in this source…

160

TikTokenTokenizer.encode: TIKTOKEN MAX ENCODE CHARS

In TikTokenTokenizer.encode, this statement defines or updates TIKTOKEN_MAX_ENCODE_CHARS. This statement fixes a numeric parameter or initializes a numerical state in this source…

161

TikTokenTokenizer.encode: MAX NO WHITESPACES CHARS

In TikTokenTokenizer.encode, this statement defines or updates MAX_NO_WHITESPACES_CHARS. This statement fixes a numeric parameter or initializes a numerical state in this source…

162

TikTokenTokenizer._split_whitespaces_or_nonwhitespaces: current slice len

In TikTokenTokenizer._split_whitespaces_or_nonwhitespaces, this statement defines or updates current_slice_len. This statement fixes a numeric parameter or initializes a numerical…

163

TikTokenTokenizer._split_whitespaces_or_nonwhitespaces: slice start

In TikTokenTokenizer._split_whitespaces_or_nonwhitespaces, this statement defines or updates slice_start. This statement fixes a numeric parameter or initializes a numerical state…

164

TikTokenTokenizer.save_vocabulary: out vocab file

In TikTokenTokenizer.save_vocabulary, this statement defines or updates out_vocab_file. The addition or subtraction updates the named state or combines residual contributions. The…

165

TikTokenTokenizer.encode: substrs

In TikTokenTokenizer.encode, this statement defines or updates substrs. The addition or subtraction updates the named state or combines residual contributions. The input shapes…

166

TikTokenTokenizer._split_whitespaces_or_nonwhitespaces: current slice len

In TikTokenTokenizer._split_whitespaces_or_nonwhitespaces, this statement defines or updates current_slice_len. This statement fixes a numeric parameter or initializes a numerical…

167

TikTokenTokenizer._split_whitespaces_or_nonwhitespaces: current slice len

In TikTokenTokenizer._split_whitespaces_or_nonwhitespaces, this statement defines or updates current_slice_len. This statement fixes a numeric parameter or initializes a numerical…

168

KimiK25Processor.update_raw_text: text

In KimiK25Processor.update_raw_text, this statement defines or updates text. The addition or subtraction updates the named state or combines residual contributions. The input…

169

KimiK25VisionProcessor.split_video_chunks: sample fps

In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates sample_fps. This statement evaluates the named projection, normalization or numeric transform with…

170

KimiK25VisionProcessor.split_video_chunks: sampled nframes

In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates sampled_nframes. The expression forms a quotient or integer division. The denominator, integer…

171

KimiK25VisionProcessor.split_video_chunks: frame inds

In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates frame_inds. The addition or subtraction updates the named state or combines residual contributions.…

172

KimiK25VisionProcessor.split_video_chunks: num chunks

In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates num_chunks. This statement fixes a numeric parameter or initializes a numerical state in this…

173

KimiK25VisionProcessor.to_json_string: return

In KimiK25VisionProcessor.to_json_string, this statement returns the displayed result. The addition or subtraction updates the named state or combines residual contributions. The…

174

KimiK25VisionProcessor.split_video_chunks: start time

In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates start_time. The expression forms a quotient or integer division. The denominator, integer rounding…

175

KimiK25VisionProcessor.split_video_chunks: num chunks

In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates num_chunks. This statement fixes a numeric parameter or initializes a numerical state in this…

176

KimiK25VisionProcessor.split_video_chunks: chunk

In KimiK25VisionProcessor.split_video_chunks, this statement defines or updates chunk. The displayed multiplication combines the source factors. For tensors this is elementwise…

177

KimiK25VisionProcessor.preprocess: image std inv

In KimiK25VisionProcessor.preprocess, this statement defines or updates image_std_inv. The expression forms a quotient or integer division. The denominator, integer rounding and…

178

KimiK25VisionProcessor.get_resize_config: fps

In KimiK25VisionProcessor.get_resize_config, this statement defines or updates fps. This statement fixes a numeric parameter or initializes a numerical state in this source…

179

media_utils.py: MediaInput

In media_utils.py, this statement defines or updates MediaInput. This statement evaluates the named projection, normalization or numeric transform with the source arguments. Its…

180

navit_resize_image: s1

In navit_resize_image, this statement defines or updates s1. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…

181

navit_resize_image: s2

In navit_resize_image, this statement defines or updates s2. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…

182

navit_resize_image: s3

In navit_resize_image, this statement defines or updates s3. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…

183

navit_resize_image: scale

In navit_resize_image, this statement defines or updates scale. This statement evaluates the named projection, normalization or numeric transform with the source arguments. Its…

184

navit_resize_image: (new w, new h)

In navit_resize_image, this statement defines or updates (new_w, new_h). The displayed multiplication combines the source factors. For tensors this is elementwise unless the…

185

navit_resize_image: new w

In navit_resize_image, this statement defines or updates new_w. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…

186

navit_resize_image: new h

In navit_resize_image, this statement defines or updates new_h. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…

187

navit_resize_image: factor

In navit_resize_image, this statement defines or updates factor. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…

188

navit_resize_image: pad height

In navit_resize_image, this statement defines or updates pad_height. The addition or subtraction updates the named state or combines residual contributions. The input shapes and…

189

navit_resize_image: pad width

In navit_resize_image, this statement defines or updates pad_width. The addition or subtraction updates the named state or combines residual contributions. The input shapes and…

190

navit_resize_video: sample fps

In navit_resize_video, this statement defines or updates sample_fps. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…

191

navit_resize_video: sampled nframes

In navit_resize_video, this statement defines or updates sampled_nframes. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape…

192

navit_patchify: patches

In navit_patchify, this statement defines or updates patches. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…

193

navit_patchify: grid thw

In navit_patchify, this statement defines or updates grid_thw. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…

194

normalize: x

In normalize, this statement defines or updates x. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention determine its…

195

navit_resize_image: token height

In navit_resize_image, this statement defines or updates token_height. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape…

196

navit_resize_image: token width

In navit_resize_image, this statement defines or updates token_width. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape…

197

navit_resize_image: num tokens

In navit_resize_image, this statement defines or updates num_tokens. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…

198

navit_resize_video: sampled nframes

In navit_resize_video, this statement defines or updates sampled_nframes. This statement evaluates the named projection, normalization or numeric transform with the source…

199

navit_resize_video: in patch limit each frame

In navit_resize_video, this statement defines or updates in_patch_limit_each_frame. The expression forms a quotient or integer division. The denominator, integer rounding and…

200

image_to_np: scale

In image_to_np, this statement defines or updates scale. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…

201

image_to_np: new width

In image_to_np, this statement defines or updates new_width. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…

202

image_to_np: new height

In image_to_np, this statement defines or updates new_height. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…

203

image_to_np: padding left

In image_to_np, this statement defines or updates padding_left. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…

204

image_to_np: padding right

In image_to_np, this statement defines or updates padding_right. The addition or subtraction updates the named state or combines residual contributions. The input shapes and the…

205

image_to_np: padding top

In image_to_np, this statement defines or updates padding_top. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…

206

image_to_np: padding bottom

In image_to_np, this statement defines or updates padding_bottom. The addition or subtraction updates the named state or combines residual contributions. The input shapes and the…

207

image_to_np: padding right

In image_to_np, this statement defines or updates padding_right. The addition or subtraction updates the named state or combines residual contributions. The input shapes and the…

208

image_to_np: padding bottom

In image_to_np, this statement defines or updates padding_bottom. The addition or subtraction updates the named state or combines residual contributions. The input shapes and the…

209

tool_declaration_ts.py: ParameterType

In tool_declaration_ts.py, this statement defines or updates _ParameterType. This statement evaluates the named projection, normalization or numeric transform with the source…

Source identity

Recovered source transcript

Mathematical records from this source

Scaled dot-product attention logits

Float32 normalization of vision-attention logits

Aggregate value vectors with vision-attention weights

get_rope_shape: return

apply_rope: xq out

apply_rope: xk out

get_1d_sincos_pos_embed_from_grid: omega

get_1d_sincos_pos_embed_from_grid: omega

get_1d_sincos_pos_embed_from_grid: omega

get_1d_sincos_pos_embed_from_grid: out

get_1d_sincos_pos_embed_from_grid: emb sin

get_1d_sincos_pos_embed_from_grid: emb cos

get_1d_sincos_pos_embed: grid t

tpool_patch_merger: pre sum

Learnable2DInterpPosEmbDivided_fixed.forward: out

Rope2DPosEmbRepeated._precompute_freqs_cis: N

Rope2DPosEmbRepeated._precompute_freqs_cis: flat pos

Rope2DPosEmbRepeated._precompute_freqs_cis: x pos

Rope2DPosEmbRepeated._precompute_freqs_cis: y pos

Rope2DPosEmbRepeated._precompute_freqs_cis: dim range

Rope2DPosEmbRepeated._precompute_freqs_cis: freqs

Rope2DPosEmbRepeated._precompute_freqs_cis: x freqs

Rope2DPosEmbRepeated._precompute_freqs_cis: y freqs

Rope2DPosEmbRepeated.get_freqs_cis: freqs cis

MoonViTEncoderLayer.__init__: hidden size per attention head

MoonViTEncoderLayer.__init__: wqkv

MoonViTEncoderLayer.attention_qkvpacked: qkv shape

Residual addition around a transformed hidden state

MoonViT3dEncoder.__init__: rope 2d

MoonViT3dEncoder.forward: lengths

MoonViT3dEncoder.forward: max seqlen

MoonViT3dEncoder.forward: cu seqlens

tpool_patch_merger: seq

tpool_patch_merger: (new height, new width)

tpool_patch_merger: reshaped seq

tpool_patch_merger: padded seq

tpool_patch_merger: pre sum

MLP.forward: x

Width of a concatenated Kimi patch group

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: left padding

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: max embed dim

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: new token positions

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: nb image pad

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: image to overwrite

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: position ids

KimiK25ForConditionalGeneration._merge_input_ids_with_image_features: final embedding[batch indices, indices to mask]

Apply the patch-merger projection to each variable-sized group

Normalize, concatenate, and project Kimi patch features

KimiK25ForConditionalGeneration.prepare_inputs_for_generation: position ids

Learnable2DInterpPosEmbDivided_fixed.forward: pos emb 3d

KimiK25ForConditionalGeneration.prepare_inputs_for_generation: input ids

KimiK25ForConditionalGeneration.prepare_inputs_for_generation: attention mask

KimiK25ForConditionalGeneration.forward: (batch index, non attended tokens)

KimiK25ForConditionalGeneration.forward: extended attention mask[new batch index, new non attended tokens]

KimiK25ForConditionalGeneration.forward: position ids

KimiK25ForConditionalGeneration.prepare_inputs_for_generation: input ids

_get_unpad_data: seqlens in batch

_get_unpad_data: max seqlen in batch

_get_unpad_data: cu seqlens

Solve for the rotary frequency index at a chosen rotation count

yarn_find_correction_range: low

yarn_find_correction_range: high

yarn_find_correction_range: return

Logarithmic magnitude adjustment for extended context

yarn_linear_ramp_mask: linear func

yarn_linear_ramp_mask: ramp func

rotate_half: x1

rotate_half: x2

apply_rotary_pos_emb: q

apply_rotary_pos_emb: k

Rotate paired query coordinates using a position-dependent angle

Rotate paired key coordinates using a position-dependent angle

convert_kimi_to_exact: n

repeat_kv: return

get_usable_length: return

RMSNorm mean square, despite the local variable name

Normalize by the root mean square

MoonViTEncoderLayer.init: hidden size per attention head

MoonViTEncoderLayer.init: wqkv

MoonViT3dEncoder.init: rope 2d

DeepSeek RotaryEmbedding.init: inv freq

DeepSeek Attention.init: q head dim

DeepSeek Attention.init: kv a proj with mqa

DeepSeek Attention.init: kv b proj

DeepSeek Attention.init: o proj

DeepSeek DecoderLayer.init: mlp

DeepSeek MoE.init: experts per rank

DeepSeek MoE.init: experts

DeepSeek MoE.init: ep size

DeepSeek MoE.init: ep rank

DeepSeek MoE.init: intermediate size

DeepSeek Attention.init: q proj

DeepSeek Attention.init: q b proj