DEEPSEEK_R1_COMPLETE.txt

1================================================================================ 2 D E E P S E E K - R 1 3 Complete reference: architecture map, dataflow, file architecture, 4 API surface, endpoints, and full source code 5 as installed on MEGAMIND-AGI, 2026-08-05 6================================================================================ 7 8Model DeepSeek-R1 9Architecture DeepseekV3ForCausalLM (deepseek_v3) 10Parameters 671B total / 37B activated per token [per official model card] 11Location /home/joseph/DeepSeek-R1 on MEGAMIND-AGI (WSL2 Ubuntu) 12Size 643 GB on disk / 641.29 GiB tensor payload / 163 shards 13Downloaded 2026-08-04 05:35-07:21 (huggingface_hub snapshot_download) 14Scanned 2026-08-05, live, by parsing all 163 safetensors file headers 15Paper arXiv:2501.12948 16License MIT (weights and code; commercial use and distillation permitted) 17 18CONTENTS 19 PART 1 MEGAMIND-AGI MACHINE SPECIFICATIONS 20 PART 2 FILE ARCHITECTURE 21 2.1 Directory inventory 22 2.2 The safetensors container format 23 2.3 model.safetensors.index.json structure 24 2.4 Layer -> shard map (all 62 layer indices) 25 2.5 Shard -> layer map (all 163 shards) 26 2.6 On-disk expert layout 27 2.7 Integrity verification 28 PART 3 ARCHITECTURE SPECIFICATION (every config value) 29 PART 4 COMPLETE ARCHITECTURE MAP (module tree, every parameter) 30 4.1 Top-level module tree 31 4.2 Dense layer manifest (layers 0-2) 32 4.3 MoE layer manifest (layers 3-60) 33 4.4 One routed expert 34 4.5 MTP layer manifest (layer 61) 35 4.6 Global tensors 36 4.7 Parameter accounting 37 PART 5 COMPLETE TENSOR CENSUS 38 PART 6 FP8 QUANTIZATION SCHEME 39 PART 7 MoE ROUTING ALGORITHM 40 PART 8 MLA ATTENTION 41 PART 9 YaRN POSITION ENCODING 42 PART 10 COMPLETE DATAFLOW CHART 43 10.1 Prefill vs decode 44 10.2 Full tensor-shape dataflow 45 10.3 KV cache lifecycle 46 10.4 MoE dispatch dataflow 47 10.5 MTP / speculative decode dataflow 48 PART 11 TOKENIZER AND WIRE FORMAT 49 11.1 Tokenizer pipeline 50 11.2 Complete special-token table 51 11.3 Chat template semantics 52 11.4 A rendered example 53 PART 12 API SURFACE AND ENDPOINTS 54 12.1 HuggingFace auto_map and load path 55 12.2 Python class API (complete) 56 12.3 DeepseekV3Config parameters 57 12.4 forward() input/output contract 58 12.5 Official hosted API endpoints 59 12.6 Self-hosted serving endpoints 60 12.7 The reasoning-content contract 61 PART 13 FOOTPRINT: MEMORY, DISK I/O, COMPUTE 62 PART 14 DISCREPANCIES, TRAPS AND GOTCHAS 63 PART 15 SOURCE CODE (verbatim) 64 15.1 config.json 65 15.2 configuration_deepseek.py (199 lines) 66 15.3 modeling_deepseek.py (1,848 lines) 67 15.4 generation_config.json 68 15.5 tokenizer_config.json 69 15.6 README.md (official model card) 70 71NOTATION 72 [MEASURED] read off this machine during the scan 73 [SPEC] read out of the model's own config.json / reference source 74 [COMPUTED] arithmetic from measured values; formula always shown 75 [CARD] stated by the official model card (README.md, reproduced in 15.6) 76 [EXTERNAL] about services outside this machine; verify against live docs 77 78 79================================================================================ 80PART 1 MEGAMIND-AGI MACHINE SPECIFICATIONS [MEASURED] 81================================================================================ 82 83IDENTITY 84 Hostname MEGAMIND-AGI 85 Chassis Dell Precision 3630 Tower 86 Tailnet 100.69.120.61 Windows host 87 100.77.114.40 WSL2 guest (megamind-agi-wsl) 88 Users WSL: joseph Windows: Josep 89 90CPU 91 Model Intel Core i7-8700K @ 3.70 GHz 92 Topology 1 socket / 6 cores / 12 threads (Hyper-Threading enabled) 93 Cache L1d 192 KiB (6 inst) L1i 192 KiB (6 inst) 94 L2 1.5 MiB (6 inst) L3 12 MiB (1 inst, shared) 95 ISA PRESENT avx2 fma f16c popcnt sse4_2 aes bmi1 bmi2 rdseed adx 96 movbe xsave xsavec xsaves clflushopt erms invpcid rtm hle 97 ISA ABSENT avx512f avx512bw avx512vnni avx_vnni amx_int8 amx_bf16 98 -> no hardware fp8 or bf16 conversion instruction 99 -> no VNNI int8 dot; int8 MAC needs vpmaddubsw + vpmaddwd 100 -> 256-bit vectors are the widest available 101 102GPU 103 Intel UHD 630 integrated graphics only. No discrete GPU installed. 104 105MEMORY 106 Visible to WSL2 25 GiB (.wslconfig memory=26GB) 107 Swap 8 GiB (/dev/sdc) 108 At scan time 15 GiB used | 784 MiB free | 14 GiB buff/cache 109 9.6 GiB available | 2.3 GiB of swap already in use 110 111STORAGE 112 /dev/sdd 1007 GiB ext4, WSL2 vhdx on a 1 TB WD Green SN350 NVMe SSD 113 729 GiB used / 228 GiB available / 77% full, mounted at / 114 -- this is where DeepSeek-R1 lives 115 /mnt/c 931 GiB, 800 GiB used, 132 GiB free (Windows side, same NVMe) 116 /dev/sdc 8 GiB swap 117 118 MEASURED THROUGHPUT against the R1 shards: 119 --------------------------------------------------------------------------- 120 access pattern throughput 121 --------------------------------------------------------------------------- 122 single stream, 8 MiB blocks, 2 GiB read 507 MB/s 123 same file immediately again (page cache warm) 408 MB/s 124 4 parallel streams, 4 different shards, 512 MiB each 1220 MB/s aggregate 125 56 scattered 4 MiB preads at random offsets 326 MB/s 126 --------------------------------------------------------------------------- 127 Four-way parallelism yields 2.4x over a single stream, so this device has real 128 concurrency headroom. The warm re-read being SLOWER than the cold one (408 vs 129 507) shows the page cache does not help: 25 GiB of RAM against a 643 GB model 130 has nothing to cache, and the vhdx indirection costs. 131 132OPERATING SYSTEM 133 Windows 11 Pro (host) + WSL2 Ubuntu (guest) 134 Kernel 6.18.33.2-microsoft-standard-WSL2 x86_64 135 /etc/wsl.conf [boot] systemd=true [network] generateResolvConf=true 136 .wslconfig [wsl2] memory=26GB processors=12 vmIdleTimeout=-1 swap=8GB 137 138TOOLCHAIN 139 clang 21.1.8 140 gcc 15.2.0 141 Python 3.14.4 142 pip packages huggingface_hub 1.26.0 | numpy 2.5.1 143 NOT INSTALLED torch, transformers, safetensors (python), tokenizers, 144 accelerate, llama.cpp, ollama, vllm, sglang 145 -> there is currently no way to run this model on this machine 146 -> PyTorch has no stable cp314 wheels; a torch path needs a 147 separate Python 3.11/3.12 interpreter or venv 148 149ACCESS NOTES 150 The Windows host's default SSH shell is cmd.exe, not bash. 151 Run inside the distro with: wsl.exe -d Ubuntu -- bash -c "..." 152 cmd.exe caps command lines at 8191 characters; for anything long the reliable 153 pattern is base64 -> wsl -- bash -c "echo <b64> | base64 -d | bash". 154 WSL2 tears down backgrounded jobs when a one-shot command returns, so long 155 jobs must run under systemd (systemd-run or a unit file). 156 157 158================================================================================ 159PART 2 FILE ARCHITECTURE [MEASURED] 160================================================================================ 161 162-------------------------------------------------------------------------------- 1632.1 DIRECTORY INVENTORY 164-------------------------------------------------------------------------------- 165/home/joseph/DeepSeek-R1 643 GB, 175 entries 166 167 bytes file role 168 --------------------------------------------------------------------------- 169 1,660 config.json architecture + quantization + rope 170 9,897 configuration_deepseek.py DeepseekV3Config, 199 lines 171 75,741 modeling_deepseek.py reference forward pass, 1,848 lines 172 8,898,324 model.safetensors.index.json weight map: 91,991 -> 163 shards 173 7,847,602 tokenizer.json BPE vocab + merges + added tokens 174 3,594 tokenizer_config.json chat template, special tokens 175 171 generation_config.json sampling defaults 176 15,994 README.md official model card 177 1,064 LICENSE MIT 178 1,519 .gitattributes git-lfs filter rules 179 777,314 figures/benchmark.jpg model card benchmark chart 180 --------------------------------------------------------------------------- 181 + 163 x model-000NN-of-000163.safetensors the weights 182 + .cache/huggingface/download/ 353 entries of download bookkeeping 183 184-------------------------------------------------------------------------------- 1852.2 THE SAFETENSORS CONTAINER FORMAT 186-------------------------------------------------------------------------------- 187Every shard has the same three-region layout: 188 189 byte 0 8 8+N EOF 190 +-----------------+-------------------+---------------------------+ 191 | u64 little-end | JSON header | raw tensor data | 192 | N = header len | N bytes | concatenated, no padding | 193 +-----------------+-------------------+---------------------------+ 194 195 The JSON header maps every tensor name to an object: 196 197 "model.layers.3.mlp.experts.0.gate_proj.weight": { 198 "dtype": "F8_E4M3", 199 "shape": [2048, 7168], 200 "data_offsets": [start, end] // relative to 8+N, not to 0 201 } 202 203 plus one reserved key "__metadata__" which here is always {"format": "pt"}. 204 205 Absolute file offset of a tensor = 8 + N + data_offsets[0] 206 Tensor byte length = data_offsets[1] - data_offsets[0] 207 208 MEASURED header sizes: 209 shard 001: N = 34,119 B data begins at byte 34,127 file 5,234,139,343 B 210 shard 080: N = 76,042 B data begins at byte 76,050 file 4,302,384,914 B 211 shard 163: N = 48,183 B data begins at byte 48,191 file 6,584,784,447 B 212 213 THE HEADER IS THE AUTHORITATIVE SOURCE for dtype, full shape (all ranks), 214 offset and length. The index file carries only the name -> shard mapping. 215 216-------------------------------------------------------------------------------- 2172.3 model.safetensors.index.json STRUCTURE 218-------------------------------------------------------------------------------- 219 Top-level keys: ["metadata", "weight_map"] 220 221 metadata {"total_size": 1369062772000} 222 weight_map 91,991 entries, each "<tensor name>": "model-000NN-of-000163.safetensors" 223 e.g. "model.embed_tokens.weight" -> "model-00001-of-000163.safetensors" 224 225 It names 163 distinct shards and 91,991 distinct tensors. 226 227 WARNING: metadata.total_size reads 1,369,062,772,000 bytes (1.245 TiB) while the 228 actual payload is 688,574,839,360 bytes (641.29 GiB). It overstates by 1.99x 229 because the exporter costed FP8 elements at two bytes each. Never size buffers, 230 allocations or progress reporting from this field. 231 232-------------------------------------------------------------------------------- 2332.4 LAYER -> SHARD MAP 234-------------------------------------------------------------------------------- 235Which files must be opened to materialize a given layer. A loader that walks the 236model layer by layer needs this table; a loader that walks shard by shard needs 2.5. 237 238 layer shards n role layer shards n role 239 -------------------------------------- -------------------------------------- 240 global 1, 160 2 embed/norm/head 31 79-81 3 MoE 241 0 1 1 dense 32 81-84 4 MoE 242 1 1 1 dense 33 84-87 4 MoE 243 2 1 1 dense 34 87-89 3 MoE 244 3 1-4 4 MoE 35 89-92 4 MoE 245 4 4-6 3 MoE 36 92-95 4 MoE 246 5 6-9 4 MoE 37 95-97 3 MoE 247 6 9-12 4 MoE 38 97-100 4 MoE 248 7 13-15 3 MoE 39 101-103 3 MoE 249 8 15-18 4 MoE 40 103-106 4 MoE 250 9 18-21 4 MoE 41 106-109 4 MoE 251 10 21-23 3 MoE 42 109-111 3 MoE 252 11 23-26 4 MoE 43 111-114 4 MoE 253 12 26-29 4 MoE 44 114-117 4 MoE 254 13 29-31 3 MoE 45 117-119 3 MoE 255 14 31-34 4 MoE 46 119-122 4 MoE 256 15 35-37 3 MoE 47 123-125 3 MoE 257 16 37-40 4 MoE 48 125-128 4 MoE 258 17 40-43 4 MoE 49 128-131 4 MoE 259 18 43-45 3 MoE 50 131-133 3 MoE 260 19 45-48 4 MoE 51 133-136 4 MoE 261 20 48-51 4 MoE 52 136-139 4 MoE 262 21 51-53 3 MoE 53 139-141 3 MoE 263 22 53-56 4 MoE 54 142-144 3 MoE 264 23 57-59 3 MoE 55 144-147 4 MoE 265 24 59-62 4 MoE 56 147-150 4 MoE 266 25 62-65 4 MoE 57 150-152 3 MoE 267 26 65-67 3 MoE 58 152-155 4 MoE 268 27 67-70 4 MoE 59 155-158 4 MoE 269 28 70-73 4 MoE 60 158-160 3 MoE 270 29 73-75 3 MoE 61 160-163 4 MTP 271 30 75-78 4 MoE 272 -------------------------------------- -------------------------------------- 273 274 Every MoE layer spans 3 or 4 shards. No layer is contained in a single file 275 except the three dense layers, which all share shard 1 with embed_tokens. 276 The global tensors are split: embed_tokens in shard 1, model.norm and lm_head 277 in shard 160 (NOT the last shard - shards 161-163 are entirely MTP layer 61). 278 279-------------------------------------------------------------------------------- 2802.5 SHARD -> LAYER MAP (all 163) 281-------------------------------------------------------------------------------- 282 shard layers tensors | shard layers tensors | shard layers tensors 283 -------------------------+--------------------------+------------------------- 284 1 0-3 270 * | 56 22 241 | 111 42-43 576 285 2 3 586 | 57 23 573 | 112 43 586 286 3 3 586 | 58 23 586 | 113 43 586 287 4 3-4 576 | 59 23-24 576 | 114 43-44 576 288 5 4 586 | 60 24 586 | 115 44 586 289 6 4-5 584 | 61 24 586 | 116 44 586 290 7 5 588 | 62 24-25 576 | 117 44-45 576 291 8 5 586 | 63 25 586 | 118 45 586 292 9 5-6 576 | 64 25 586 | 119 45-46 576 293 10 6 586 | 65 25-26 576 | 120 46 586 294 11 6 586 | 66 26 586 | 121 46 586 295 12 6 183 | 67 26-27 576 | 122 46 241 296 13 7 573 | 68 27 586 | 123 47 573 297 14 7 586 | 69 27 586 | 124 47 586 298 15 7-8 576 | 70 27-28 576 | 125 47-48 576 299 16 8 586 | 71 28 586 | 126 48 586 300 17 8 586 | 72 28 586 | 127 48 586 301 18 8-9 576 | 73 28-29 576 | 128 48-49 576 302 19 9 586 | 74 29 586 | 129 49 586 303 20 9 586 | 75 29-30 576 | 130 49 586 304 21 9-10 576 | 76 30 586 | 131 49-50 576 305 22 10 586 | 77 30 586 | 132 50 586 306 23 10-11 576 | 78 30 241 | 133 50-51 576 307 24 11 586 | 79 31 573 | 134 51 586 308 25 11 586 | 80 31 586 | 135 51 586 309 26 11-12 576 | 81 31-32 576 | 136 51-52 576 310 27 12 586 | 82 32 586 | 137 52 586 311 28 12 586 | 83 32 586 | 138 52 586 312 29 12-13 576 | 84 32-33 576 | 139 52-53 576 313 30 13 586 | 85 33 586 | 140 53 586 314 31 13-14 576 | 86 33 586 | 141 53 431 315 32 14 586 | 87 33-34 576 | 142 54 573 316 33 14 586 | 88 34 586 | 143 54 586 317 34 14 241 | 89 34-35 576 | 144 54-55 576 318 35 15 573 | 90 35 586 | 145 55 586 319 36 15 586 | 91 35 586 | 146 55 586 320 37 15-16 576 | 92 35-36 576 | 147 55-56 576 321 38 16 586 | 93 36 586 | 148 56 586 322 39 16 586 | 94 36 586 | 149 56 586 323 40 16-17 576 | 95 36-37 576 | 150 56-57 576 324 41 17 586 | 96 37 586 | 151 57 586 325 42 17 586 | 97 37-38 576 | 152 57-58 576 326 43 17-18 576 | 98 38 586 | 153 58 586 327 44 18 586 | 99 38 586 | 154 58 586 328 45 18-19 576 | 100 38 241 | 155 58-59 576 329 46 19 586 | 101 39 573 | 156 59 586 330 47 19 586 | 102 39 586 | 157 59 586 331 48 19-20 576 | 103 39-40 576 | 158 59-60 576 332 49 20 586 | 104 40 586 | 159 60 586 333 50 20 586 | 105 40 586 | 160 60-61 452 ** 334 51 20-21 576 | 106 40-41 576 | 161 61 586 335 52 21 586 | 107 41 586 | 162 61 586 336 53 21-22 576 | 108 41 586 | 163 61 373 337 54 22 586 | 109 41-42 576 | 338 55 22 586 | 110 42 586 | 339 -------------------------+--------------------------+------------------------- 340 * shard 1 also holds model.embed_tokens.weight 341 ** shard 160 also holds model.norm.weight and lm_head.weight 342 343-------------------------------------------------------------------------------- 3442.6 ON-DISK EXPERT LAYOUT 345-------------------------------------------------------------------------------- 346Each routed expert's six tensors are stored CONTIGUOUSLY, always in this order: 347 348 gate_proj.weight 14,680,064 B F8_E4M3 (2048, 7168) 349 gate_proj.weight_scale_inv 3,584 B F32 (16, 56) 350 up_proj.weight 14,680,064 B F8_E4M3 (2048, 7168) 351 up_proj.weight_scale_inv 3,584 B F32 (16, 56) 352 down_proj.weight 14,680,064 B F8_E4M3 (7168, 2048) 353 down_proj.weight_scale_inv 3,584 B F32 (56, 16) 354 ---------------------------------------------------------------- 355 TOTAL PER EXPERT 44,050,944 B = 42.01 MiB exactly 356 357 Verified on shard 80: 98 of 98 experts fully contiguous, span exactly equal to 358 payload, zero interior gaps. One expert is one seek and one read. 359 360 Experts CAN straddle a shard boundary. Observed: layer 31 expert 92 begins in 361 shard 79 (gate_proj) and continues in shard 80 (its scale onward), leaving a 362 within-shard span of 29,370,880 rather than 44,050,944. With 162 shard 363 boundaries, roughly 162 of the 14,848 inference-path experts need two reads. 364 365-------------------------------------------------------------------------------- 3662.7 INTEGRITY VERIFICATION PERFORMED THIS SESSION 367-------------------------------------------------------------------------------- 368 Method: read the 8-byte length prefix and JSON header of every shard; compare 369 the header+data extent to the on-disk file size; cross-check every tensor name 370 and shard assignment against model.safetensors.index.json. 371 372 --------------------------------------------------------------------------- 373 shards where 8 + N + max(data_offsets) != filesize ....... 0 of 163 374 tensors in headers but absent from the index ............. 0 375 tensors in the index but absent from headers ............. 0 376 shard-placement disagreements (index vs header) .......... 0 377 --------------------------------------------------------------------------- 378 The checkpoint is complete and internally consistent; nothing is truncated. 379 This is a STRUCTURAL check. Per-shard content hashes against the upstream 380 repository were not computed. 381 382 DOWNLOAD RESIDUE 383 .cache/huggingface/download/ holds 353 entries: .lock and .metadata files plus 384 8 orphaned *.incomplete blobs (274 MB, 268 MB, 134 MB, 67 MB, 8.4 MB, 35 KB and 385 two zero-byte) left by an aborted first download attempt. No live shard depends 386 on them; deleting the directory reclaims about 0.70 GiB. 387 388 389================================================================================ 390PART 3 ARCHITECTURE SPECIFICATION [SPEC] 391================================================================================ 392 393 architectures ["DeepseekV3ForCausalLM"] 394 model_type deepseek_v3 395 torch_dtype bfloat16 the unquantized reference dtype 396 transformers_version 4.46.3 397 398 CORE DIMENSIONS 399 hidden_size H 7168 400 num_hidden_layers L 61 (+1 MTP layer at index 61 = 62 in checkpoint) 401 first_k_dense_replace 3 layers 0,1,2 dense MLP; layers 3..60 MoE 402 intermediate_size 18432 dense MLP inner dimension 403 hidden_act silu SwiGLU: down( silu(gate(x)) * up(x) ) 404 rms_norm_eps 1e-6 RMSNorm throughout; no LayerNorm, no biases 405 vocab_size V 129280 406 tie_word_embeddings false lm_head is a separate 1.726 GiB tensor 407 initializer_range 0.02 408 use_cache true 409 410 MIXTURE OF EXPERTS 411 n_routed_experts E 256 412 num_experts_per_tok k 8 413 moe_intermediate_size 2048 per-expert inner dimension 414 n_shared_experts 1 always-on expert in every MoE layer 415 moe_layer_freq 1 every layer past the dense prefix is MoE 416 scoring_func sigmoid (NOT softmax) 417 topk_method noaux_tc group-limited routing, see PART 7 418 n_group 8 256 experts partitioned into 8 groups of 32 419 topk_group 4 only 4 of the 8 groups eligible per token 420 norm_topk_prob true selected weights renormalized to sum 1 421 routed_scaling_factor 2.5 applied AFTER renormalization 422 ep_size 1 no expert parallelism in this config 423 424 MULTI-HEAD LATENT ATTENTION (MLA) 425 num_attention_heads 128 426 num_key_value_heads 128 no GQA sharing; MLA compresses instead 427 q_lora_rank 1536 query down-projection rank 428 kv_lora_rank 512 KV down-projection rank - the cached dimension 429 qk_nope_head_dim 128 non-positional part of each QK head 430 qk_rope_head_dim 64 RoPE-carrying part of each QK head 431 q_head_dim 192 = 128 + 64 432 v_head_dim 128 433 attention_bias false 434 attention_dropout 0.0 435 softmax_scale 0.135237 [COMPUTED] = 192^-0.5 * mscale^2, where 436 mscale = 0.1*1.0*ln(40)+1 = 1.3688879. 437 mscale_all_dim=1.0 activates this path, so it 438 is NOT the plain 1/sqrt(192) = 0.0721688. 439 440 POSITION ENCODING - YaRN 441 rope_theta 10000 442 max_position_embeddings 163840 443 rope_scaling.type yarn 444 rope_scaling.factor 40 445 original_max_position_embeddings 4096 (4096 x 40 = 163840) 446 rope_scaling.beta_fast 32 447 rope_scaling.beta_slow 1 448 rope_scaling.mscale 1.0 449 rope_scaling.mscale_all_dim 1.0 450 451 QUANTIZATION 452 quant_method fp8 453 fmt e4m3 454 weight_block_size [128, 128] 455 activation_scheme dynamic 456 457 MULTI-TOKEN PREDICTION 458 num_nextn_predict_layers 1 checkpoint layer 61, speculative decoding. 459 Not part of the base autoregressive forward. 460 461 GENERATION DEFAULTS (generation_config.json) 462 do_sample true | temperature 0.6 | top_p 0.95 | bos_token_id 0 | eos_token_id 1 463 464 465================================================================================ 466PART 4 COMPLETE ARCHITECTURE MAP [MEASURED + SPEC] 467================================================================================ 468Every parameter tensor in the model, organized by module. Shapes and byte counts 469are read from the safetensors headers; module structure from the reference source. 470 471-------------------------------------------------------------------------------- 4724.1 TOP-LEVEL MODULE TREE 473-------------------------------------------------------------------------------- 474 475DeepseekV3ForCausalLM 476 | 477 +-- model : DeepseekV3Model 478 | | 479 | +-- embed_tokens : nn.Embedding(129280, 7168) BF16 1.726 GiB 480 | | 481 | +-- layers : ModuleList[62] 482 | | | 483 | | +-- [0] [1] [2] DeepseekV3DecoderLayer DENSE 556.6 MiB each 484 | | | +-- input_layernorm : DeepseekV3RMSNorm(7168) 485 | | | +-- self_attn : DeepseekV3Attention 486 | | | +-- post_attention_layernorm : DeepseekV3RMSNorm(7168) 487 | | | +-- mlp : DeepseekV3MLP(7168 -> 18432) 488 | | | 489 | | +-- [3] .. [60] DeepseekV3DecoderLayer MoE 10.73 GiB each 490 | | | +-- input_layernorm : DeepseekV3RMSNorm(7168) 491 | | | +-- self_attn : DeepseekV3Attention 492 | | | +-- post_attention_layernorm : DeepseekV3RMSNorm(7168) 493 | | | +-- mlp : DeepseekV3MoE 494 | | | +-- gate : MoEGate 495 | | | | +-- weight (256, 7168) BF16 496 | | | | +-- e_score_correction_bias (256,) F32 497 | | | +-- experts : ModuleList[256] of DeepseekV3MLP 498 | | | | (7168 -> 2048), fp8 499 | | | +-- shared_experts : DeepseekV3MLP(7168 -> 2048), fp8 500 | | | 501 | | +-- [61] MTP LAYER (multi-token prediction) 14.34 GiB 502 | | +-- enorm / hnorm : RMSNorm(7168) 503 | | +-- eh_proj : (7168, 14336) BF16 504 | | +-- embed_tokens : (129280, 7168) BF16 own copy 505 | | +-- input_layernorm : RMSNorm(7168) 506 | | +-- self_attn : DeepseekV3Attention 507 | | +-- post_attention_layernorm : RMSNorm(7168) 508 | | +-- mlp : DeepseekV3MoE (256 experts) 509 | | +-- shared_head.norm : RMSNorm(7168) 510 | | +-- shared_head.head : (129280, 7168) BF16 own head 511 | | 512 | +-- norm : DeepseekV3RMSNorm(7168) BF16 14,336 B 513 | 514 +-- lm_head : nn.Linear(7168 -> 129280, bias=False) BF16 1.726 GiB 515 516 517 DeepseekV3Attention, expanded (identical in all 62 layers): 518 519 hidden (7168) 520 +-- q_a_proj (1536, 7168) fp8 11,010,048 B 521 +-- q_a_layernorm (1536,) BF16 3,072 B 522 +-- q_b_proj (24576, 1536) fp8 37,748,736 B 523 +-- kv_a_proj_with_mqa (576, 7168) fp8 4,128,768 B 524 +-- kv_a_layernorm (512,) BF16 1,024 B 525 +-- kv_b_proj (32768, 512) fp8 16,777,216 B 526 +-- o_proj (7168, 16384) fp8 117,440,512 B 527 ------------- 528 attention subtotal 187,105,280 B 529 (+ 45,792 B of F32 scales) 530 531-------------------------------------------------------------------------------- 5324.2 DENSE LAYER MANIFEST - layers 0, 1, 2 (20 tensors each) 533-------------------------------------------------------------------------------- 534 dtype shape bytes name (relative to model.layers.N.) 535 --------------------------------------------------------------------------- 536 BF16 [7168] 14,336 input_layernorm.weight 537 F8_E4M3 [1536, 7168] 11,010,048 self_attn.q_a_proj.weight 538 F32 [12, 56] 2,688 self_attn.q_a_proj.weight_scale_inv 539 BF16 [1536] 3,072 self_attn.q_a_layernorm.weight 540 F8_E4M3 [24576, 1536] 37,748,736 self_attn.q_b_proj.weight 541 F32 [192, 12] 9,216 self_attn.q_b_proj.weight_scale_inv 542 F8_E4M3 [576, 7168] 4,128,768 self_attn.kv_a_proj_with_mqa.weight 543 F32 [5, 56] 1,120 self_attn.kv_a_proj_with_mqa.weight_scale_inv 544 BF16 [512] 1,024 self_attn.kv_a_layernorm.weight 545 F8_E4M3 [32768, 512] 16,777,216 self_attn.kv_b_proj.weight 546 F32 [256, 4] 4,096 self_attn.kv_b_proj.weight_scale_inv 547 F8_E4M3 [7168, 16384] 117,440,512 self_attn.o_proj.weight 548 F32 [56, 128] 28,672 self_attn.o_proj.weight_scale_inv 549 BF16 [7168] 14,336 post_attention_layernorm.weight 550 F8_E4M3 [18432, 7168] 132,120,576 mlp.gate_proj.weight 551 F32 [144, 56] 32,256 mlp.gate_proj.weight_scale_inv 552 F8_E4M3 [18432, 7168] 132,120,576 mlp.up_proj.weight 553 F32 [144, 56] 32,256 mlp.up_proj.weight_scale_inv 554 F8_E4M3 [7168, 18432] 132,120,576 mlp.down_proj.weight 555 F32 [56, 144] 32,256 mlp.down_proj.weight_scale_inv 556 --------------------------------------------------------------------------- 557 TOTAL 583,642,336 B = 556.6 MiB per dense layer, x3 layers = 1.63 GiB 558 559-------------------------------------------------------------------------------- 5604.3 MoE LAYER MANIFEST - layers 3..60 (22 non-expert + 256 x 6 tensors) 561-------------------------------------------------------------------------------- 562 dtype shape bytes name (relative to model.layers.N.) 563 --------------------------------------------------------------------------- 564 BF16 [7168] 14,336 input_layernorm.weight 565 F8_E4M3 [1536, 7168] 11,010,048 self_attn.q_a_proj.weight 566 F32 [12, 56] 2,688 self_attn.q_a_proj.weight_scale_inv 567 BF16 [1536] 3,072 self_attn.q_a_layernorm.weight 568 F8_E4M3 [24576, 1536] 37,748,736 self_attn.q_b_proj.weight 569 F32 [192, 12] 9,216 self_attn.q_b_proj.weight_scale_inv 570 F8_E4M3 [576, 7168] 4,128,768 self_attn.kv_a_proj_with_mqa.weight 571 F32 [5, 56] 1,120 self_attn.kv_a_proj_with_mqa.weight_scale_inv 572 BF16 [512] 1,024 self_attn.kv_a_layernorm.weight 573 F8_E4M3 [32768, 512] 16,777,216 self_attn.kv_b_proj.weight 574 F32 [256, 4] 4,096 self_attn.kv_b_proj.weight_scale_inv 575 F8_E4M3 [7168, 16384] 117,440,512 self_attn.o_proj.weight 576 F32 [56, 128] 28,672 self_attn.o_proj.weight_scale_inv 577 BF16 [7168] 14,336 post_attention_layernorm.weight 578 F32 [256] 1,024 mlp.gate.e_score_correction_bias 579 BF16 [256, 7168] 3,670,016 mlp.gate.weight 580 F8_E4M3 [2048, 7168] 14,680,064 mlp.shared_experts.gate_proj.weight 581 F32 [16, 56] 3,584 mlp.shared_experts.gate_proj.weight_scale_inv 582 F8_E4M3 [2048, 7168] 14,680,064 mlp.shared_experts.up_proj.weight 583 F32 [16, 56] 3,584 mlp.shared_experts.up_proj.weight_scale_inv 584 F8_E4M3 [7168, 2048] 14,680,064 mlp.shared_experts.down_proj.weight 585 F32 [56, 16] 3,584 mlp.shared_experts.down_proj.weight_scale_inv 586 --------------------------------------------------------------------------- 587 non-expert subtotal 234,905,824 B = 224.0 MiB 588 + 256 routed experts x 44,050,944 B = 11,277,041,664 B = 10.50 GiB 589 --------------------------------------------------------------------------- 590 TOTAL PER MoE LAYER 11,511,947,488 B = 10.72 GiB x58 = 621.9 GiB 591 592-------------------------------------------------------------------------------- 5934.4 ONE ROUTED EXPERT - complete model.layers.N.mlp.experts.M. 594-------------------------------------------------------------------------------- 595 F8_E4M3 [2048, 7168] 14,680,064 gate_proj.weight 596 F32 [16, 56] 3,584 gate_proj.weight_scale_inv 597 F8_E4M3 [2048, 7168] 14,680,064 up_proj.weight 598 F32 [16, 56] 3,584 up_proj.weight_scale_inv 599 F8_E4M3 [7168, 2048] 14,680,064 down_proj.weight 600 F32 [56, 16] 3,584 down_proj.weight_scale_inv 601 --------------------------------------------------------------------------- 602 TOTAL 44,050,944 B = 42.01 MiB, stored contiguously 603 Computation: down( silu(gate(x)) * up(x) ), 7168 -> 2048 -> 7168 604 605-------------------------------------------------------------------------------- 6064.5 MTP LAYER MANIFEST - layer 61 (28 non-expert + 256 x 6 tensors) 607-------------------------------------------------------------------------------- 608 dtype shape bytes name (relative to model.layers.61.) 609 --------------------------------------------------------------------------- 610 BF16 [7168] 14,336 enorm.weight 611 BF16 [7168] 14,336 hnorm.weight 612 BF16 [7168, 14336] 205,520,896 eh_proj.weight 613 BF16 [129280, 7168] 1,853,358,080 embed_tokens.weight (own copy) 614 BF16 [7168] 14,336 input_layernorm.weight 615 F8_E4M3 [1536, 7168] 11,010,048 self_attn.q_a_proj.weight 616 F32 [12, 56] 2,688 self_attn.q_a_proj.weight_scale_inv 617 BF16 [1536] 3,072 self_attn.q_a_layernorm.weight 618 F8_E4M3 [24576, 1536] 37,748,736 self_attn.q_b_proj.weight 619 F32 [192, 12] 9,216 self_attn.q_b_proj.weight_scale_inv 620 F8_E4M3 [576, 7168] 4,128,768 self_attn.kv_a_proj_with_mqa.weight 621 F32 [5, 56] 1,120 self_attn.kv_a_proj_with_mqa.weight_scale_inv 622 BF16 [512] 1,024 self_attn.kv_a_layernorm.weight 623 F8_E4M3 [32768, 512] 16,777,216 self_attn.kv_b_proj.weight 624 F32 [256, 4] 4,096 self_attn.kv_b_proj.weight_scale_inv 625 F8_E4M3 [7168, 16384] 117,440,512 self_attn.o_proj.weight 626 F32 [56, 128] 28,672 self_attn.o_proj.weight_scale_inv 627 BF16 [7168] 14,336 post_attention_layernorm.weight 628 F32 [256] 1,024 mlp.gate.e_score_correction_bias 629 BF16 [256, 7168] 3,670,016 mlp.gate.weight 630 F8_E4M3 [2048, 7168] 14,680,064 mlp.shared_experts.gate_proj.weight 631 F32 [16, 56] 3,584 mlp.shared_experts.gate_proj.weight_scale_inv 632 F8_E4M3 [2048, 7168] 14,680,064 mlp.shared_experts.up_proj.weight 633 F32 [16, 56] 3,584 mlp.shared_experts.up_proj.weight_scale_inv 634 F8_E4M3 [7168, 2048] 14,680,064 mlp.shared_experts.down_proj.weight 635 F32 [56, 16] 3,584 mlp.shared_experts.down_proj.weight_scale_inv 636 BF16 [7168] 14,336 shared_head.norm.weight 637 BF16 [129280, 7168] 1,853,358,080 shared_head.head.weight (own head) 638 --------------------------------------------------------------------------- 639 non-expert subtotal 4,147,185,888 B = 3.86 GiB 640 + 256 routed experts = 11,277,041,664 B = 10.50 GiB 641 --------------------------------------------------------------------------- 642 TOTAL MTP LAYER 15,424,227,552 B = 14.36 GiB 643 644 The MTP layer carries its OWN embed_tokens and its OWN output head, which is 645 why the checkpoint contains four 129280x7168 BF16 tensors rather than two. 646 647-------------------------------------------------------------------------------- 6484.6 GLOBAL TENSORS 649-------------------------------------------------------------------------------- 650 BF16 [129280, 7168] 1,853,358,080 model.embed_tokens.weight [shard 1] 651 BF16 [7168] 14,336 model.norm.weight [shard 160] 652 BF16 [129280, 7168] 1,853,358,080 lm_head.weight [shard 160] 653 --------------------------------------------------------------------------- 654 These are the only three tensors outside the model.layers.* namespace. 655 656-------------------------------------------------------------------------------- 6574.7 PARAMETER ACCOUNTING 658-------------------------------------------------------------------------------- 659 component count bytes GiB 660 --------------------------------------------------------------------------- 661 3 dense layers x 583,642,336 1,750,927,008 1.63 662 58 MoE layers x 11,511,947,488 667,692,954,304 621.83 663 1 MTP layer 15,424,227,552 14.36 664 embed_tokens + norm + lm_head 3,706,730,496 3.45 665 --------------------------------------------------------------------------- 666 TOTAL 688,574,839,360 641.29 667 --------------------------------------------------------------------------- 668 Matches the measured payload exactly, to the byte. 669 MoE layers are 96.97% of the checkpoint. Routed experts alone (all 59 MoE-bearing 670 layers x 256 x 44,050,944 B = 665,345,458,176 B = 619.65 GiB) are 96.6% of it. 671 672 BASE INFERENCE PATH (excluding the MTP layer): 673,150,611,808 B = 626.93 GiB 673 ACTIVATED PER TOKEN: 37B parameters [CARD] - out of 671B total, because only 674 8 of 256 routed experts fire per layer. 675 676 677================================================================================ 678PART 5 COMPLETE TENSOR CENSUS [MEASURED] 679================================================================================ 680Produced by parsing the JSON header of all 163 safetensors files, which is the 681authoritative source for dtype, full shape, byte offset and byte length. 682 683 TOTAL TENSORS 91,991 684 TOTAL PAYLOAD 688,574,839,360 bytes = 641.29 GiB 685 LAYER INDICES 0 .. 61 (62 distinct: 61 transformer + 1 MTP) 686 687BY DTYPE 688 --------------------------------------------------------------------------- 689 dtype count bytes GiB what it is 690 --------------------------------------------------------------------------- 691 F8_E4M3 45,808 680,571,043,840 633.83 every projection weight 692 F32 45,867 166,222,400 0.15 weight_scale_inv + gate bias 693 BF16 316 7,837,573,120 7.30 embeds, lm_head, norms, gates 694 --------------------------------------------------------------------------- 695 Half of all tensors BY COUNT are tiny F32 scale grids. Tensor count is not a 696 proxy for bytes or for work. 697 698BY FAMILY (layer and expert indices collapsed to N; sorted by total size) 699 ------------------------------------------------------------------------------ 700 family n dtype GiB shape 701 ------------------------------------------------------------------------------ 702 mlp.experts.N.gate_proj.weight 15,104 F8_E4M3 206.500 (2048, 7168) 703 mlp.experts.N.up_proj.weight 15,104 F8_E4M3 206.500 (2048, 7168) 704 mlp.experts.N.down_proj.weight 15,104 F8_E4M3 206.500 (7168, 2048) 705 self_attn.o_proj.weight 62 F8_E4M3 6.781 (7168, 16384) 706 self_attn.q_b_proj.weight 62 F8_E4M3 2.180 (24576, 1536) 707 model.embed_tokens.weight 1 BF16 1.726 (129280, 7168) 708 lm_head.weight 1 BF16 1.726 (129280, 7168) 709 L61 embed_tokens.weight (MTP) 1 BF16 1.726 (129280, 7168) 710 L61 shared_head.head.weight (MTP) 1 BF16 1.726 (129280, 7168) 711 self_attn.kv_b_proj.weight 62 F8_E4M3 0.969 (32768, 512) 712 mlp.shared_experts.gate_proj.weight 59 F8_E4M3 0.807 (2048, 7168) 713 mlp.shared_experts.up_proj.weight 59 F8_E4M3 0.807 (2048, 7168) 714 mlp.shared_experts.down_proj.weight 59 F8_E4M3 0.807 (7168, 2048) 715 self_attn.q_a_proj.weight 62 F8_E4M3 0.636 (1536, 7168) 716 mlp.gate_proj.weight (dense) 3 F8_E4M3 0.369 (18432, 7168) 717 mlp.up_proj.weight (dense) 3 F8_E4M3 0.369 (18432, 7168) 718 mlp.down_proj.weight (dense) 3 F8_E4M3 0.369 (7168, 18432) 719 self_attn.kv_a_proj_with_mqa.weight 62 F8_E4M3 0.238 (576, 7168) 720 mlp.gate.weight (router) 59 BF16 0.202 (256, 7168) 721 L61 eh_proj.weight (MTP) 1 BF16 0.191 (7168, 14336) 722 mlp.experts.N.*.weight_scale_inv 45,312 F32 0.150 (16,56)/(56,16) 723 self_attn.o_proj.weight_scale_inv 62 F32 0.002 (56, 128) 724 input_layernorm.weight 62 BF16 0.001 (7168,) 725 post_attention_layernorm.weight 62 BF16 0.001 (7168,) 726 self_attn.q_b_proj.weight_scale_inv 62 F32 0.001 (192, 12) 727 self_attn.kv_b_proj.weight_scale_inv 62 F32 ~0 (256, 4) 728 mlp.shared_experts.*.weight_scale_inv 177 F32 ~0 (16,56)/(56,16) 729 self_attn.q_a_layernorm.weight 62 BF16 ~0 (1536,) 730 self_attn.q_a_proj.weight_scale_inv 62 F32 ~0 (12, 56) 731 self_attn.kv_a_proj_with_mqa.wsi 62 F32 ~0 (5, 56) 732 self_attn.kv_a_layernorm.weight 62 BF16 ~0 (512,) 733 mlp.gate.e_score_correction_bias 59 F32 ~0 (256,) 734 mlp.{gate,up,down}_proj.weight_scale_inv 9 F32 ~0 dense layers 735 model.norm.weight 1 BF16 ~0 (7168,) 736 L61 enorm / hnorm / shared_head.norm 3 BF16 ~0 (7168,) 737 ------------------------------------------------------------------------------ 738 739 COUNT ARITHMETIC 740 15,104 = 59 MoE-bearing layers x 256 experts 741 59 = layers 3..60 (58 transformer MoE layers) + MTP layer 61 742 62 = layers 0..61 inclusive (attention and layernorms exist in all) 743 3 = the dense prefix, layers 0,1,2 744 91,991 = 62*(8 attn + 2 norm) + 3*6 + 59*(4 + 6) + 15,104*6 + 3 globals + 5 MTP 745 746 747================================================================================ 748PART 6 FP8 QUANTIZATION SCHEME [SPEC] 749================================================================================ 750 751 config.json quantization_config: 752 { "quant_method": "fp8", "fmt": "e4m3", 753 "weight_block_size": [128, 128], "activation_scheme": "dynamic" } 754 755FORMAT - FP8 E4M3 (torch float8_e4m3fn semantics) 756 1 sign bit | 4 exponent bits | 3 mantissa bits | 1 byte per element 757 Exponent bias 7. No infinities. NaN encoded as 0x7F / 0xFF. 758 Maximum finite magnitude 448.0. Subnormals present (exponent field 0). 759 The i7-8700K has no hardware fp8 convert instruction; decode is a 256-entry 760 lookup table indexed by the raw byte, built once at load. 761 762BLOCK DEQUANTIZATION 763 For a weight tensor W of shape (R, C) with companion scale S of shape 764 (ceil(R/128), ceil(C/128)): 765 766 W_real[r][c] = fp8_to_f32( W[r][c] ) * S[ r/128 ][ c/128 ] 767 768 The scale is named "weight_scale_inv" but is applied by multiplication in the 769 standard fp8 kernels. Confirm the direction against a reference on a real cell 770 rather than inferring it from the name. 771 772 ACTIVATION QUANTIZATION IS DYNAMIC: activations are quantized at runtime from 773 their own observed range, not from any stored scale. There are no activation 774 scale tensors in the checkpoint, and there is nothing to load for them. 775 776 Every block grid observed, confirming 128x128 tiling: 777 (2048, 7168) -> (16, 56) 2048/128=16 7168/128=56 778 (7168, 2048) -> (56, 16) 779 (7168,16384) -> (56, 128) 780 (24576,1536) -> (192, 12) 781 (32768, 512) -> (256, 4) 782 (1536, 7168) -> (12, 56) 783 (18432,7168) -> (144, 56) 784 (7168,18432) -> (56, 144) 785 ( 576, 7168) -> (5, 56) 576/128 = 4.5 -> PADDED, see PART 14 786 787WHAT IS NOT QUANTIZED 788 All BF16, requiring a separate load path: 789 model.embed_tokens.weight lm_head.weight 790 mlp.gate.weight (the 256x7168 router, one per MoE layer) 791 every *_layernorm.weight and model.norm.weight 792 the MTP layer's eh_proj / enorm / hnorm / shared_head.* 793 All F32: 794 every weight_scale_inv 795 mlp.gate.e_score_correction_bias 796 797 So routing runs entirely in high precision: BF16 weight, F32 bias, and the 798 reference explicitly casts both matmul operands to float32 (source line 426-428). 799 800IMPORTANT: NO FP8 CODE SHIPS WITH THIS CHECKPOINT 801 modeling_deepseek.py contains no fp8 handling at all. Its layers are plain 802 nn.Linear on bfloat16. The only quantization-related lines in the entire 803 1,848-line file are a generic guard at 966-968: 804 805 # Handle the case where the model is quantized 806 if hasattr(self.config, "_pre_quantization_dtype"): 807 target_dtype = self.config._pre_quantization_dtype 808 809 The shipped forward therefore assumes weights have ALREADY been dequantized to 810 bfloat16 by the loading framework. DeepSeek's fp8 kernels live in a separate 811 inference repository which was not downloaded here. The model card says plainly: 812 "Hugging Face's Transformers has not been directly supported yet." 813 814 815================================================================================ 816PART 7 MoE ROUTING ALGORITHM [SPEC] 817================================================================================ 818Transcribed step by step from MoEGate.forward (source lines 422-473). This 819"noaux_tc" group-limited scheme is considerably more involved than plain top-k. 820 821 Input: hidden states h, shape (T, 7168) after flattening batch and sequence. 822 823 1. logits = F.linear( h.float32 , gate.weight.float32 , None ) -> (T, 256) 824 Both operands explicitly cast to float32 although gate.weight is BF16. 825 826 2. scores = sigmoid(logits) -> (T, 256) 827 Sigmoid, not softmax. Scores do not sum to 1 and are not a distribution. 828 829 3. scores_for_choice = scores + e_score_correction_bias -> (T, 256) 830 The F32 (256,) bias, broadcast over the token dimension. 831 832 4. group_scores = scores_for_choice.view(T, 8, 32).topk(2, -1)[0].sum(-1) 833 -> (T, 8) 834 Each of the 8 groups is scored by THE SUM OF ITS TOP TWO members. 835 836 5. group_idx = topk(group_scores, k=4) -> (T, 4) 837 Only 4 of 8 groups survive; the other 4 (128 experts) are masked to -inf. 838 839 6. topk_idx = topk( masked scores_for_choice , k=8 ) -> (T, 8) 840 The 8 experts are chosen among the 128 that remain. 841 842 7. topk_weight = scores.gather(1, topk_idx) 843 *** Weights are gathered from `scores` - the PRE-bias sigmoid - while the 844 SELECTION in steps 3-6 used scores_for_choice, the POST-bias value. 845 Selection and weighting use different quantities, deliberately. 846 847 8. topk_weight = topk_weight / (topk_weight.sum(-1, keepdim=True) + 1e-20) 848 topk_weight = topk_weight * 2.5 # routed_scaling_factor 849 Renormalize to sum 1, THEN scale by 2.5. 850 851 Then in DeepseekV3MoE.forward (lines 522-532): 852 853 identity = h # the PRE-MoE residual 854 y = moe_infer(h, topk_idx, topk_weight) # sum_i w_i * Expert_i(h) 855 y = y + shared_experts(identity) # shared expert added afterwards 856 857 The shared expert consumes the pre-MoE hidden state, not the routed output. 858 859 moe_infer (lines 535-608) is a gather/scatter batching routine: sort tokens by 860 assigned expert via argsort, run each expert once over its contiguous slice, 861 scatter results back to original order, then form the weighted sum. It is a 862 performance arrangement; the semantics are sum_i w_i * Expert_{idx_i}(h). 863 864 TWO NESTED DECISION BOUNDARIES, not one: 865 - the group cut: rank 4 vs rank 5 among 8 groups, each scored by a 2-elem sum 866 - the expert cut: rank 8 vs rank 9 among the 128 surviving experts 867 A flip at the group boundary can swap up to 8 experts at once. 868 869 870================================================================================ 871PART 8 MLA ATTENTION - MULTI-HEAD LATENT ATTENTION [SPEC] 872================================================================================ 873From DeepseekV3Attention (source lines 627-858). 874 875 QUERY PATH 876 h (7168) 877 -> q_a_proj -> 1536 fp8 (1536, 7168) 878 -> q_a_layernorm (RMSNorm 1536) 879 -> q_b_proj -> 24576 fp8 (24576, 1536) 880 -> reshape to 128 heads x 192 881 -> split each head: q_nope (128) | q_rope (64) 882 883 KEY / VALUE PATH 884 h (7168) 885 -> kv_a_proj_with_mqa -> 576 fp8 (576, 7168) 886 -> split: compressed_kv (512) | k_rope (64) 887 -> kv_a_layernorm (RMSNorm 512) on the 512 part ONLY 888 -> kv_b_proj -> 32768 fp8 (32768, 512) 889 -> reshape to 128 heads x 256 890 -> split each head: k_nope (128) | v (128) 891 892 ROTARY 893 RoPE applied to q_rope (64) and k_rope (64) only. 894 k_rope is SHARED across all 128 heads - this is the "mqa" in the name. 895 q_rope is per-head. 896 897 ATTENTION 898 q = concat(q_nope, q_rope) -> 192 per head 899 k = concat(k_nope, k_rope) -> 192 per head (k_rope broadcast to all heads) 900 scores = q @ k^T * 0.135237 901 probs = softmax(scores, dim=-1) computed in float32, cast back to q.dtype 902 out = probs @ v -> 128 per head 903 out = o_proj(out.reshape(T, 16384)) -> 7168 fp8 (7168, 16384) 904 905 WHAT IS CACHED - THE POINT OF MLA 906 A conventional 128-head KV cache would store K and V per head: 907 128 heads x (128 + 128) x 2 bytes = 65,536 B per token per layer. 908 MLA caches the LATENT instead: 909 compressed_kv (512) + k_rope (64) = 576 elements x 2 B = 1,152 B. 910 A 56.9x reduction. 911 k_nope and v are RECONSTRUCTED from the 512-dim latent through kv_b_proj on 912 every decode step, so the memory saving is paid for in compute: the KV read 913 path and the kv_b_proj matmul are coupled. 914 915 ATTENTION IMPLEMENTATIONS IN THE FILE 916 DeepseekV3Attention eager reference implementation 917 DeepseekV3FlashAttention2 FlashAttention-2 path (lines 860-1140); pads the 918 192-dim query head to v_head_dim and slices the 919 result back; requires the flash_attn package 920 ATTENTION_CLASSES = {"eager": ..., "flash_attention_2": ...} 921 Selected by config._attn_implementation. 922 923 924================================================================================ 925PART 9 YaRN POSITION ENCODING [SPEC] 926================================================================================ 927From DeepseekV3YarnRotaryEmbedding and helpers (source lines 226-330). 928 929 PARAMETERS IN THIS CHECKPOINT 930 dim (qk_rope_head_dim) 64 931 base (rope_theta) 10000 932 original_max_position_embeddings 4096 933 scaling factor 40 934 max_position_embeddings 163840 = 4096 x 40 935 beta_fast / beta_slow 32 / 1 936 mscale / mscale_all_dim 1.0 / 1.0 937 938 THE CORRECTION RANGE 939 yarn_find_correction_dim(rot, dim, base, orig_max) 940 = dim * ln( orig_max / (rot * 2 * pi) ) / ( 2 * ln(base) ) 941 yarn_find_correction_range(beta_fast=32, beta_slow=1, ...) 942 -> (low, high), clamped to [0, dim-1] 943 944 A linear ramp over [low, high] blends two frequency schedules: 945 inv_freq_interpolation = 1 / (factor * base^(2i/dim)) low freq, scaled 946 inv_freq_extrapolation = 1 / (base^(2i/dim)) high freq, raw 947 inv_freq = interp * (1 - ramp) + extrap * ramp 948 High-frequency dimensions extrapolate unchanged; low-frequency dimensions 949 are interpolated by the factor of 40. 950 951 THE MAGNITUDE CORRECTION 952 yarn_get_mscale(scale, mscale) = 0.1 * mscale * ln(scale) + 1.0 for scale>1 953 With scale=40, mscale=1.0: 0.1 * ln(40) + 1.0 = 1.3688879 954 955 cos/sin caches are multiplied by _mscale, computed as 956 yarn_get_mscale(factor, mscale) / yarn_get_mscale(factor, mscale_all_dim) 957 which equals 1.0 here since mscale == mscale_all_dim == 1.0. 958 959 SEPARATELY, in DeepseekV3Attention.__init__ (lines 690-695), because 960 mscale_all_dim is truthy: 961 softmax_scale = 192^-0.5 * mscale * mscale 962 = 0.0721688 * 1.3688879^2 963 = 0.135237 964 This is NOT 1/sqrt(192) = 0.0721688. The 1.874x factor is load-bearing. 965 966 APPLICATION 967 rotate_half(x) splits the last dim in half and returns concat(-x2, x1). 968 apply_rotary_pos_emb indexes cos/sin by position_ids and computes 969 q_out = q*cos + rotate_half(q)*sin 970 after an interleave-to-blocked reshape: (b, h, d) -> (b, h, d/2, 2) 971 -> transpose(4,3) -> reshape back. Get that reshape wrong and the model 972 produces fluent nonsense rather than an obvious error. 973 974 975================================================================================ 976PART 10 COMPLETE DATAFLOW CHART 977================================================================================ 978 979-------------------------------------------------------------------------------- 98010.1 PREFILL VS DECODE 981-------------------------------------------------------------------------------- 982 PREFILL (first pass over the prompt, T tokens at once) 983 - all T tokens flow through all 61 layers in one batch 984 - the causal mask is a full lower-triangular (T x T) 985 - every token contributes 576 elements per layer to the KV cache 986 - MoE routing is computed per token, so a prefill of T tokens may touch far 987 more than 8 experts per layer - up to all 256 if T is large 988 - cost scales with T for compute, and with distinct-experts-touched for I/O 989 990 DECODE (each subsequent token, T = 1) 991 - one token flows through all 61 layers 992 - attention reads the whole KV cache (length P) and appends 576 elements 993 - exactly 8 routed experts fire per MoE layer -> 464 expert reads per token 994 - cost is dominated by streaming 19.04 GiB of expert weights 995 996 The two regimes have different bottlenecks. Prefill is compute- and mask-bound; 997 decode is expert-I/O-bound. Measure them separately. 998 999-------------------------------------------------------------------------------- 100010.2 FULL TENSOR-SHAPE DATAFLOW (decode step, batch 1, T=1, cache length P) 1001-------------------------------------------------------------------------------- 1002 1003 input_ids (1,) int64 1004 | 1005 v 1006 +======================================================================+ 1007 | model.embed_tokens : Embedding(129280, 7168) BF16 | 1008 +======================================================================+ 1009 | h : (1, 7168) bf16 1010 | 1011 |#################### LAYERS 0,1,2 - DENSE ######################### 1012 v 1013 +----------------------------------------------------------------------+ 1014 | residual = h | 1015 | h = input_layernorm(h) RMSNorm(7168) -> (1, 7168) | 1016 | h = self_attn(h, cache[l]) -> (1, 7168) | 1017 | h = residual + h | 1018 | residual = h | 1019 | h = post_attention_layernorm(h) -> (1, 7168) | 1020 | | 1021 | DENSE MLP (DeepseekV3MLP, intermediate 18432) | 1022 | g = gate_proj(h) (18432,7168)fp8 -> (1, 18432) | 1023 | u = up_proj(h) (18432,7168)fp8 -> (1, 18432) | 1024 | h = down_proj( silu(g) * u ) | 1025 | (7168,18432)fp8 -> (1, 7168) | 1026 | h = residual + h | 1027 +----------------------------------------------------------------------+ 1028 | 1029 |#################### LAYERS 3..60 - MoE ########################### 1030 v 1031 +----------------------------------------------------------------------+ 1032 | residual = h | 1033 | h = input_layernorm(h) -> (1, 7168) | 1034 | | 1035 | +================ MLA ATTENTION =================================+ | 1036 | | | | 1037 | | QUERY | | 1038 | | q = q_a_proj(h) (1536,7168)fp8 -> (1, 1536) | | 1039 | | q = q_a_layernorm(q) RMSNorm(1536) -> (1, 1536) | | 1040 | | q = q_b_proj(q) (24576,1536)fp8 -> (1, 24576) | | 1041 | | q = view(1, 128, 192) | | 1042 | | q_nope, q_rope = split(q, [128, 64], -1) | | 1043 | | q_nope (1,128,128) q_rope (1,128,64) | | 1044 | | | | 1045 | | KEY / VALUE | | 1046 | | c = kv_a_proj_with_mqa(h) (576,7168)fp8 -> (1, 576) | | 1047 | | c_kv, k_rope = split(c, [512, 64], -1) | | 1048 | | c_kv (1,512) k_rope (1,1,64) | | 1049 | | c_kv = kv_a_layernorm(c_kv) RMSNorm(512) -> (1, 512) | | 1050 | | | | 1051 | | ROPE cos,sin = yarn_rotary(position_id) | | 1052 | | q_rope, k_rope = apply_rotary_pos_emb(q_rope, k_rope, | | 1053 | | cos, sin, pos_ids) | | 1054 | | | | 1055 | | CACHE APPEND <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< | | 1056 | | cache[l].k_latent <- append c_kv (P,512) -> (P+1,512) | | 1057 | | cache[l].k_rope <- append k_rope (P,64) -> (P+1,64) | | 1058 | | 1,152 bytes added per layer per token | | 1059 | | | | 1060 | | RECONSTRUCT from the full cache | | 1061 | | kv = kv_b_proj(cache.k_latent) (32768,512)fp8 | | 1062 | | -> (P+1, 32768) | | 1063 | | kv = view(P+1, 128, 256) | | 1064 | | k_nope, v = split(kv, [128, 128], -1) | | 1065 | | k_nope (P+1,128,128) v (P+1,128,128) | | 1066 | | k = concat(k_nope, k_rope.expand(128)) -> (P+1,128,192) | | 1067 | | | | 1068 | | ATTEND | | 1069 | | s = q @ k^T * 0.135237 -> (1, 128, 1, P+1) | | 1070 | | s = s + causal_mask | | 1071 | | a = softmax(s, -1, float32).to(q.dtype) | | 1072 | | o = a @ v -> (1, 128, 1, 128) | | 1073 | | o = o.transpose.reshape(1, 16384) | | 1074 | | o = o_proj(o) (7168,16384)fp8 -> (1, 7168) | | 1075 | +================================================================+ | 1076 | | 1077 | h = residual + o | 1078 | residual = h | 1079 | h = post_attention_layernorm(h) -> (1, 7168) | 1080 | | 1081 | +================ MoE BLOCK =====================================+ | 1082 | | identity = h | | 1083 | | | | 1084 | | ROUTER (float32) | | 1085 | | logits = h.f32 @ gate.weight.f32^T (256,7168)BF16 | | 1086 | | -> (1, 256) | | 1087 | | scores = sigmoid(logits) -> (1, 256) | | 1088 | | choice = scores + e_score_bias (256,)F32 | | 1089 | | grp = choice.view(1,8,32).topk(2)[0].sum(-1) -> (1, 8) | | 1090 | | keep = topk(grp, 4) -> (1, 4) groups | | 1091 | | mask non-kept groups to -inf | | 1092 | | idx = topk(masked_choice, 8) -> (1, 8) expert ids | | 1093 | | w = scores.gather(1, idx) <- PRE-bias scores | | 1094 | | w = w / w.sum(); w = w * 2.5 -> (1, 8) | | 1095 | | | | | 1096 | | +----------+----------------------+ | | 1097 | | v v | | 1098 | | 8 ROUTED EXPERTS 1 SHARED EXPERT | | 1099 | | for i in idx: g = gate_proj(identity) | | 1100 | | READ 44,050,944 B from disk (2048,7168)fp8 | | 1101 | | g = gate_proj(h) -> (1,2048) u = up_proj(identity) | | 1102 | | u = up_proj(h) -> (1,2048) s = down_proj(silu(g)*u) | | 1103 | | e_i = down_proj(silu(g)*u) (7168,2048)fp8 | | 1104 | | -> (1,7168) -> (1, 7168) | | 1105 | | y = sum_i w_i * e_i -> (1,7168) | | 1106 | | | | | | 1107 | | +------------ y + s --------------+ | | 1108 | +================================================================+ | 1109 | | 1110 | h = residual + (y + s) -> (1, 7168) | 1111 +----------------------------------------------------------------------+ 1112 | 1113 | repeat for all 58 MoE layers 1114 | ==> 464 expert reads, 19.04 GiB streamed, per token 1115 v 1116 +======================================================================+ 1117 | model.norm : RMSNorm(7168) BF16 -> (1, 7168) | 1118 | lm_head : Linear(7168 -> 129280) BF16 -> (1, 129280) | 1119 +======================================================================+ 1120 | logits (1, 129280) float32 1121 v 1122 +----------------------------------------------------------------------+ 1123 | SAMPLING temperature 0.6 -> top_p 0.95 -> multinomial | 1124 +----------------------------------------------------------------------+ 1125 | next_token_id 1126 | 1127 +---- if == eos (1) : STOP 1128 +---- else : back to model.embed_tokens 1129 1130-------------------------------------------------------------------------------- 113110.3 KV CACHE LIFECYCLE 1132-------------------------------------------------------------------------------- 1133 1134 Per layer l in 0..60, the cache holds two arrays: 1135 1136 k_latent[l] : (P, 512) bf16 the normalized compressed KV 1137 k_rope[l] : (P, 64) bf16 the rotated positional key, head-shared 1138 1139 PREFILL, T tokens P: 0 -> T writes T * 70,272 B total 1140 DECODE step P: n -> n+1 writes 70,272 B total 1141 reads (n+1) * 70,272 B total 1142 1143 Growth per token, all 61 layers: 61 * 576 * 2 = 70,272 B 1144 1145 context P cache size (bf16) cache size (int8) 1146 --------------------------------------------------------- 1147 4,096 0.27 GiB 0.13 GiB 1148 16,384 1.07 GiB 0.54 GiB 1149 32,768 2.14 GiB 1.07 GiB 1150 65,536 4.29 GiB 2.14 GiB 1151 163,840 10.72 GiB 5.36 GiB 1152 1153 NOTE the coupling: k_nope and v are NOT cached. They are recomputed from 1154 k_latent through kv_b_proj (32768x512 fp8) on every single decode step, over 1155 the entire cache length. That matmul is 16.7 MB of weights x (P+1) rows, so 1156 attention cost grows linearly in P even though the cache itself is small. 1157 This is the term that makes long context expensive here, not cache memory. 1158 1159 In the reference, caching is delegated to the transformers Cache class 1160 (DynamicCache / StaticCache); the model declares _supports_cache_class = True 1161 and _skip_keys_device_placement = "past_key_values". 1162 1163-------------------------------------------------------------------------------- 116410.4 MoE DISPATCH DATAFLOW (moe_infer, prefill with T tokens) 1165-------------------------------------------------------------------------------- 1166 1167 topk_ids (T, 8) topk_weight (T, 8) x (T, 7168) 1168 | 1169 v 1170 cnts = zeros(T, 256); cnts.scatter_(1, topk_ids, 1) 1171 tokens_per_expert = cnts.sum(0) -> (256,) 1172 | 1173 v 1174 idxs = topk_ids.view(-1).argsort() -> (T*8,) 1175 sorted_tokens = x[idxs // 8] -> (T*8, 7168) 1176 | 1177 | now all tokens destined for expert 0 are contiguous, then expert 1, ... 1178 v 1179 start = 0 1180 for i, n in enumerate(tokens_per_expert): 1181 if n == 0: continue <-- experts with no tokens are SKIPPED 1182 out_i = experts[i]( sorted_tokens[start : start+n] ) 1183 start += n 1184 | 1185 v 1186 outs = cat(out_i) -> (T*8, 7168) 1187 new_x = empty_like(outs); new_x[idxs] = outs <-- scatter back 1188 | 1189 v 1190 final = new_x.view(T, 8, 7168) 1191 .mul_(topk_weight.unsqueeze(-1)) 1192 .sum(dim=1) -> (T, 7168) 1193 1194 The `if n == 0: continue` is why decode touches exactly 8 experts per layer 1195 while prefill may touch far more: with T tokens there are up to min(256, 8T) 1196 distinct experts with work to do. 1197 1198 (The ep_size > 1 branch adds dist.all_to_all exchanges for expert parallelism. 1199 ep_size is 1 in this config, so that path is inert here.) 1200 1201-------------------------------------------------------------------------------- 120210.5 MTP / SPECULATIVE DECODE DATAFLOW (layer 61) 1203-------------------------------------------------------------------------------- 1204 1205 h_prev (1, 7168) next_token_id 1206 from the main stack | 1207 | v 1208 | embed_tokens (layer 61's OWN copy) 1209 | | e (1, 7168) 1210 v v 1211 hnorm(h_prev) enorm(e) 1212 | | 1213 +------------ concat -------+ -> (1, 14336) 1214 | 1215 v 1216 eh_proj (7168, 14336) BF16 -> (1, 7168) 1217 | 1218 v 1219 +--------------------------------------------+ 1220 | input_layernorm -> MLA attention -> +res | 1221 | post_attn_layernorm -> MoE (256 exp) -> +res| 1222 +--------------------------------------------+ 1223 | 1224 v 1225 shared_head.norm (RMSNorm 7168) 1226 | 1227 v 1228 shared_head.head (129280, 7168) BF16 1229 | 1230 v 1231 logits for token t+2 (1, 129280) 1232 1233 Purpose: predict one extra token ahead so a verifier pass can accept or reject 1234 it, raising tokens-per-forward. It is NOT required for correct generation and 1235 is not part of the base autoregressive loop. Excluding it saves 14.36 GiB. 1236 In a coverage audit it must be CLASSIFIED as speculative-decoding machinery, 1237 not left as orphan tensors. 1238 1239 1240================================================================================ 1241PART 11 TOKENIZER AND WIRE FORMAT [MEASURED] 1242================================================================================ 1243 1244-------------------------------------------------------------------------------- 124511.1 TOKENIZER PIPELINE 1246-------------------------------------------------------------------------------- 1247 File tokenizer.json, 7,847,602 bytes 1248 Class LlamaTokenizerFast (tokenizer_config.json) 1249 Algorithm BPE 1250 BPE vocab 128,000 entries, ids 0..127,999 1251 Merge rules 127,741 1252 added_tokens 818 (see 11.2) 1253 config vocab 129,280 1254 -> ids 0..128,814 are used; 128,815..129,279 (465 slots) are unused 1255 padding in the embedding matrix, present so the vocab dimension is a 1256 convenient multiple. A loader must still allocate all 129,280 rows. 1257 1258 normalizer Sequence[] - EMPTY. No NFC/NFKC, no lowercasing, no stripping. 1259 Input bytes reach the pre-tokenizer unmodified. 1260 1261 pre_tokenizer Sequence of Split rules then ByteLevel: 1262 1. Split on regex \p{N}{1,3} Isolated 1263 -> digits are grouped 1-3 at a time 1264 2. Split on regex [一-龥぀-ゟ゠-ヿ]+ 1265 Isolated 1266 -> CJK ideographs, Hiragana, Katakana isolated as runs 1267 3. Split on a large punctuation/letter/whitespace regex 1268 4. ByteLevel 1269 decoder ByteLevel, add_prefix_space=true, trim_offsets=true, use_regex=true 1270 post_processor ByteLevel, add_prefix_space=true, trim_offsets=false 1271 1272 truncation null padding null (both unset in the file) 1273 model_max_length 16384 (tokenizer_config.json) 1274 1275-------------------------------------------------------------------------------- 127611.2 COMPLETE SPECIAL-TOKEN TABLE 1277-------------------------------------------------------------------------------- 1278818 added tokens. 798 are reserved placeholders; 20 are meaningful. 1279 1280 id token role 1281 --------------------------------------------------------------------------- 1282 0 <|begin_of_sentence|> BOS. add_bos_token = true 1283 1 <|end_of_sentence|> EOS, and the pad token 1284 2 <|_pad_|> a distinct pad token, unused by config 1285 --------------------------------------------------------------------------- 1286 128000 \ 1287 .. <|place_holder_no_0..797|> > 798 reserved placeholders 1288 128797 / 1289 --------------------------------------------------------------------------- 1290 128798 <think> opens the reasoning trace 1291 128799 </think> closes the reasoning trace 1292 --------------------------------------------------------------------------- 1293 128800 <|fim_hole|> fill-in-the-middle: the gap 1294 128801 <|fim_begin|> fill-in-the-middle: prefix start 1295 128802 <|fim_end|> fill-in-the-middle: suffix end 1296 --------------------------------------------------------------------------- 1297 128803 <|User|> user turn marker 1298 128804 <|Assistant|> assistant turn marker 1299 128805 <|EOT|> end of turn 1300 --------------------------------------------------------------------------- 1301 128806 <|tool_calls_begin|> opens a block of tool calls 1302 128807 <|tool_calls_end|> closes the block 1303 128808 <|tool_call_begin|> opens ONE call 1304 128809 <|tool_call_end|> closes ONE call 1305 128810 <|tool_outputs_begin|> opens a block of tool results 1306 128811 <|tool_outputs_end|> closes the block 1307 128812 <|tool_output_begin|> opens ONE result 1308 128813 <|tool_output_end|> closes ONE result 1309 128814 <|tool_sep|> separates call type from function name 1310 --------------------------------------------------------------------------- 1311 NOTE: the real tokens use fullwidth vertical bars U+FF5C and the U+2581 lower 1312 one-eighth block as a word separator, e.g. the literal bytes are 1313 "<｜begin▁of▁sentence｜>". They are rendered here with plain 1314 ASCII pipes and underscores for legibility. <|EOT|> is the one exception - it 1315 genuinely uses ASCII pipes. Copy them from tokenizer.json, not from this table. 1316 1317-------------------------------------------------------------------------------- 131811.3 CHAT TEMPLATE SEMANTICS 1319-------------------------------------------------------------------------------- 1320The Jinja template in tokenizer_config.json (verbatim in section 15.5) does this: 1321 1322 1. Concatenate every system message, in order, joined by "\n\n", into one 1323 system_prompt string. Emit bos_token, then that string. There is no system 1324 turn marker; the system prompt is bare text immediately after BOS. 1325 2. For each user message: emit <|User|> + content. 1326 3. For each assistant message WITHOUT tool_calls: 1327 - if the content contains "</think>", keep ONLY what follows it 1328 - emit <|Assistant|> + content + <|end_of_sentence|> 1329 *** Reasoning traces from prior turns are STRIPPED before being fed back. 1330 The model does not see its own past <think> blocks. 1331 4. For each assistant message WITH tool_calls: emit <|Assistant|> then, for 1332 each call, <|tool_call_begin|> + type + <|tool_sep|> + function.name + "\n" 1333 + "```json\n" + arguments + "\n```" + <|tool_call_end|>, all wrapped in 1334 <|tool_calls_begin|> ... <|tool_calls_end|> + <|end_of_sentence|>. 1335 5. For each tool message: wrap content in <|tool_output_begin|> ... 1336 <|tool_output_end|>, with the first one opening <|tool_outputs_begin|>. 1337 6. If add_generation_prompt and the last message was not a tool output: 1338 emit <|Assistant|><think>\n 1339 1340 Step 6 is the mechanism behind the model card's instruction to force reasoning: 1341 the template itself seeds the opening <think> tag. 1342 1343-------------------------------------------------------------------------------- 134411.4 A RENDERED EXAMPLE 1345-------------------------------------------------------------------------------- 1346 messages = [ {"role":"user", "content":"What is 2+2?"} ] 1347 add_generation_prompt = True 1348 1349 renders to the literal string: 1350 1351 <|begin_of_sentence|><|User|>What is 2+2?<|Assistant|><think>\n 1352 1353 as token ids: 1354 1355 [0, 128803, ...content ids..., 128804, 128798, <newline id>] 1356 1357 The model then generates its reasoning, emits </think> (128799), generates the 1358 answer, and emits <|end_of_sentence|> (1) to stop. 1359 1360 1361================================================================================ 1362PART 12 API SURFACE AND ENDPOINTS 1363================================================================================ 1364 1365-------------------------------------------------------------------------------- 136612.1 HUGGINGFACE auto_map AND LOAD PATH [SPEC] 1367-------------------------------------------------------------------------------- 1368 config.json declares: 1369 1370 "auto_map": { 1371 "AutoConfig": "configuration_deepseek.DeepseekV3Config", 1372 "AutoModel": "modeling_deepseek.DeepseekV3Model", 1373 "AutoModelForCausalLM":"modeling_deepseek.DeepseekV3ForCausalLM" 1374 } 1375 1376 So the model uses REMOTE CODE: the two .py files in the repo are the 1377 implementation, and loading requires trust_remote_code=True. 1378 1379 from transformers import AutoModelForCausalLM, AutoTokenizer 1380 tok = AutoTokenizer.from_pretrained("/home/joseph/DeepSeek-R1") 1381 model = AutoModelForCausalLM.from_pretrained( 1382 "/home/joseph/DeepSeek-R1", 1383 trust_remote_code=True, 1384 torch_dtype="auto", 1385 device_map="auto") 1386 1387 CAVEAT [CARD]: "Hugging Face's Transformers has not been directly supported 1388 yet." The shipped modeling code has no fp8 dequantization, so this path needs a 1389 framework that understands the fp8 quantization_config. It will not work as-is 1390 on this machine (no torch installed, Python 3.14). 1391 1392-------------------------------------------------------------------------------- 139312.2 PYTHON CLASS API - COMPLETE [SPEC] 1394-------------------------------------------------------------------------------- 1395Every public symbol in the two source files, with line numbers into section 15. 1396 1397 configuration_deepseek.py 1398 --------------------------------------------------------------------------- 1399 DEEPSEEK_PRETRAINED_CONFIG_ARCHIVE_MAP = {} L5 1400 class DeepseekV3Config(PretrainedConfig) L7 1401 model_type = "deepseek_v3" 1402 keys_to_ignore_at_inference = ["past_key_values"] 1403 __init__(vocab_size=129280, hidden_size=7168, intermediate_size=18432, 1404 moe_intermediate_size=2048, num_hidden_layers=61, ...) L112 1405 1406 modeling_deepseek.py 1407 --------------------------------------------------------------------------- 1408 _CONFIG_FOR_DOC = "DeepseekV3Config" 1409 _get_unpad_data(attention_mask) L80 1410 1411 class DeepseekV3RMSNorm(nn.Module) L94 1412 __init__(hidden_size, eps=1e-06) L95 1413 forward(hidden_states) L103 1414 1415 class DeepseekV3RotaryEmbedding(nn.Module) L114 1416 __init__(dim, max_position_embeddings=2048, base=10000, device=None) L115 1417 _set_cos_sin_cache(seq_len, device, dtype) L134 1418 forward(x, seq_len=None) L146 1419 1420 class DeepseekV3LinearScalingRotaryEmbedding(DeepseekV3RotaryEmbedding) L158 1421 __init__(..., scaling_factor=1.0) L161 1422 _set_cos_sin_cache(seq_len, device, dtype) L172 1423 1424 class DeepseekV3DynamicNTKScalingRotaryEmbedding(DeepseekV3RotaryEmbedding) L187 1425 __init__(..., scaling_factor=1.0) L190 1426 _set_cos_sin_cache(seq_len, device, dtype) L201 1427 1428 yarn_find_correction_dim(num_rotations, dim, base, max_position_embeddings) L226 1429 yarn_find_correction_range(low_rot, high_rot, dim, base, max_pos) L235 1430 yarn_get_mscale(scale=1, mscale=1) L247 1431 yarn_linear_ramp_mask(min, max, dim) L253 1432 1433 class DeepseekV3YarnRotaryEmbedding(DeepseekV3RotaryEmbedding) L262 1434 __init__(dim, max_position_embeddings, base, device, scaling_factor, 1435 original_max_position_embeddings, beta_fast, beta_slow, 1436 mscale, mscale_all_dim) L264 1437 _set_cos_sin_cache(seq_len, device, dtype) L285 1438 1439 rotate_half(x) L331 1440 apply_rotary_pos_emb(q, k, cos, sin, position_ids, unsqueeze_dim=1) L339 1441 1442 class DeepseekV3MLP(nn.Module) L374 1443 __init__(config, hidden_size=None, intermediate_size=None) L375 1444 forward(x) L388 1445 1446 class MoEGate(nn.Module) L393 1447 __init__(config) L394 1448 reset_parameters() L417 1449 forward(hidden_states) -> (topk_idx, topk_weight) L422 1450 1451 class DeepseekV3MoE(nn.Module) L475 1452 __init__(config) L480 1453 forward(hidden_states) L522 1454 @torch.no_grad moe_infer(x, topk_ids, topk_weight) L535 1455 1456 repeat_kv(hidden_states, n_rep) L612 1457 1458 class DeepseekV3Attention(nn.Module) L627 1459 __init__(config, layer_idx=None) L630 1460 _init_rope() L697 1461 _shape(tensor, seq_len, bsz) L743 1462 forward(hidden_states, attention_mask, position_ids, 1463 past_key_value, output_attentions, use_cache, **kwargs) L750 1464 1465 class DeepseekV3FlashAttention2(DeepseekV3Attention) L860 1466 __init__(*args, **kwargs) L867 1467 forward(...) L875 1468 _flash_attention_forward(query_states, key_states, value_states, 1469 attention_mask, query_length, dropout, softmax_scale) L1010 1470 _upad_input(query_layer, key_layer, value_layer, 1471 attention_mask, query_length) L1090 1472 1473 ATTENTION_CLASSES = {"eager": DeepseekV3Attention, 1474 "flash_attention_2": DeepseekV3FlashAttention2} 1475 1476 class DeepseekV3DecoderLayer(nn.Module) L1142 1477 __init__(config, layer_idx) L1143 1478 forward(...) L1167 1479 1480 class DeepseekV3PreTrainedModel(PreTrainedModel) L1251 1481 _no_split_modules = ["DeepseekV3DecoderLayer"] 1482 _skip_keys_device_placement = "past_key_values" 1483 _supports_flash_attn_2 = True 1484 _supports_cache_class = True 1485 _init_weights(module) L1260 1486 1487 class DeepseekV3Model(DeepseekV3PreTrainedModel) L1346 1488 __init__(config) L1354 1489 get_input_embeddings() / set_input_embeddings(value) L1375 L1378 1490 forward(...) -> BaseModelOutputWithPast L1382 1491 1492 class DeepseekV3ForCausalLM(DeepseekV3PreTrainedModel) L1514 1493 __init__(config) L1517 1494 get_input_embeddings() / set_input_embeddings(value) L1526 L1529 1495 get_output_embeddings() / set_output_embeddings(new) L1532 L1535 1496 set_decoder(decoder) / get_decoder() L1538 L1541 1497 forward(...) -> CausalLMOutputWithPast L1548 1498 prepare_inputs_for_generation(input_ids, past_key_values, 1499 attention_mask, inputs_embeds, **kwargs) L1642 1500 @staticmethod _reorder_cache(past_key_values, beam_idx) L1707