Architecture and routing
The source describes a 7,168-wide residual stream with three dense layers followed by 58 MoE layers, 256 routed experts, eight selected experts, one shared expert and a separately described MTP layer. Group-limited routing first selects four of eight groups using each group’s two strongest corrected scores. It then selects eight experts, weights them with pre-bias sigmoid scores, normalizes and scales the routed mixture by 2.5.
Quantized payload and reference code
The dossier describes FP8 E4M3 weights with 128×128 F32 block scales, while the embedded Python uses ordinary linear layers and assumes an appropriate loading/dequantization path. The recovered text is a reference dossier, not the hundreds of gigabytes of tensor bytes listed in its census.
Internal implementation discrepancies
The overview’s latent-only cache arithmetic does not describe the embedded eager path, which caches expanded keys and values. The overview’s YaRN ramp weighting is reversed if ramp denotes the actual helper output, and its FlashAttention description says query padding while the code pads the values. Separate equation pages preserve the source expression and the source-code correction.
Historical machine and resource plan
The source reports a memory-constrained WSL installation and explicitly says no end-to-end inference had run. Its 464 expert reads and roughly 19.04 GiB of per-token traffic are a no-reuse streaming estimate. The stated drive rates, CPU assumptions, memory limits and installation package status belong to the August scan and were not measured on the current workspace.
What this recovery establishes
Both embedded Python files parse as source, and the full dossier, formulas, dtype/shape claims and configuration remain readable. Current inference, model quality, model weights, local model-serving endpoints and actual runtime throughput have not been verified by this website audit.
Equations in this build
128 linked mathematical records. Each opens its notation, interpretation and source.
Residual addition around a transformed hidden state
Add the block input saved in residual to the output of the intervening transformation. Both operands must have the same residual-stream shape. The same expression appears around…
KimiK25ForConditionalGeneration.prepare_inputs_for_generation: position ids
In KimiK25ForConditionalGeneration.prepare_inputs_for_generation, this statement defines or updates position_ids. The addition or subtraction updates the named state or combines…
KimiK25ForConditionalGeneration.prepare_inputs_for_generation: input ids
In KimiK25ForConditionalGeneration.prepare_inputs_for_generation, this statement defines or updates input_ids. The addition or subtraction updates the named state or combines…
_get_unpad_data: seqlens in batch
In _get_unpad_data, this statement defines or updates seqlens_in_batch. The reduction aggregates values along the explicit source axis; preserving the axis and keep-dimension…
_get_unpad_data: max seqlen in batch
In _get_unpad_data, this statement defines or updates max_seqlen_in_batch. This statement evaluates the named projection, normalization or numeric transform with the source…
_get_unpad_data: cu seqlens
In _get_unpad_data, this statement defines or updates cu_seqlens. This statement evaluates the named projection, normalization or numeric transform with the source arguments. Its…
Solve for the rotary frequency index at a chosen rotation count
At index i, the unscaled angular frequency is base^(−2 i/dim). Over orig_max positions it completes orig_max*base^(−2 i/dim)/(2π) rotations. Setting that count equal to rot and…
yarn_find_correction_range: low
In yarn_find_correction_range, this statement defines or updates low. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…
yarn_find_correction_range: high
In yarn_find_correction_range, this statement defines or updates high. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…
yarn_find_correction_range: return
In yarn_find_correction_range, this statement returns the displayed result. The addition or subtraction updates the named state or combines residual contributions. The input…
Logarithmic magnitude adjustment for extended context
For scale>1, the adjustment adds 0.1*mscale*ln(scale) to one. The code returns one for scale≤1. One parameter specifies the context extension, while the other controls adjustment…
yarn_linear_ramp_mask: linear func
In yarn_linear_ramp_mask, this statement defines or updates linear_func. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape…
yarn_linear_ramp_mask: ramp func
In yarn_linear_ramp_mask, this statement defines or updates ramp_func. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…
rotate_half: x1
In rotate_half, this statement defines or updates x1. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention determine…
rotate_half: x2
In rotate_half, this statement defines or updates x2. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention determine…
apply_rotary_pos_emb: q
In apply_rotary_pos_emb, this statement defines or updates q. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…
apply_rotary_pos_emb: k
In apply_rotary_pos_emb, this statement defines or updates k. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape convention…
Rotate paired query coordinates using a position-dependent angle
rotate_half maps a pair of coordinate blocks(x 1,x 2) to(−x 2,x 1). Multiplication by cosine and addition of the sine-weighted rotated vector implements a two-dimensional rotation…
Rotate paired key coordinates using a position-dependent angle
rotate_half maps a pair of coordinate blocks(x 1,x 2) to(−x 2,x 1). Multiplication by cosine and addition of the sine-weighted rotated vector implements a two-dimensional rotation…
repeat_kv: return
In repeat_kv, this statement returns the displayed result. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called operation…
RMSNorm mean square, despite the local variable name
This expression averages squared components along the final feature axis and preserves that axis for broadcasting. It computes the second raw moment, not centered variance: the…
Normalize by the root mean square
Multiply each feature by 1/√(mean(x²)+epsilon). Positive epsilon prevents division by zero and changes exact unit-RMS normalization near the zero vector. Unlike LayerNorm, this…
Apply learned per-feature RMSNorm gains
After RMS scaling, multiply each component by its learned gain and restore the earlier input dtype. The gain allows the model to choose different output scales by feature; it does…
DeepSeek RotaryEmbedding.__init__: inv freq
In DeepseekV3RotaryEmbedding.__init__, this statement defines or updates inv_freq. This is part of the positional-phase calculation. Frequency, position, sine/cosine and…
DeepSeek RotaryEmbedding._set_cos_sin_cache: t
In DeepseekV3RotaryEmbedding._set_cos_sin_cache, this statement defines or updates t. This is part of the positional-phase calculation. Frequency, position, sine/cosine and…
DeepSeek RotaryEmbedding._set_cos_sin_cache: freqs
In DeepseekV3RotaryEmbedding._set_cos_sin_cache, this statement defines or updates freqs. This is part of the positional-phase calculation. Frequency, position, sine/cosine and…
DeepSeek LinearScalingRotaryEmbedding._set_cos_sin_cache: t
In DeepseekV3LinearScalingRotaryEmbedding._set_cos_sin_cache, this statement defines or updates t. The expression forms a quotient or integer division. The denominator, integer…
DeepSeek LinearScalingRotaryEmbedding._set_cos_sin_cache: freqs
In DeepseekV3LinearScalingRotaryEmbedding._set_cos_sin_cache, this statement defines or updates freqs. This is part of the positional-phase calculation. Frequency, position,…
yarn_get_mscale: return
In yarn_get_mscale, this statement returns the displayed result. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…
yarn_linear_ramp_mask: max
In yarn_linear_ramp_mask, this statement defines or updates max. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…
DeepSeek YarnRotaryEmbedding._set_cos_sin_cache: freq extra
In DeepseekV3YarnRotaryEmbedding._set_cos_sin_cache, this statement defines or updates freq_extra. The expression forms a quotient or integer division. The denominator, integer…
DeepSeek YarnRotaryEmbedding._set_cos_sin_cache: freq inter
In DeepseekV3YarnRotaryEmbedding._set_cos_sin_cache, this statement defines or updates freq_inter. The expression forms a quotient or integer division. The denominator, integer…
Complement the increasing YaRN frequency ramp
The helper produces a ramp increasing from zero to one across frequency-pair indices. Subtracting it from one makes inv_freq_mask select original frequencies at the small-index…
Blend original and divided YaRN frequency schedules
The source uses inv_freq_mask as the weight on the original schedule and its complement as the weight on the divided schedule. Since inv_freq_mask=1−ramp, this is…
DeepSeek YarnRotaryEmbedding._set_cos_sin_cache: t
In DeepseekV3YarnRotaryEmbedding._set_cos_sin_cache, this statement defines or updates t. This statement evaluates the named projection, normalization or numeric transform with…
DeepSeek YarnRotaryEmbedding._set_cos_sin_cache: freqs
In DeepseekV3YarnRotaryEmbedding._set_cos_sin_cache, this statement defines or updates freqs. This is part of the positional-phase calculation. Frequency, position, sine/cosine…
Ratio controlling the rotary cache magnitude
This coefficient divides the adjustment computed with mscale by the adjustment computed with mscale_all_dim. Equal strengths give a ratio of one. The attention initializer can…
SwiGLU-style gated feed-forward projection
Two learned maps project x into the intermediate width. The configured activation, SiLU in these dossiers, is applied to the gate projection; its result multiplies the…
MoEGate.forward: logits
In MoEGate.forward, this statement defines or updates logits. The matrix or indexed contraction combines the stated axes. Compatible tensor dimensions and the transposes in this…
MoEGate.forward: topk weight
In MoEGate.forward, this statement defines or updates topk_weight. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive…
DeepSeek MoE.moe_infer: tokens per expert
In DeepseekV3MoE.moe_infer, this statement defines or updates tokens_per_expert. The reduction aggregates values along the explicit source axis; preserving the axis and…
DeepSeek MoE.moe_infer: sorted tokens
In DeepseekV3MoE.moe_infer, this statement defines or updates sorted_tokens. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries…
DeepSeek MoE.moe_infer: start idx
In DeepseekV3MoE.moe_infer, this statement defines or updates start_idx. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is…
Sum the selected expert outputs with their routing weights
Each chosen expert evaluates the same input representation h. Its output vector is multiplied by the associated routing weight, and the selected vectors are added. The…
DeepSeek Attention.__init__: q head dim
In DeepseekV3Attention.__init__, this statement defines or updates self.q_head_dim. The addition or subtraction updates the named state or combines residual contributions. The…
DeepSeek Attention.__init__: kv a proj with mqa
In DeepseekV3Attention.__init__, this statement defines or updates self.kv_a_proj_with_mqa. The addition or subtraction updates the named state or combines residual contributions.…
DeepSeek Attention.__init__: kv b proj
In DeepseekV3Attention.__init__, this statement defines or updates self.kv_b_proj. The displayed multiplication combines the source factors. For tensors this is elementwise unless…
DeepSeek Attention.__init__: o proj
In DeepseekV3Attention.__init__, this statement defines or updates self.o_proj. The displayed multiplication combines the source factors. For tensors this is elementwise unless…
Baseline inverse-square-root attention coefficient
Initialize the score multiplier to 1/√d, where d is the total rotary plus non-rotary query-head width. This is the starting coefficient; the following YaRN branch can multiply it…
Expand the compressed latent into per-head keys and values
Normalize the compressed key/value latent, apply the learned up-projection, and reshape into heads. Each head receives its non-rotary key and value components. This eagerly…
DeepSeek Attention.forward: attn weights
In DeepseekV3Attention.forward, this statement defines or updates attn_weights. Softmax exponentiates and normalizes scores along the chosen axis, producing nonnegative weights…
Normalize each query’s attention scores
Apply softmax across key positions for each query and head. The resulting nonnegative weights sum to one when a row has at least one finite eligible score. The source computes…
DeepSeek Attention.forward: attn output
In DeepseekV3Attention.forward, this statement defines or updates attn_output. The matrix or indexed contraction combines the stated axes. Compatible tensor dimensions and the…
DeepSeek Attention.forward: attn output
In DeepseekV3Attention.forward, this statement defines or updates attn_output. The displayed multiplication combines the source factors. For tensors this is elementwise unless the…
DeepSeek Attention.forward: attn output
In DeepseekV3Attention.forward, this statement defines or updates attn_output. This statement evaluates the named projection, normalization or numeric transform with the source…
DeepSeek FlashAttention2.forward: attn output
In DeepseekV3FlashAttention2.forward, this statement defines or updates attn_output. The displayed multiplication combines the source factors. For tensors this is elementwise…
DeepSeek FlashAttention2._upad_input: key layer
In DeepseekV3FlashAttention2._upad_input, this statement defines or updates key_layer. The displayed multiplication combines the source factors. For tensors this is elementwise…
DeepSeek FlashAttention2._upad_input: value layer
In DeepseekV3FlashAttention2._upad_input, this statement defines or updates value_layer. The displayed multiplication combines the source factors. For tensors this is elementwise…
DeepSeek DecoderLayer.__init__: mlp
In DeepseekV3DecoderLayer.__init__, this statement defines or updates self.mlp. This statement evaluates the named projection, normalization or numeric transform with the source…
DeepSeek Model.forward: past key values length
In DeepseekV3Model.forward, this statement defines or updates past_key_values_length. This statement fixes a numeric parameter or initializes a numerical state in this source…
DeepSeek Model.forward: hidden states
In DeepseekV3Model.forward, this statement defines or updates hidden_states. This statement evaluates the named projection, normalization or numeric transform with the source…
DeepSeek ForSequenceClassification.forward: pooled logits
In DeepseekV3ForSequenceClassification.forward, this statement defines or updates pooled_logits. This statement evaluates the named projection, normalization or numeric transform…
DeepSeek DynamicNTKScalingRotaryEmbedding._set_cos_sin_cache: base
In DeepseekV3DynamicNTKScalingRotaryEmbedding._set_cos_sin_cache, this statement defines or updates base. The expression forms a quotient or integer division. The denominator,…
DeepSeek DynamicNTKScalingRotaryEmbedding._set_cos_sin_cache: inv freq
In DeepseekV3DynamicNTKScalingRotaryEmbedding._set_cos_sin_cache, this statement defines or updates inv_freq. This is part of the positional-phase calculation. Frequency,…
Independent sigmoid scores for the router
The sigmoid 1/(1+exp(−z)) is applied to each expert logit independently. It compresses each finite real logit into (0,1), without normalizing across experts. These values are…
Bias the choice of experts while retaining original affinities
An expert-specific correction vector is added to the sigmoid affinities for ranking. This addition can change which groups and experts are selected. The later mixture weights are…
MoEGate.forward: group scores
In MoEGate.forward, this statement defines or updates group_scores. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive…
MoEGate.forward: group idx
In MoEGate.forward, this statement defines or updates group_idx. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive the…
MoEGate.forward: score mask
In MoEGate.forward, this statement defines or updates score_mask. The expression forms a quotient or integer division. The denominator, integer rounding and tensor shape…
MoEGate.forward: ( , topk idx)
In MoEGate.forward, this statement defines or updates (_, topk_idx). The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive…
MoEGate.forward: denominator
In MoEGate.forward, this statement defines or updates denominator. The top-k operation selects the largest scores. Ties and group masking affect which experts or entries survive…
Normalize the selected routing affinities
Each gathered affinity is divided by the sum across the selected experts plus 10⁻²⁰. Ignoring that stabilizer, the selected weights sum to one. With the stabilizer retained, the…
DeepSeek MoE.__init__: experts per rank
In DeepseekV3MoE.__init__, this statement defines or updates self.experts_per_rank. The expression forms a quotient or integer division. The denominator, integer rounding and…
DeepSeek MoE.__init__: experts
In DeepseekV3MoE.__init__, this statement defines or updates self.experts. The displayed multiplication combines the source factors. For tensors this is elementwise unless the…
DeepSeek MoE.__init__: ep size
In DeepseekV3MoE.__init__, this statement defines or updates self.ep_size. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is…
DeepSeek MoE.__init__: ep rank
In DeepseekV3MoE.__init__, this statement defines or updates self.ep_rank. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is…
DeepSeek MoE.__init__: intermediate size
In DeepseekV3MoE.__init__, this statement defines or updates intermediate_size. The displayed multiplication combines the source factors. For tensors this is elementwise unless…
Add an always-active shared expert
The shared expert processes identity, the input representation saved before the routed expert computation. Its output is added to the weighted routed sum. It therefore contributes…
DeepSeek MoE.moe_infer: tokens per ep rank
In DeepseekV3MoE.moe_infer, this statement defines or updates tokens_per_ep_rank. The reduction aggregates values along the explicit source axis; preserving the axis and…
DeepSeek MoE.moe_infer: output splits
In DeepseekV3MoE.moe_infer, this statement defines or updates output_splits. The reduction aggregates values along the explicit source axis; preserving the axis and keep-dimension…
DeepSeek MoE.moe_infer: gathered tokens
In DeepseekV3MoE.moe_infer, this statement defines or updates gathered_tokens. The reduction aggregates values along the explicit source axis; preserving the axis and…
DeepSeek MoE.moe_infer: tokens per expert post gather
In DeepseekV3MoE.moe_infer, this statement defines or updates tokens_per_expert_post_gather. The reduction aggregates values along the explicit source axis; preserving the axis…
DeepSeek MoE.moe_infer: s
In DeepseekV3MoE.moe_infer, this statement defines or updates s. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…
DeepSeek MoE.moe_infer: end idx
In DeepseekV3MoE.moe_infer, this statement defines or updates end_idx. The addition or subtraction updates the named state or combines residual contributions. The input shapes and…
DeepSeek MoE.moe_infer: expert
In DeepseekV3MoE.moe_infer, this statement defines or updates expert. The displayed multiplication combines the source factors. For tensors this is elementwise unless the called…
DeepSeek Attention.__init__: q proj
In DeepseekV3Attention.__init__, this statement defines or updates self.q_proj. The displayed multiplication combines the source factors. For tensors this is elementwise unless…
DeepSeek Attention.__init__: q b proj
In DeepseekV3Attention.__init__, this statement defines or updates self.q_b_proj. The displayed multiplication combines the source factors. For tensors this is elementwise unless…
DeepSeek Attention.forward: q
In DeepseekV3Attention.forward, this statement defines or updates q. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…
DeepSeek Attention.forward: q
In DeepseekV3Attention.forward, this statement defines or updates q. This statement evaluates the named projection, normalization or numeric transform with the source arguments.…
DeepSeek Attention.forward: attn weights
In DeepseekV3Attention.forward, this statement defines or updates attn_weights. The addition or subtraction updates the named state or combines residual contributions. The input…
Pad value vectors to the query-head width
The FlashAttention path appends zero-valued components to V when the value width is smaller than the query width. With the displayed 192 and 128 widths, it appends 64 zeros. The…
DeepSeek FlashAttention2._upad_input: query layer
In DeepseekV3FlashAttention2._upad_input, this statement defines or updates query_layer. The displayed multiplication combines the source factors. For tensors this is elementwise…
DeepSeek Model.forward: position ids
In DeepseekV3Model.forward, this statement defines or updates position_ids. The addition or subtraction updates the named state or combines residual contributions. The input…
DeepSeek ForSequenceClassification.forward: output
In DeepseekV3ForSequenceClassification.forward, this statement defines or updates output. The addition or subtraction updates the named state or combines residual contributions.…
DeepSeek ForSequenceClassification.forward: return
In DeepseekV3ForSequenceClassification.forward, this statement returns the displayed result. The addition or subtraction updates the named state or combines residual…
DeepSeek MoE.moe_infer: gatherd idxs[s:s + k]
In DeepseekV3MoE.moe_infer, this statement defines or updates gatherd_idxs[s:s + k]. This statement evaluates the named projection, normalization or numeric transform with the…
Apply the additional YaRN score adjustment
Multiply the existing inverse-square-root attention coefficient by mscale². Here mscale is an evaluated logarithmic adjustment, not an independent second query dimension. The…
DeepSeek FlashAttention2._upad_input: max seqlen in batch q
In DeepseekV3FlashAttention2._upad_input, this statement defines or updates max_seqlen_in_batch_q. This statement fixes a numeric parameter or initializes a numerical state in…
DeepSeek FlashAttention2._upad_input: cu seqlens q
In DeepseekV3FlashAttention2._upad_input, this statement defines or updates cu_seqlens_q. The addition or subtraction updates the named state or combines residual contributions.…
DeepSeek ForSequenceClassification.forward: sequence lengths
In DeepseekV3ForSequenceClassification.forward, this statement defines or updates sequence_lengths. The addition or subtraction updates the named state or combines residual…
Decode an FP8 value and apply its block scale
The encoded byte first becomes its FP8 numerical value in float32, then is multiplied by the scale shared by its 128-by-128 block. The block indices mean floor(r/128) and…
Float32 linear expert-routing logits
Flattened token representations h are multiplied by the transpose of the gate weight matrix, giving one raw routing logit per expert. Both operands are cast to float32. None means…
Independent sigmoid scores for the router
The sigmoid 1/(1+exp(−z)) is applied to each expert logit independently. It compresses each finite real logit into (0,1), without normalizing across experts. These values are…
Bias the choice of experts while retaining original affinities
An expert-specific correction vector is added to the sigmoid affinities for ranking. This addition can change which groups and experts are selected. The later mixture weights are…
Score each group using its strongest two adjusted affinities
The 256 experts are arranged as eight groups of thirty-two. Within each token and group, take the two largest bias-adjusted scores and add them. The resulting eight group scores…
Keep the four highest-ranked expert groups
For each token, select the indices of the four largest group scores. This leaves four groups of thirty-two experts eligible for the next selection. The selection is a discrete…
Choose eight experts from the retained groups
The adjusted expert scores outside the selected groups are masked to negative infinity. Top-k then returns eight expert indices among the 128 eligible entries. The group…
Recover the original affinities of the selected experts
For each selected expert index, gather the corresponding value from the pre-bias sigmoid score array. Ranking used bias-adjusted values, but this weighting stage deliberately uses…
Normalize the selected routing affinities
Each gathered affinity is divided by the sum across the selected experts plus 10⁻²⁰. Ignoring that stabilizer, the selected weights sum to one. With the stabilizer retained, the…
Scale the routed branch by 2.5
After normalization, all selected mixture weights are multiplied by 2.5 in this DeepSeek configuration. Their sum is then approximately 2.5, so the routed contribution is…
Sum the selected expert outputs with their routing weights
Each chosen expert evaluates the same input representation h. Its output vector is multiplied by the associated routing weight, and the selected vectors are added. The…
Add an always-active shared expert
The shared expert processes identity, the input representation saved before the routed expert computation. Its output is added to the weighted routed sum. It therefore contributes…
Scaled query–key similarities with a rounded YaRN coefficient
The query–key matrix product forms similarities between each query position and key position. The prose uses 0.135237 as its attention multiplier. From the displayed exact…
Normalize each query’s attention scores
Apply softmax across key positions for each query and head. The resulting nonnegative weights sum to one when a row has at least one finite eligible score. The source computes…
Use attention weights to average value vectors
Matrix multiplication sums the value vector at each key position weighted by that query’s attention probability. A 128-component value vector per head gives a 128-component output…
Concatenate the heads and project to the residual width
The 128 heads contribute 128 value components each, giving 16,384 components per token. The output linear map then projects that concatenation into the 7,168-dimensional residual…
Latent-cache arithmetic and the embedded implementation mismatch
The arithmetic (512+64)×2=1152 bytes is correct for a cache storing one compressed key/value latent and one rotary key vector per token and layer at two bytes per element. The…
Solve for the rotary frequency index at a chosen rotation count
At index i, the unscaled angular frequency is base^(−2 i/dim). Over orig_max positions it completes orig_max*base^(−2 i/dim)/(2π) rotations. Setting that count equal to rot and…
Reduce the inverse-frequency schedule by the extension factor
The base rotary frequency is divided by factor. At a fixed token position, this reduces its phase advance by that factor. The later blend applies this scaled schedule toward the…
Retain the original inverse-frequency schedule
This is the original geometric rotary frequency schedule, base^(−2 i/dim). Its frequency decreases as i grows when base>1. The later blend retains the unscaled schedule at the…
YaRN prose blend with the ramp orientation reversed
If ramp is the increasing yarn_linear_ramp_mask defined in the embedded source, this prose reverses the blend. The body defines inv_freq_mask=1−ramp and returns…
Logarithmic magnitude adjustment for extended context
For scale>1, the adjustment adds 0.1*mscale*ln(scale) to one. The code returns one for scale≤1. One parameter specifies the context extension, while the other controls adjustment…
Evaluate the YaRN adjustment at factor forty
Substitution of scale=40 and adjustment strength 1 gives 1+0.1 ln 40=1.3688879454113936. The displayed 1.3688879 is a rounded form of that calculation. This is a deterministic…
Square the YaRN multiplier in the attention coefficient
The baseline coefficient is 1/√192. Multiplying it by the squared magnitude adjustment yields(1+0.1 ln 40)²/√192≈0.1352337788608801, a factor 1.8738542071 above the baseline. The…
Rotate paired query coordinates using a position-dependent angle
rotate_half maps a pair of coordinate blocks(x 1,x 2) to(−x 2,x 1). Multiplication by cosine and addition of the sine-weighted rotated vector implements a two-dimensional rotation…
Count routed expert evaluations along one token’s pass
Eight selected experts in each of fifty-eight MoE transformer layers give 464 routed expert evaluations for one token’s ordinary pass. The count excludes the dense prefix, shared…
A conditional total for routed expert weight traffic
Multiplying the dossier’s 44,050,944 bytes per expert by 464 expert evaluations gives 20,439,638,016 bytes, about 19.04 GiB. This is the stated one-load-per-evaluation accounting…
Convert a multiply–accumulate estimate into a FLOP estimate
Using the conventional count of one multiplication plus one addition per multiply–accumulate,36.52 GMAC corresponds to 73.04 GFLOP, rounded to 73.0. This is arithmetic applied to…
Sources & provenance
================================================================================ D E E P S E E K - R 1 Complete reference: architecture map, dataflow, file architecture, API surface, endpoints, and full source code as installed on MEGAMIND-AGI, 2026-08-05 ================================================================================ Model DeepSeek-R1 Architecture DeepseekV3ForCausalLM (deepseek_v3) Parameters 671B total / 37B activated per token [per official model card] Location /home/joseph/DeepSeek-R1 on MEGAMIND-AGI (WSL2 Ubuntu) Size 643 GB on disk / 641.29 GiB tensor payload / 163 shards Downloaded 2026-08-04 05:35-07:21 (huggingface_hub snapshot_download) Scanned 2026-08-05, live, by parsing all 163 safetensors file headers Paper arXiv:2501.12948 License MIT (weights and code; commercial use and distillation permitted) CONTENTS PART 1 MEGAMIND-AGI MACHINE SPECIFICATIONS PART 2 FILE ARCHITECTURE 2.1 Directory inventory 2.2 The safetensors container format 2.3 model.safetensors.index.json structure…
PART 1 MEGAMIND-AGI MACHINE SPECIFICATIONS [MEASURED] ================================================================================ IDENTITY Hostname MEGAMIND-AGI Chassis Dell Precision 3630 Tower Tailnet 100.69.120.61 Windows host 100.77.114.40 WSL2 guest (megamind-agi-wsl) Users WSL: joseph Windows: Josep CPU Model Intel Core i7-8700K @ 3.70 GHz Topology 1 socket / 6 cores / 12 threads (Hyper-Threading enabled) Cache L1d 192 KiB (6 inst) L1i 192 KiB (6 inst) L2 1.5 MiB (6 inst) L3 12 MiB (1 inst, shared) ISA PRESENT avx2 fma f16c popcnt sse4_2 aes bmi1 bmi2 rdseed adx movbe xsave xsavec xsaves clflushopt erms invpcid rtm hle ISA ABSENT avx512f avx512bw avx512vnni avx_vnni amx_int8 amx_bf16 -> no hardware fp8 or bf16 conversion instruction -> no VNNI int8 dot; int8 MAC needs vpmaddubsw + vpmaddwd -> 256-bit vectors are the widest available GPU Intel UHD 630 integrated graphics only. No discrete GPU installed. MEMORY…
PART 6 FP8 QUANTIZATION SCHEME [SPEC] ================================================================================ config.json quantization_config: { "quant_method": "fp8", "fmt": "e4m3", "weight_block_size": [128, 128], "activation_scheme": "dynamic" } FORMAT - FP8 E4M3 (torch float8_e4m3fn semantics) 1 sign bit | 4 exponent bits | 3 mantissa bits | 1 byte per element Exponent bias 7. No infinities. NaN encoded as 0x7F / 0xFF. Maximum finite magnitude 448.0. Subnormals present (exponent field 0). The i7-8700K has no hardware fp8 convert instruction; decode is a 256-entry lookup table indexed by the raw byte, built once at load. BLOCK DEQUANTIZATION For a weight tensor W of shape (R, C) with companion scale S of shape (ceil(R/128), ceil(C/128)): W_real[r][c] = fp8_to_f32( W[r][c] ) * S[ r/128 ][ c/128 ] The scale is named "weight_scale_inv" but is applied by multiplication in the standard fp8 kernels. Confirm the direction against a reference on a real cell rather than inferring it from the name. ACTIVATION QUANTIZATION IS DYNAMIC: activations are quantized at runtime from their own…
PART 13 FOOTPRINT: MEMORY, DISK I/O, COMPUTE [COMPUTED] ================================================================================ Derived from the measured tensor census. Batch 1, decode step, layers 0..60 (the MTP layer excluded). WEIGHTS THAT MUST BE RESIDENT (everything except the routed experts) --------------------------------------------------------------------------- component GiB basis --------------------------------------------------------------------------- attention, 5 projections x 61 layers 10.630 187,105,280 B per layer shared experts x 58 layers 2.379 3 x 2048 x 7168 fp8 dense MLP, layers 0-2 1.107 3 x 3 x 18432 x 7168 fp8 MoE router weights x 58 0.198 256 x 7168 BF16 e_score_correction_bias + all norms 0.002 F32 / BF16 embed_tokens 1.726 129280 x 7168 BF16 lm_head 1.726 129280 x 7168 BF16 --------------------------------------------------------------------------- TOTAL RESIDENT, native dtypes…
key_states = k_pe.new_empty(bsz, self.num_heads, q_len, self.q_head_dim) key_states[:, :, :, : self.qk_nope_head_dim] = k_nope key_states[:, :, :, self.qk_nope_head_dim :] = k_pe if past_key_value is not None: cache_kwargs = {"sin": sin, "cos": cos} # Specific to RoPE models key_states, value_states = past_key_value.update( key_states, value_states, self.layer_idx, cache_kwargs )