The proposed substitution
When a routed expert and use_exact_generators are both enabled, the modified MLP instantiates PackedInt4Linear for its gate, up and down projections. Shared experts and dense layers otherwise keep ordinary linear modules. The to_exact path calls ExactLinear.from_weight with four-bit or eight-bit choices and a group size of 32.
The preserved forward form
The MLP still multiplies its activated gate projection by its up projection and feeds that product through a down projection. Preserving this high-level form does not by itself prove equality of the floating outputs: activation quantization, scales, rounding and accumulator bounds also matter.
The claim and the missing implementation
A code docstring claims roughly 950-fold greater precision and a smaller representation, while retaining attention/router/output computations in BF16. The underlying exact_generators.py and complete exactness-test files are named in the file tree but their bodies are not included as FILE sections. This record establishes the adaptation hooks and claim; it does not supply an independently reproducible proof of that precision factor or the missing class implementation.
Equations in this build
2 linked mathematical records. Each opens its notation, interpretation and source.
convert_kimi_to_exact: n
In convert_kimi_to_exact, this statement defines or updates n. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…
convert_kimi_to_exact: n
In convert_kimi_to_exact, this statement defines or updates n. This statement fixes a numeric parameter or initializes a numerical state in this source version; it is not a…
Sources & provenance
OUR ADDITIONS (kimi-blanket work, dropped into the model dir — NOT official) exact_generators.py ExactLinear + PackedInt4Linear (int64 exact int4 MAC) load_exact.py convert_kimi_to_exact loader test_exact_expert.py / test_exact_native.py our int4-exactness tests activation_geometry.py measure_compaction.py our analysis scripts __pycache__/modeling_kimi_k25.cpython-314.pyc (byte-compiled cache)
# ROUTED EXPERTS (the int4 intelligence bulk): when exact generators are on, the cells ARE # PackedInt4Linear whose buffers (weight_packed, weight_scale) match the shipped checkpoint keys — so the # native int4 loads straight in with zero conversion, never expands to bf16, and computes the MAC exactly. if is_routed_expert and getattr(config, "use_exact_generators", False): from exact_generators import PackedInt4Linear ab = getattr(config, "exact_act_bits", 16) self.gate_proj = PackedInt4Linear(self.hidden_size, self.intermediate_size, act_bits=ab) self.up_proj = PackedInt4Linear(self.hidden_size, self.intermediate_size, act_bits=ab) self.down_proj = PackedInt4Linear(self.intermediate_size, self.hidden_size, act_bits=ab) else: # shared expert + dense layer stay bf16 nn.Linear self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False) self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False) self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size,…