Inkling 2 Small: Native 4-Bit Forward Pass From Pretraining to RL

Thinking Machines just described Inkling 2 Small as the largest model it knows of that was trained end-to-end with a native 4-bit forward pass. The same numeric recipe runs from the first step of pretraining through the final RL rollout, so a quantized checkpoint can ship without a calibration stage. On its largest jobs, the recipe is 8% faster in pretraining wall-clock and 20% faster in RL than full MXFP8 training. The closing number is sharper still: a few months ago, a run of similar size took several weeks; Inkling 2 Small was pretrained in 68 hours on GB300 NVL72 racks.

A few misreads to kill immediately. This is not a new model. Inkling 2 Small itself shipped on November 6. It is also not "4-bit everywhere." Expert-layer forward passes run in NVFP4; backward passes are re-quantized with stochastic rounding and computed in MXFP8. NVFP4 is not a clean 4 bits either: with a shared E4M3 scale over each block of 16 values, it averages 4.5 bits. And the 68 hours should not be credited to 4-bit math alone. The post is explicit that the numerics recipe is only one piece of a larger stack of science and engineering improvements.

Against BF16, NVFP4 stores about 3.6× fewer bits and can run up to 6× faster on GB300 matrix multiplies: 15 PFLOPS for NVFP4, 5 for MXFP8, 2.5 for BF16. The speed-versus-error tradeoff has to be scored by stage. Pretraining and RL sampling are the high-volume steps, where speed and memory both matter. Backward updates average randomly distributed rounding error across a batch, but they are highly sensitive to quantization bias that compounds across layers, samples, and steps. Full NVFP4 in the expert layers produced an unacceptable hit to compute efficiency (CE). Splitting the effect showed that NVFP4 in the forward pass, with a different treatment on the backward pass, was the tradeoff they would accept.

Why the backward pass cannot reuse the forward blocks

NVFP4 scales consecutive blocks of 16 values. In the forward pass, quantized activations and weights are multiplied in those row blocks. On the backward pass the tensors are transposed, so each value is grouped with 15 new neighbors, and both the scale factors and the quantized values change. NVIDIA's published approach computes one scale over each 16×16 tile so the representation is bit-identical after a transpose. Thinking Machines tried that and found the tiles too coarse: a single outlier can flush the rest of the block to zero, and the CE hit was again too large. They needed 1×16 scales without letting the forward and backward tensors drift apart.

The fix was to de-quantize the saved forward tensors, transpose them in higher precision, and re-quantize with stochastic rounding (SR). SR rounds up or down in proportion to distance, so the backward values match the forward ones in expectation, and it is hardware-accelerated on Blackwell. The design draws on Quartet II, but instead of randomized rotations plus a bias-correction term, they apply elementwise SR to the de-quantized values and re-quantize weights and activations to MXFP8, then compute gradients with MXFP8 GEMMs. That is why this should not be described as end-to-end 4-bit training of both passes.

Once the GEMMs got faster, the bottleneck moved to expert-parallel communication. On a GB300 NVL72, NVLink is 900 GB/s in each direction and NVFP4 GEMMs reach 15 PFLOPS, which works out to about 16,000 floating-point operations in the time it takes to receive one byte. Tokens are dispatched in NVFP4. Combined outputs, though, need to stay in higher precision (BF16 by default); switching that path to MXFP8 hurt accuracy because of outliers. The working recipe is a 32-dimensional random Hadamard transform followed by block-scaled S2F6, a fixed-point format, cutting the combine payload from 16 bits per value to 8.25 with no measurable accuracy drop during training. Blackwell has PTX conversions for this format, and the team fused the transform and quantization into the epilogue of the w2 grouped GEMM.

Kernels, the last sixth of the network, and a 2% CE bill

Saturating NVFP4 depends on an in-house MoE fusion kernel stack, codenamed BBBK. The compute path uses the largest 2SM UMMA instruction unique to GB300 (K=96). ALU-heavy work such as scale-factor computation, re-quantization, stochastic rounding, and coalescing is kept from stalling the Tensor Cores through epilogue fusion, vectorization, and scheduling, so NVLink, the ALUs, and the Tensor Cores stay busy at once. In RL sampling, latency is pushed down to the floor set by NVLink and HBM.

The rest of the recipe is equally specific. The last 16.6% of layers stay in higher precision, computed in MXFP8 rather than BF16. NVIDIA's published recipe uses 15%; Thinking Machines scales layer count by a factor of 6 in its pretraining ladder, so it used one sixth. Activation quantization uses Four Over Six, which picks between two scale factors; weight quantization does not. Outer scales are per-expert for weights and per-token for activations, for efficiency, batch invariance, and no causality leakage. The same settings carried through SFT and RL without changes.

The pretraining bill is about 2% compute efficiency: the model needs roughly 2% more FLOPs to reach the loss it would have hit in full MXFP8. In RL the end-to-end speedup was substantial, with no discernible difference in reward. The lab says it is already exploring FP4 in the backward pass, and how the recipe will change on Vera Rubin.

The weights and benchmark numbers for Inkling 2 Small were already public on November 6. What landed today is the training stack: how to run a 594B-class MoE from scratch on Blackwell hardware that was built for NVFP4, with a 4-bit forward pass and an 8-bit backward pass, and how to take calibration out of the delivery path. The 68 hours are the wall-clock of many stacked improvements, not a format switch by itself. Without a numeric recipe that was consistent from step one, those kernel and communication optimizations would not have stacked the same way.