Cartesia's H-Nets Drop Fixed Tokenizers for Raw Bytes

Language models learn almost everything from data—except tokenization, which is still a hand-built preprocess. Cartesia Research has published new results on Hierarchical Networks (H-Nets): move that segmentation inside the model and learn hierarchy straight from raw bytes.

From a fixed vocabulary to chunking on the fly

An H-Net compresses data with learned dynamic chunking. An encoder reads bytes, a routing step groups them into coarser chunks, a main network models those chunks, and a decoder returns to the original resolution. Stack another H-Net as the main network and you get two stages of hierarchy. In Cartesia's framing, that replaces offline BPE merges with end-to-end, content-aware units.

In carefully matched scaling ladders—changing only the tokenization path of a BPE Transformer into a one-stage byte H-Net—the company reports about a 1.28× compute gain on validation loss and larger gains on downstream suites, especially on code and out-of-distribution languages.

What “about 10× less data” actually means

The headline comparison is a larger two-stage run against Hugging Face's SmolLM3 (~3B), following its public recipe and matching FLOPs per byte as closely as possible. Cartesia says the byte-level H-Net reached SmolLM3's final average score after roughly 1T tokens, versus about 11T for the baseline—on the order of a tenfold cut in data and compute. Parameter counts were not identical (about 3.67B vs 3.08B), so the claim is efficiency under a compute-per-byte match, not a same-size bake-off.

Qualitatively, the models show context-dependent chunks: multi-word English phrases, whitespace-aware code spans, and in-context tokenization where a novel word is split at first and later compressed as it recurs. None of that comes from a hand-written vocab rule.

Beyond text: DNA, images, audio

The same idea extends to other “atoms”: nucleotides for DNA, where Cartesia reports multi-fold compute gains over plain Transformers; multi-dimensional H-Nets in pixel-space diffusion that discover unsupervised semantic chunks; and audio, a long-standing Cartesia focus where learning phonemes from waveforms is still a bottleneck.

This is a research release, not a consumer chat model you can open tomorrow. Cartesia says it plans to open more of the architecture over time. For teams chasing pretraining efficiency, hard-to-tokenize domains like code and Chinese, or multimodal models that start from raw signals, H-Nets push the bitter lesson one step further: even the tokenizer can be learned at scale.