Measured, per sector

Results

Each lexicon is trained on one sector's traffic and measured on held-out text from that same sector, split by content hash, with the round trip verified lossless. Eleven sectors, every one of them trained on a matched corpus of 50 MB or more.

Token reduction on held-out traffic

Against tiktoken's o200k_base, the tokenizer behind GPT-4o. Higher is fewer tokens for the same bytes.

1.0×1.5×2.0×2.5×3.0×

Agentic & reasoning traffic Technical & document domains

What the reduction is worth

Two layers, kept deliberately separate: token counts are measured; dollars are that measurement times a published rate you can substitute for your own.

Per terabyte of traffic, at $3.00 per million tokens.
Sector	Reduction	Tokens saved	Value / TB

Scale tiers multiply directly: 1 TB mid-size deployment, 10 TB large enterprise, 100 TB platform, 1,000 TB hyperscale. Both layers hold whether you are paying an API invoice or your own prefill compute.

The same window, carrying more

A context window is counted in tokens, not in meaning. Fewer tokens for the same material is a larger effective window at no extra cost.

Shown at the measured 2.916× agentic ratio. On a general corpus the same arithmetic gives 1.527×; tuned to a single customer's own traffic it reaches 3.14×.

Measured against whatever you already run

Six production tokenizers, four families, one held-out corpus. The comparison does not depend on which vendor you are on.

Published figures on this site are quoted against GPT-4o's o200k_base, which is the most token-efficient of the baselines tested — so it is the hardest comparison available. Against every other tokenizer measured, the same lexicon does better.

One sector's held-out corpus, encoded by each tokenizer. Higher means we need proportionally fewer tokens for identical text.
Tokenizer	Family	Tokens for the same text	We need
p50k_base	OpenAI	1,158,821	2.03× fewer
Qwen3	Open weights	1,101,025	1.93× fewer
cl100k_base	OpenAI	1,094,765	1.92× fewer
SmolLM3	Open weights	1,093,745	1.92× fewer
OLMoE	Open weights	1,087,037	1.91× fewer
o200k_base	OpenAI — GPT-4o	1,082,031	1.90× fewer

The spread across four tokenizer families is 1.90× to 2.03× — a range of 7%. Tokenizer choice moves the result far less than domain does, which is why the engagement measures your corpus rather than your vendor.

How to read these figures

Compression and privacyA tuned vocabulary changes what a request costs and who can read the token stream. Every claim here is sized to what has been measured.

Tuned versus generalThe 3.14× case is a lexicon tuned to one customer's own traffic; 1.53× is a general-purpose build across 26 domains. We quote whichever matches the traffic in question.

Your corpus sets your numberGains vary by domain, so the figure that matters is the one measured on your own traffic — and that measurement comes before any commitment.

Provenance travels with each rowTraining corpus size is published beside every sector result. Reasoning & agent traces is measured on a corpus assembled from public chain-of-thought datasets.