Tokenization is the hidden tax on every LLM call you make. Every time you send a prompt, the model has to split your text into tokens before it can do any real work. This step normally costs milliseconds per request, which adds up fast when you are processing millions of inputs.
GigaToken just showed up with a 1,000x speedup and a drop-in replacement API. It hit 1,500 GitHub stars in its first week on the trending page, and for good reason. The benchmark numbers are hard to ignore: 8,327 MB/s on a standard EPYC CPU for GPT-2 tokenization, compared to HuggingFace tokenizers at roughly 6 MB/s.
This is not a toy benchmark. Marcel Roed, the creator, published a paper alongside the repo showing how SIMD optimizations and a multi-level cache hierarchy deliver real throughput at scale. At the fastest rates measured, you could tokenize the entire Common Crawl dataset (130 trillion tokens) in under 6.5 hours on a single server.
Why Tokenization Matters More Than You Think
Most founders building on LLMs never profile their tokenization pipeline. They install HuggingFace tokenizers, call encode_batch, and move on. But tokenization is where a surprising amount of latency hides, especially in high-throughput systems like batch inference pipelines, dataset preprocessing, and real-time streaming apps.
Here is what GigaToken does differently:
- SIMD-optimized pretokenization. Instead of outsourcing string splitting to a regex engine, GigaToken uses CPU SIMD instructions to process text in parallel. This is where most of the speedup comes from.
- Smart pretoken caching. If a word has been seen before, the encoded tokens are looked up from a cache. The challenge is that the cache grows fast and pretoken distributions are long-tailed. GigaToken uses a custom cache hierarchy that does not degrade under real-world workloads.
- File-level parallelism. The native API reads files directly and splits work across cores without the overhead of Python-level batching.
The compatibility story is where this gets practical. You can wrap any HuggingFace tokenizer with gt.Tokenizer(hf_tokenizer).as_hf() and keep using your existing code. The outputs match exactly. In compatibility mode, you lose some of the 1,000x gain but still get way faster performance across the board.
The Numbers That Matter
On an AMD EPYC CPU with the GPT-2 tokenizer, GigaToken hit 8,327 MB/s. HuggingFace tokenizers managed 6.15 MB/s. That is a 1,353x speed difference on the same hardware. On an ARM-based server, the gap was still 989x.
The native GigaToken API (skipping compatibility mode and calling encode_files directly) is the fastest path. It reads raw file data in the Rust layer, avoids Python serialization overhead, and parallelizes automatically. For batch preprocessing jobs, this is the mode you want.
The only current blind spot is SentencePiece-based tokenizers. Models that use SentencePiece (mostly Google models and BERT-style architectures) are not well optimized in GigaToken yet. The creator acknowledges this is low priority since BPE tokenizers dominate the current landscape.
What This Means for Founders
If you run batch inference or preprocess training data at scale, GigaToken can cut one of your hidden costs by three orders of magnitude. Here is the concrete math:
- Training data preprocessing. If your pipeline tokenizes terabytes of text before training, GigaToken turns hours into seconds. A dataset that takes 6 hours to tokenize with HuggingFace takes roughly 20 seconds with GigaToken's native API.
- Batch inference pipelines. Each request still needs tokenization. At high throughput (thousands of requests per second), the tokenization layer becomes a real bottleneck. Replacing it is a simple import swap.
- Streaming and real-time apps. Lower tokenization latency means lower end-to-end P50 for every API call. For apps where every millisecond matters (voice agents, real-time translation, code completion), this is free performance.
The install is a single pip command. pip install gigatoken. No Rust compiler needed, no build step. The package ships prebuilt wheels. If you want to test without installing, uvx --with tokenizers gigatoken bench 'openai-community/gpt2' runs a benchmark on the fly.
One caution. GigaToken is new. The repo is under active development, and the GitHub Issues list shows a few edge cases with exotic tokenizers and unicode normalization. Do not drop it into production without validating that your specific tokenizer produces identical outputs. The README explicitly warns that compatibility mode trades some speed for exact output matching.
Bottom line. Tokenization has been an afterthought in the LLM stack for too long. GigaToken proves that most of the overhead was an implementation problem, not a fundamental bottleneck. If you process text at scale, this is a no-brainer upgrade.