Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers

Article automatically generated from technical news.

Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s on a 144-core AMD EPYC 9565, against 24.8 MB/s for HuggingFace tokenizers and 36.0 MB/s for tiktoken on the same machine Both baselines are multithreaded Rust implementations. The difference comes from how the work is structured, not the language. Pretokenization without a regex engine Most tokenizers delegate pretokenization to a regex engine. Gigatoken implements it directly: → A 256-byte lookup ta