FlashTokenizer
10x faster C++ tokenizer for Python. EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING. I built a tokenizer in C++ with a Python binding that outperforms HuggingFace tokenizers by 10x on large inputs.
Open source
Reactions (1)
Similar products
by description similarity
A blazing-fast BPE tokenizer for Qwen models built with Rust
Achilles
github.comAutomatic profiling and C++ optimization of Python via LLMs
Tiny-vLLM
github.comhigh performance LLM inference engine in C++ and CUDA
Tokensift
github.coman open-sourced token-efficiency linter for LLM prompts
TokenDagger
github.comA tokenizer faster than OpenAI's Tiktoken