← Latest products

Rust BPE tokenizer for Qwen models that's 12x faster than HuggingFace

A blazing-fast BPE tokenizer for Qwen models built with Rust.

Open source

29 stars · GitHubexternal site

Visit github.comexternal site

Reactions (1)

Similar products

by description similarity

  1. FlashTokenizer

    github.com

    10x faster C++ tokenizer for Python

    Hacker News5 pts

  2. Rust Based Per-Token Late Interaction Dense Search

    github.com

    High-Performance Engine for Multi-Vector Search

    Hacker News8 pts

  3. I embedded 685M public texts in 32 minutes (on 8x A100

    github.com

    Rust, TensorRT)

    Hacker News7 pts

  4. Witchcraft and Pickbrain

    github.com

    fast multi-vector semantic search in Rust

    Hacker News6 pts

  5. Extractous

    github.com

    Rust based data extraction ~25x faster than unstructured-io

    Hacker News6 pts