← Trending

Tangerine Index lists products launched since ChatGPT and people's reactions to them.

tokenizers-cpp

github.com

Universal cross-platform tokenizers binding to HF and sentencepiece

Open source

Visit github.comexternal siteGitHub513 starsexternal site

Reactions (1)

Similar products

by description similarity

  1. Tokensift

    an open-sourced token-efficiency linter for LLM prompts

    Hacker News6 ptsGitHub19 starsgithub.comCLIOpen source2 reactionslatest reaction

  2. Steerling-8B

    a language model that can explain any token it generates

    Hacker News328 ptsguidelabs.ailatest reaction

  3. Byte-Pair Encoding tokenizer for training LLMs on large datasets

    Byte-Pair Encoding tokenizer for training large language models on huge datasets

    Hacker News5 ptsGitHub6 starsgithub.comOpen source2 reactionslatest reaction

  4. S3Tokenizer

    Reverse Engineering of Supervised Semantic Speech Tokenizer (S3Tokenizer) proposed in CosyVoice

    GitHub535 starsgithub.comOpen sourcelatest reaction

  5. magvit2-pytorch

    Implementation of MagViT2 Tokenizer in Pytorch

    GitHub669 starsgithub.comOpen sourcelatest reaction