Byte-Pair Encoding tokenizer for training LLMs on large datasets
Byte-Pair Encoding tokenizer for training large language models on huge datasets.
Open source
Reactions (1)
Similar products
by description similarity
Tokensift
github.coman open-sourced token-efficiency linter for LLM prompts
Efficient and general syntactical decoding for Large Language Models
LLaMArch
gpavanb1.github.ioMinimal, Building Blocks for Large Language Models
A surprisingly effective way to predict token importance in LLM prompts
heatmap.demos.watchful.ioWe explored a novel method to gauge the significance of tokens in prompts given to large language models, without needing direct model access.
LLM Token Visualizer
v0-llm-token-visualizer.vercel.appHow big is 128k token input