Byte-Pair Encoding tokenizer for training LLMs on large datasets
Byte-Pair Encoding tokenizer for training large language models on huge datasets.
오픈소스
반응 (1)
비슷한 프로덕트
설명이 비슷한 순
Tokensift
github.coman open-sourced token-efficiency linter for LLM prompts
Efficient and general syntactical decoding for Large Language Models
LLaMArch
gpavanb1.github.ioMinimal, Building Blocks for Large Language Models
A surprisingly effective way to predict token importance in LLM prompts
heatmap.demos.watchful.ioWe explored a novel method to gauge the significance of tokens in prompts given to large language models, without needing direct model access.
LLM Token Visualizer
v0-llm-token-visualizer.vercel.appHow big is 128k token input