KTransformers:671B DeepSeek-R1 on a Single Machine-286 tokens/s Prefill
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations. We are excited to share the new version of KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations!
LibraryOpen source
Reactions (2)
Similar products
by description similarity
Tiny-vLLM
github.comhigh performance LLM inference engine in C++ and CUDA
Free Inference Engineer and Model Training Roadmap
inferquest.orgFree, gamified roadmaps for LLM engineering: an Inference Engineering path (KV caches, CUDA kernels, production vLLM serving) and a Model Training path (pretraining on a budget, scaling laws, SFT/DPO/GRPO) — 185 tasks with auto-verified milestones instead of a paper certificate.
OpenLake is a high performance storage engine for efficient LLM inference and GPU Training
Speeding up LLM inference 2x times (possibly)
asciinema.orgHere's a project I've been working on for the last few months.
80% faster
github.com50% less memory, 0% loss of accuracy Llama finetuning