Tiny-vLLM
high performance LLM inference engine in C++ and CUDA. Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM.
Open source
Reactions (1)
Similar products
by description similarity
Free Inference Engineer and Model Training Roadmap
inferquest.orgFree, gamified roadmaps for LLM engineering: an Inference Engineering path (KV caches, CUDA kernels, production vLLM serving) and a Model Training path (pretraining on a budget, scaling laws, SFT/DPO/GRPO) — 185 tasks with auto-verified milestones instead of a paper certificate.
Lightweight Llama3 Inference Engine
github.comCUDA C
OpenLake is a high performance storage engine for efficient LLM inference and GPU Training
Tensor library & inference framework for machine learning
LLM
github.coma Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)