OS Megakernel that match M5 Max Tok/w at 2x the Throughput on RTX 3090
LLM speculative inference server for heterogeneous hardware & consumer GPUs. On an RTX 3090 power-limited to 220W: - 411 tok/s vs 229 tok/s on M5 Max (1.8x) - 1.87 tok/J, beating M5 Max efficiency - 1.55x faster decode than llama.cpp on the same GPU - 3.4x faster prefill The RTX 3090 launched in 2020.
Open source
Reactions (1)
Similar products
by description similarity
High-efficiency LLM inference engine in C++/CUDA. Run Llama 70B on RTX 3090.
LLMKube
github.comKubernetes for Local LLMs with GPU Acceleration
LLM Inference Calculator
llm-inference-calculator-delta.vercel.appEstimate VRAM, Latency, and Throughput
I run 30B 22tok/s
github.com109tok/s not novel,6GB/16GB RAM overcoming llama.cpp
Can your GPU run this LLM?
github.comCalculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization