Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPU
High-efficiency LLM inference engine in C++/CUDA. Run Llama 70B on RTX 3090.
Open source
Reactions (1)
Similar products
by description similarity
Lightweight Llama3 Inference Engine
github.comCUDA C
Can your GPU run this LLM?
github.comCalculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization
How to run Llama3.1 405B on a 8GB VRAM
Tiny-vLLM
github.comhigh performance LLM inference engine in C++ and CUDA
80% faster
github.com50% less memory, 0% loss of accuracy Llama finetuning