Lightweight Llama3 Inference Engine
CUDA C. Lightweight Llama 3 8B Inference Engine in CUDA C. I recently wanted to explore my newly founded interest in CUDA programming and my passion for machine learning.
Open source
Reactions (1)
Similar products
by description similarity
High-efficiency LLM inference engine in C++/CUDA. Run Llama 70B on RTX 3090.
Tiny-vLLM
github.comhigh performance LLM inference engine in C++ and CUDA
A high-performance inference engine for AI models
Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.
Tensor library & inference framework for machine learning