Distributed Llama
Run LLMs on multiple devices in parallel. Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Open source
Reactions (1)
Similar products
by description similarity
A high-performance RDMA distributed file system for fast LLM Inference and GPU Training.
Speeding up LLM inference 2x times (possibly)
asciinema.orgHere's a project I've been working on for the last few months.
LLM
github.coma Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)
Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.
Reame
github.coma CPU inference server that gets faster as it runs