Open-source load balancer for llama.cpp
Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale ππ¦ Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU. Stateful load balancer customized for llama.cpp (with a reverse proxy).
Self-hostedOpen source
Reactions (1)
Similar products
by description similarity
LLM
github.coma Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)
Llama.go
github.comport of llama.cpp to pure Go
STT β> LLM β> TTS pipeline in C
github.comPure C wrapper library to use llama.cpp with Linux and Windows as simple as possible.
This project collects GPU benchmarks from various cloud providers and compares them to fixed per token costs. Use our tool for efficient LLM GPU selections and cost-effective AI models. LLM provider price comparison, gpu benchmarks to price per token calculation, gpu benchmark table
A tool to benchmark LLM APIs (OpenAI
llmapitest.comClaude, local/self-hosted)