โ† ์ตœ์‹  ํ”„๋กœ๋•ํŠธ

Open-source load balancer for llama.cpp

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale ๐Ÿ“๐Ÿฆ™ Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU. Stateful load balancer customized for llama.cpp (with a reverse proxy).

์…€ํ”„ ํ˜ธ์ŠคํŒ…์˜คํ”ˆ์†Œ์Šค

1,685 ์Šคํƒ€ ยท GitHub์™ธ๋ถ€ ์‚ฌ์ดํŠธ

github.com ๋ฐฉ๋ฌธ์™ธ๋ถ€ ์‚ฌ์ดํŠธ

๋ฐ˜์‘ (1)

๋น„์Šทํ•œ ํ”„๋กœ๋•ํŠธ

์„ค๋ช…์ด ๋น„์Šทํ•œ ์ˆœ

  1. LLM

    github.com

    a Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)

    Hacker News45์ 

  2. Llama.go

    github.com

    port of llama.cpp to pure Go

    Hacker News25์ 

  3. STT โ€“> LLM โ€“> TTS pipeline in C

    github.com

    Pure C wrapper library to use llama.cpp with Linux and Windows as simple as possible.

    Hacker News11์ 

  4. Open-source LLM provider price comparison

    github.com

    This project collects GPU benchmarks from various cloud providers and compares them to fixed per token costs. Use our tool for efficient LLM GPU selections and cost-effective AI models. LLM provider price comparison, gpu benchmarks to price per token calculation, gpu benchmark table

    Hacker News125์ 

  5. A tool to benchmark LLM APIs (OpenAI

    llmapitest.com

    Claude, local/self-hosted)

    Hacker News55์ 