Docker Model Runner Integrates vLLM for High-Throughput Inference
Docker Model Runner. We’ve updated Docker Model Runner to support vLLM alongside the existing llama.cpp backend.
Open source
Reactions (3)
Similar products
by description similarity
Ollama
github.comRun LLMs on your Mac
Open-source load balancer for llama.cpp
github.comOpen-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
LLMKube
github.comKubernetes for Local LLMs with GPU Acceleration
Tiny-vLLM
github.comhigh performance LLM inference engine in C++ and CUDA
Pantry
github.comBrew/Docker for LLMs