Docker Model Runner Integrates vLLM for High-Throughput Inference
Docker Model Runner. We’ve updated Docker Model Runner to support vLLM alongside the existing llama.cpp backend.
오픈소스
반응 (3)
비슷한 프로덕트
설명이 비슷한 순
Ollama
github.comRun LLMs on your Mac
Open-source load balancer for llama.cpp
github.comOpen-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
LLMKube
github.comKubernetes for Local LLMs with GPU Acceleration
Tiny-vLLM
github.comhigh performance LLM inference engine in C++ and CUDA
Pantry
github.comBrew/Docker for LLMs