LLMKube
Kubernetes for Local LLMs with GPU Acceleration. Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed. I built LLMKube, a Kubernetes operator for deploying GPU-accelerated LLMs in production.
Self-hostedOpen source
Reactions (1)
Similar products
by description similarity
High-efficiency LLM inference engine in C++/CUDA. Run Llama 70B on RTX 3090.
InferX: Inference as a Service Platform
Tiny-vLLM
github.comhigh performance LLM inference engine in C++ and CUDA
LLM
github.coma Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)
Can your GPU run this LLM?
github.comCalculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization