← 최신 프로덕트

LLMKube

Kubernetes for Local LLMs with GPU Acceleration. Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed. I built LLMKube, a Kubernetes operator for deploying GPU-accelerated LLMs in production.

셀프 호스팅오픈소스

249 스타 · GitHub외부 사이트

github.com 방문외부 사이트

반응 (1)

비슷한 프로덕트

설명이 비슷한 순

  1. Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPU

    github.com

    High-efficiency LLM inference engine in C++/CUDA. Run Llama 70B on RTX 3090.

    Hacker News395점

  2. 50+ LLMs on 2 GPUs with 2-Second Swapping? We built AI-Native Runtime

    github.com

    InferX: Inference as a Service Platform

    Hacker News5점

  3. Tiny-vLLM

    github.com

    high performance LLM inference engine in C++ and CUDA

    Hacker News205점

  4. LLM

    github.com

    a Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)

    Hacker News45점

  5. Can your GPU run this LLM?

    github.com

    Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization

    Hacker News15점