← 최신 프로덕트

KVSplit

Run 2-3x longer contexts on Apple Silicon. Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support. I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities.

오픈소스

361 스타 · GitHub외부 사이트

github.com 방문외부 사이트

반응 (1)

비슷한 프로덕트

설명이 비슷한 순

  1. Run 500B+ Parameter LLMs Locally on a Mac Mini

    github.com

    By combining 1.58-bit ternary quantization, dynamic sparsity with Top-K pruning and MoE routing, and mmap-based layer streaming, OpenGraviton can run models far larger than your system RAM—even on a Mac Mini.

    Hacker News17점

  2. OpenGraviton

    opengraviton.github.io

    Run 500B+ parameter models on a consumer Mac Mini

    Hacker News13점

  3. Cuts Long Horizon Inference Costs by 50% via external KV Cache Offload

    github.com

    OpenLake is a high performance storage engine for efficient LLM inference and GPU Training

    Hacker News22점

  4. 80% faster

    github.com

    50% less memory, 0% loss of accuracy Llama finetuning

    Hacker News385점

  5. MemStitch

    github.com

    Zero-copy context bridging for vLLM (25x TTFT speedup)

    Hacker News12점