← 최신 프로덕트

KV-psi

using Linux PSI to to trim an LLM KV cache. Use Linux Pressure Stall Information to trim an LLM KV cache. I thought it'd be interesting to use Linux PSI (Pressure Stall Information) for an LLM runtime to trim the KV cache.

오픈소스

5 스타 · GitHub외부 사이트

github.com 방문외부 사이트

반응 (1)

비슷한 프로덕트

설명이 비슷한 순

  1. KVSplit

    github.com

    Run 2-3x longer contexts on Apple Silicon

    Hacker News272점

  2. Lowfat

    github.com

    pluggable CLI filter that saved 91.8% of my LLM tokens

    Hacker News156점

  3. Cuts Long Horizon Inference Costs by 50% via external KV Cache Offload

    github.com

    OpenLake is a high performance storage engine for efficient LLM inference and GPU Training

    Hacker News22점

  4. Cachet

    github.com

    A drop-in semantic cache for LLM APIs, 100% local, in Rust

    Hacker News5점

  5. Llmbuffer

    github.com

    Python library for cache-optimized LLM conversation history

    Hacker News7점