← Latest products

KV-psi

using Linux PSI to to trim an LLM KV cache. Use Linux Pressure Stall Information to trim an LLM KV cache. I thought it'd be interesting to use Linux PSI (Pressure Stall Information) for an LLM runtime to trim the KV cache.

Open source

5 stars · GitHubexternal site

Visit github.comexternal site

Reactions (1)

Similar products

by description similarity

  1. KVSplit

    github.com

    Run 2-3x longer contexts on Apple Silicon

    Hacker News272 pts

  2. Lowfat

    github.com

    pluggable CLI filter that saved 91.8% of my LLM tokens

    Hacker News156 pts

  3. Cuts Long Horizon Inference Costs by 50% via external KV Cache Offload

    github.com

    OpenLake is a high performance storage engine for efficient LLM inference and GPU Training

    Hacker News22 pts

  4. Cachet

    github.com

    A drop-in semantic cache for LLM APIs, 100% local, in Rust

    Hacker News5 pts

  5. Llmbuffer

    github.com

    Python library for cache-optimized LLM conversation history

    Hacker News7 pts