← 최신 프로덕트

OS Megakernel that match M5 Max Tok/w at 2x the Throughput on RTX 3090

LLM speculative inference server for heterogeneous hardware & consumer GPUs. On an RTX 3090 power-limited to 220W: - 411 tok/s vs 229 tok/s on M5 Max (1.8x) - 1.87 tok/J, beating M5 Max efficiency - 1.55x faster decode than llama.cpp on the same GPU - 3.4x faster prefill The RTX 3090 launched in 2020.

오픈소스

2,973 스타 · GitHub외부 사이트

github.com 방문외부 사이트

반응 (1)

비슷한 프로덕트

설명이 비슷한 순

  1. Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPU

    github.com

    High-efficiency LLM inference engine in C++/CUDA. Run Llama 70B on RTX 3090.

    Hacker News395점

  2. LLMKube

    github.com

    Kubernetes for Local LLMs with GPU Acceleration

    Hacker News5점

  3. LLM Inference Calculator

    llm-inference-calculator-delta.vercel.app

    Estimate VRAM, Latency, and Throughput

    Hacker News6점

  4. I run 30B 22tok/s, 109tok/s not novel,6GB/16GB RAM overcoming llama.cpp

    github.com

    Run a 110B on a 2016 PC with 16 GB RAM. Know your tok/s before you download. Placement beats budget: predicts speed + memory fit for any GGUF on your exact hardware, self-calibrates, emits the exact llama.cpp command — or 'quantprobe auto' does it all. Falsification-tested laws; misses published at full size. pip install quantprobe

    Hacker News5점

  5. Can your GPU run this LLM?

    github.com

    Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization

    Hacker News15점