← Latest products

Example POC to run the Dolly V2 12B model in 8bit mode

This repo loads the Dolly V2 12b model (databricks/dolly-v2-12b) using the `transformers` library. The code loads it in 8-bit quantized mode. I banged this out today because nobody seems to have figured this out yet (and I wanted to play with this model).

LibraryOpen source

26 stars · GitHubexternal site

Visit github.comexternal site

Reactions (1)

Similar products

by description similarity

  1. OpenGraviton

    opengraviton.github.io

    Run 500B+ parameter models on a consumer Mac Mini

    Hacker News13 pts

  2. Lance

    github.com

    Alternative to Parquet for ML data

    Hacker News85 pts

  3. Run 500B+ Parameter LLMs Locally on a Mac Mini

    github.com

    By combining 1.58-bit ternary quantization, dynamic sparsity with Top-K pruning and MoE routing, and mmap-based layer streaming, OpenGraviton can run models far larger than your system RAM—even on a Mac Mini.

    Hacker News17 pts

  4. RISCY-V02

    github.com

    A 16-bit 2-cycle RISC-V-ish CPU in the 6502 footprint

    Hacker News7 pts

  5. I ran a language model on a PS2

    github.com

    Running a large language model on a PlayStation 2

    Hacker News46 pts