IST-DASLab/marlin

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

LLM & foundation modelsPython Apache-2.0 98created 2024-01-17

Stars

1,175

1-day growth

—

7-day growth

—

30-day growth

—

Star history

Tracked since 2026-11-13

#4bit#kernel#llm#quantization

Top contributors

Not collected yet (contributors job).

IST-DASLab/marlin · AI Repo Stats