microsoft/sarathi-serve

A low-latency & high-throughput serving engine for LLMs

LLM & foundation modelsPython Apache-2.0 68created 2023-11-02

Stars

536

1-day growth

7-day growth

—

30-day growth

—

Star history

Tracked since 2026-11-13

#llama#llm-inference#pytorch#transformer

Top contributors

Not collected yet (contributors job).

microsoft/sarathi-serve · AI Repo Stats