marcelroed/gigatoken

Language model tokenization at GB/s

LLM & foundation modelsRust MIT 226created 2025-11-10

Stars

4,155

1-day growth

—

7-day growth

—

30-day growth

—

Star history

Tracked since 2026-11-13

The chart appears after at least 2 days of data.

#llm#nlp#tokenization#tokenizer

Top contributors

Not collected yet (contributors job).

marcelroed/gigatoken · AI Repo Stats