tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

11.0k

Stars

+780

Gained

7.6%

Growth

Rust

Language

View on GitHub → ↑0.2% this week

🎭 Best For

Machine Learning and AI Engineer

🏷️ Topics & Ecosystem

bert gpt language-model natural-language-processing natural-language-understanding nlp transformers

📊 Activity

Latest commit: 2026-08-24. Over the past 285 days, this repository gained 780 stars (+7.6% growth). Activity data is based on daily RepoPi snapshots of the GitHub repository.