tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

11.2k

Stars

+952

Gained

9.3%

Growth

Rust

Language

View on GitHub → ↑0.2% this week

🎭 Best For

Machine Learning and AI Engineer

🏷️ Topics & Ecosystem

bert gpt language-model natural-language-processing natural-language-understanding nlp transformers

📊 Activity

Latest commit: 2026-10-08. Over the past 330 days, this repository gained 952 stars (+9.3% growth). Activity data is based on daily RepoPi snapshots of the GitHub repository.