gpt-neox

This project targets the challenge of training large-scale autoregressive language models on GPUs, focusing on models with billions of parameters. It builds on NVIDIA's Megatron-LM and Microsoft DeepSpeed libraries but introduces usability improvements and novel optimizations like rotary and alibi positional embeddings, parallel feedforward attention layers, and flash attention. It supports distributed training with ZeRO and 3D parallelism across diverse hardware and job schedulers, including Slurm and MPI, and has been tested on high-end supercomputers such as Oak Ridge's Summit and Frontier. Its maturity is reflected in over 7,400 stars and recent commits, though growth has not been significant. The main users are ML engineers and researchers focused on training custom large language models from scratch. It is not suitable for teams seeking inference or fine-tuning solutions; for those, Hugging Face transformers is a better fit.

Choose this when you need to train very large autoregressive transformer models from scratch on multi-GPU or multi-node setups, especially if you require advanced parallelism and hardware flexibility. For smaller-scale training or inference tasks, consider Hugging Face transformers or NVIDIA Megatron-LM, which may offer simpler setup or better community support.

Typically integrated into research or enterprise workflows that involve training new large language models for tasks like developing foundational models or domain-specific generative AI systems. It is less common in teams focused solely on deploying pre-trained models or inference pipelines.

Latest commit: 2026-04-13. Activity data is based on daily RepoPi snapshots of the GitHub repository.