x-transformers is a PyTorch library that provides a concise but fully-featured transformer implementation, aggregating promising experimental architectural improvements from recent research papers into a single accessible package.
Project overview
The library consolidates a wide collection of experimental transformer features from recent papers into a single concise codebase, making architectural research and rapid model prototyping significantly more accessible for developers and researchers.
Project type
Model Development · Model Runtime
Deployment
Refer to project documentation
License
MIT
Best for
Developers and researchers who need to rapidly prototype and experiment with various state-of-the-art transformer architectural improvements.
Key capabilities
Supports full encoder/decoder, decoder-only, and encoder-only transformer models.
Supports image classification and image-to-caption models via ViTransformerWrapper.
Offers configurable experimental features including Flash Attention, GLU feedforward variants, memory tokens, and RMSNorm.
Provides wrappers for training and generation, including XLAutoregressiveWrapper, ContinuousAutoregressiveWrapper, and XValAutoregressiveWrapper.
Limitations and risks
Flash Attention cannot be used if your model requires operating directly on the attention matrix, such as when using dynamic positional bias, talking heads, or residual attention.
Getting started
Run $ pip install x-transformers to install the package, then import XTransformer from x_transformers in Python to begin building models.
Evidence and sources
GitHub project description: A concise but complete full-attention transformer with a set of promising experimental features from various papers
README: $ pip install x-transformers
README: A concise but fully-featured transformer, complete with a set of promising e**x**perimental features from various papers.