Transformer (AI)
A neural network architecture introduced in 2017 that uses self-attention. The foundation of all modern large language models.
The transformer is a neural network architecture introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al. Its key innovation is the self-attention mechanism, which allows the model to weigh how strongly each part of the input relates to every other part when producing each output token.
Compared to earlier sequence models like RNNs and LSTMs, transformers are easier to parallelise during training, scale better to large datasets, and capture long-range dependencies more effectively. These properties enabled the rapid scale-up of language models from millions to trillions of parameters.
Transformers now power virtually every state-of-the-art language model (GPT, Claude, Gemini, Llama), most large vision models (Vision Transformers), and many speech and multimodal models.