Technology & Medicine › Internet, genomes & AI › Large language models
Transformer architecture
June 2017 (Google)· Invention or discovery· Communication & computing
Well documentedSolid records, physical evidence or scientific consensus.
A 2017 neural-network design that reads a whole passage at once using 'attention', now the basis of modern AI.
See it on the Technology & Medicine map →How it works
Transformers were introduced in a June 2017 paper, 'Attention Is All You Need', by eight Google researchers working on translation. Earlier language networks read words one at a time, which was slow and lost track of long sentences. A transformer uses 'self-attention': every word in a passage looks at every other word at once and weighs which matter most. Because this runs in parallel, it uses graphics chips efficiently and scales to huge datasets.
Why it mattered
It is the design inside nearly every modern AI chatbot (the T in GPT) and in many image, speech and protein models; the paper is among computing's most cited.
Think of it like
Like a lawyer reviewing a lease who cross-checks every clause against every other at once, rather than reading strictly line by line.
How we know
The 2017 paper, presented at that December's NeurIPS conference and cited well over 100,000 times; its translation benchmarks; open-source code released with it.
Still unknown or debated
Attention ideas existed earlier (Bahdanau and colleagues, 2014), so how novel the leap was is debated, as is whether more efficient designs will replace transformers.
Connections (2)
▸Deep learning (modern AI) was reshaped in 2017 by the Transformer architectureWell documented
The connection
By 2016, deep learning for language relied on recurrent networks that read text word by word, slowly and forgetfully over long passages. The 2017 transformer kept deep learning's core methods but replaced that step-by-step reading with attention across a whole passage at once, training far faster on graphics chips. It soon spread to images, speech and biology.
When and how
Google Brain and Google Research in Mountain View, California, 2017; the paper's eight authors included Ashish Vaswani, Noam Shazeer and Jakob Uszkoreit.
How we know
The 2017 paper showed better translation at a fraction of the training cost; within a few years transformers topped most AI benchmarks.
Debates and open questions
Whether transformers are the final architecture or will be overtaken by more efficient designs, such as state-space models, is open.
▸Transformer architecture underpins Large language modelsWell documented
The connection
Nearly every large language model is a stack of transformer layers, often dozens to over a hundred deep; GPT stands for 'generative pre-trained transformer'. Because transformers train in parallel, labs could feed them trillions of words, and the architecture's predictable gains with scale made multi-billion-dollar training runs worth the bet.
When and how
OpenAI's GPT in June 2018 and Google's BERT in October 2018, about a year after the transformer paper; ChatGPT in November 2022.
How we know
Model papers and technical reports, from GPT-1 to open models like Meta's Llama, describe transformer architectures explicitly.
Debates and open questions
Some researchers argue transformers' design limits long memory and reasoning, and hybrids are being tested; whether anything will dethrone transformers is uncertain.