Jakob Uszkoreit had a hunch that a translation model could drop recurrence altogether and rely on attention alone. It went against conventional wisdom. His own father, the computational linguist Hans Uszkoreit, was skeptical. The hunch gave a 2017 paper its title, “Attention Is All You Need,” and the model inside it changed how AI works.
What came before
Language models of the time read one token after another. Recurrent networks, including LSTMs and the GRU variants, processed text from first word to last, so they could not work on a whole sentence in parallel. Long inputs were a problem too. In early seq2seq translation systems, the encoder squeezed a whole sentence into one fixed-size vector, and detail got lost. Reversing the input sentence helped, which tells you how shaky the setup was. Attention was bolted on to ease the bottleneck, but the sequential grind stayed, and that kept the models from running well on GPUs.
Google’s 2016 neural translation system was the high point of that era: eight layers of bidirectional LSTM in the encoder and the decoder, nine months of development, and better results than a statistical system that took ten years to build.
June 2017
On June 12, 2017, researchers at Google published the original 100-million-parameter encoder-decoder transformer. It kept the dot-product attention that made earlier systems good at text, removed the recurrence, and split attention into independent heads that could be computed at once. Because training could be parallelized, models could grow. That mattered more than anything else about the design.
Earlier that spring, before the preprint appeared, one of the co-authors had already run a decoder-only version to generate fictitious Wikipedia articles. Four days after the paper, most of its authors published MultiModel, a transformer that handled more than one kind of input.
The spread
Things moved quickly. BERT, an encoder-only transformer, arrived in 2018, and Google began using it to process search queries in October 2019. OpenAI’s GPT series of decoder-only transformers, also starting in 2018, became the state of the art in text generation. Google Translate swapped in a transformer encoder in 2020. Vision transformers took the idea to images. Audio, robotics, reinforcement learning and even chess followed.
The public noticed at the end of 2022, when ChatGPT, built on a fine-tuned GPT-3.5, turned unexpectedly popular and set off the boom in large language models. Image and video generators such as DALL-E, Stable Diffusion 3 and Sora also rely on transformers, which break prompts into tokens and use self-attention to weigh how they relate.
The design has three common forms. Encoders like BERT suit classification and analysis. Decoders like GPT suit generation. Encoder-decoders like T5 handle translation and summarization.
Training them takes more than one machine. Data parallelism copies a model across devices, while tensor and pipeline parallelism split the model itself.
Sources
- Transformer (deep learning) – Wikipedia
- Understanding LLMs and Transformers: A Deep Dive into Modern AI
