Architecture
Transformer
The neural-network architecture behind nearly all modern language models, built around the attention mechanism.
Introduced in 2017, the transformer processes all tokens of a sequence in parallel and uses attention to decide which earlier tokens matter for each new one. GPT means 'generative pre-trained transformer'.
Frequently asked questions
What does Transformer mean?
The neural-network architecture behind nearly all modern language models, built around the attention mechanism. Introduced in 2017, the transformer processes all tokens of a sequence in parallel and uses attention to decide which earlier tokens matter for each new one. GPT means 'generative pre-trained transformer'.