Architecture
Attention mechanism
The part of a transformer that lets each token weigh how relevant every other token is when computing its meaning.
Attention is what lets a model connect a pronoun to the right noun pages earlier. Its cost grows with context length, which is one reason long contexts are expensive.
Frequently asked questions
What does Attention mechanism mean?
The part of a transformer that lets each token weigh how relevant every other token is when computing its meaning. Attention is what lets a model connect a pronoun to the right noun pages earlier. Its cost grows with context length, which is one reason long contexts are expensive.