Attention
The mechanism by which each position in a sequence decides how much of every other position to mix into its own representation.
For each token the model computes a weight over the other tokens and takes a weighted average of their representations. The weights are learned, not specified — which is why a model resolves a pronoun to its referent without anyone writing a rule for pronouns.
Where this comes up
Articles that define attention in context.
What a transformer is actually doing when it "reads"
Attention, explained through a single nine-token sentence: what the weights mean, why the upper triangle of the matrix is empty, and what that mask has to do with the model's failure modes.
What is Artificial Intelligence?
AI, machine learning, deep learning and generative models are nested, not synonymous. Here is what distinguishes each layer, and which one people now mean when they say AI.