In the foundational formulation of the Transformer by Vaswani et al. (2017, arXiv:1706.03762), attention is defined with concise algebraic economy: mapping a query and a set of key-value pairs to an output vector, where the output is computed as a weighted sum of the values. Despite this explicit definition, introductory treatments often focus almost exclusively on the mechanics of query-key matrix multiplication and the scaling factor $\sqrt{d_k}$. This emphasis leads to a common misconception: treating the value matrix as a passive bystander in an algorithm defined by dot products.

In physical and mathematical terms, the aggregation of values is the operation that executes attention. The dot products and the softmax normalization exist solely to construct a set of scalar coefficients. Once computed, those coefficients dictate how existing representations blend to form a new representation. To understand the geometry of this operation, one can temporarily set aside queries and keys, analyzing attention strictly as a linear aggregation operator over vectors.

Suppose a sequence contains $T$ positions, each represented by a value vector $v_j \in \mathbb{R}^{d_v}$. In the base architecture of Vaswani et al., the channel dimension of the values is set to $d_v = d_{\text{model}} / h = 64$. Given a set of scalar weights $\alpha_1, \alpha_2, \ldots, \alpha_T \in \mathbb{R}$, attention combines these vectors into a single output vector $y$:

For this aggregation to act as a normalized mixture, the weights must satisfy two conditions:

The non-negativity constraint ensures that representations cannot be inverted or subtracted; information is combined through positive accumulation. The unit sum ensures that total scale is preserved, preventing the norm of the resulting vector from growing unboundedly with sequence length.

These two conditions define a convex combination. In linear algebra, a convex combination of vectors $v_1, \ldots, v_T$ generates a point that lies within the convex hull spanned by those vectors:

The convex hull forms the smallest convex polytope containing the set of values. This geometric confinement prevents arbitrary divergence: regardless of the weights assigned, the output vector $y$ remains anchored within the coordinate boundaries established by the tokens in the sequence.

The properties of this convex mixture change according to the entropy of the weights:

• If a single weight equals one ($\alpha_k = 1$) while all others are zero, the sum simplifies to $y = v_k$. The operation acts as a hard pointer, retrieving the exact vector of position $k$.
• If the weights are uniform ($\alpha_j = 1/T$), the output becomes the arithmetic centroid $y = \frac{1}{T} \sum v_j$, corresponding to global mean pooling.
• Between these limits, attention functions as a soft, differentiable selection mechanism that blends relevant features from several sources into an integrated coordinate.

Because each value vector $v_j$ belongs to $\mathbb{R}^{d_v}$, the mixture $y$ also belongs to $\mathbb{R}^{d_v}$. Attention updates the information held by a position without changing the dimensionality of its representation space.

In neural architectures, scoring functions do not produce probabilities directly; they produce unbounded real numbers $s_j \in \mathbb{R}$, referred to as logits. To transform these raw scores into convex weights that satisfy $\alpha_j \ge 0$ and $\sum \alpha_j = 1$, Vaswani et al. apply the softmax function:

Exponentiation maps every real number to the strictly positive domain, ensuring $\exp(s_j) > 0$ for all $s_j \in (-\infty, \infty)$. Dividing by the sum of exponents normalizes the vector so that it sums to one. Substituting this into the linear mixture yields:

This expression formalizes the division of labor within the mechanism: the logits $s_j$ determine how much each position contributes, whereas the vectors $v_j$ determine the substance of what is combined.

The shared denominator in the softmax introduces direct competition across positions. Increasing a single score $s_j$ increases the numerator $\exp(s_j)$, but also expands the denominator for all other terms. Consequently, elevating the relevance of one token automatically depresses the weights allocated to the remaining tokens in the sequence.

When written for computational implementation, calculating the mixture through explicit loops is inefficient. The operation is expressed in matrix notation.

The $T$ value vectors are stacked as rows of a matrix $V \in \mathbb{R}^{T \times d_v}$. If the attention weights for a single query are written as a row vector $\alpha = [\alpha_1, \ldots, \alpha_T] \in \mathbb{R}^{1 \times T}$, the output vector is given by a single vector-matrix product:

In a self-attention layer, every token in the sequence acts as a query. If each of the $T$ positions computes its own set of normalized weights over the sequence, the individual row vectors form an attention matrix:

Because each row of $A$ represents a complete probability distribution over the $T$ positions, the entries satisfy:

A non-negative matrix whose rows sum to one is known in linear algebra as a row-stochastic matrix.

Multiplying this row-stochastic matrix by the value matrix yields the complete transformation of the sequence:

The dimensional structure remains invariant: $V$ has shape $(T, d_v)$ and the output $Y$ retains shape $(T, d_v)$. Each row $i$ in $Y$ is a convex combination of the rows of $V$, weighted by row $i$ of the attention matrix $A$.

In multi-head attention, this matrix product occurs independently across $h$ parallel subspaces. As detailed in Vaswani et al., the tensors incorporate a batch dimension $B$ and a head dimension $h$:

The batched matrix multiplication computes $B \times h$ separate row-stochastic combinations simultaneously:

Because the matrix $A$ provides an explicit map of weights between zero and one, it is often interpreted as a direct explanation of model decisions. This interpretation requires technical qualification.

A high attention weight $A_{ij} \approx 1$ establishes that position $i$ relied numerically on the vector stored at position $j$ during that specific layer. It does not prove that the semantic content of word $j$ caused the final output of the network. In deep architectures, value vectors at layer $L$ already contain mixtures of information accumulated from previous layers and residual connections. If a value vector carries diffuse or uninformative features, assigning it a large weight will not produce a semantically interpretable decision.

Similarly, low attention weights do not indicate that a token is irrelevant to the network. Across multiple stacked layers, small numerical contributions can compound through feed-forward sub-layers and residual paths to shape the final prediction. Attention weights record where numerical mass flows within an isolated matrix multiplication; they do not provide a complete causal account of the model's behavior.

Analyzing attention as a weighted mixture establishes the operational core of the Transformer. In this setting, the weights $A$ were treated as given. The subsequent step in the architecture is to make those weights depend systematically on the content of the sequence. To achieve this, Vaswani et al. introduce queries ($Q$) and keys ($K$), projecting token embeddings into compatible sub-spaces whose inner products generate the matrix of logits:

The queries specify what information a position requires; the keys specify what information each position contains; the softmax normalizes their compatibility into a convex distribution; and the values provide the vectors that are mixed to produce the updated representation.