In traditional sequence processing, a single hidden state is forced to fulfill multiple competing responsibilities: it summarizes the history of preceding words, encodes the identity of the current token, and prepares the vector that will be matched against neighboring positions. Compressing these distinct communicative duties into an unspecialized representation creates an informational compromise. A token cannot easily express what it is searching for without diluting the description of what it currently contains.

To resolve this bottleneck, the Transformer architecture introduced by Vaswani et al. (2017, arXiv:1706.03762) decomposes the representation of each token into three distinct functional projections: a query ($q$), a key ($k$), and a value ($v$). Rather than relying on a single vector to perform search, indexing, and information delivery simultaneously, the model projects the input through three separate linear transformations.

Suppose an input sequence contains token vectors $x_i \in \mathbb{R}^C$, where $C$ represents the channel embedding dimension (denoted $d_{\text{model}}$ in the original paper). At each position $i$, the model computes three specialized vectors:

where $W_Q, W_K \in \mathbb{R}^{C \times d_k}$ and $W_V \in \mathbb{R}^{C \times d_v}$ are learned parameter matrices. In the base model of Vaswani et al., the projection dimensions are set to $d_k = d_v = d_{\text{model}} / h = 64$, where $h = 8$ denotes the number of parallel attention heads.

Each of these three vectors fulfills a specialized operational role within the attention mechanism:

The query vector $q_i$ represents the inquiry formulated by position $i$. It specifies the informational attributes that the current token requires from other positions in the sequence to resolve its syntactic dependencies or semantic ambiguity.

The key vector $k_j$ functions as an indexing descriptor for position $j$. It advertises the properties of the token residing at that position, providing an addressable profile against which incoming queries can be matched.

The value vector $v_j$ contains the substantive content that position $j$ will contribute to the rest of the network if its key matches an active query.

This division of labor establishes the foundational principle of self-attention: the mechanism never evaluates compatibility between the representations that are actually transported. It compares queries with keys to determine routing weights, and then applies those weights strictly over the values. This separation allows a position to advertise one set of properties during indexing while transmitting an entirely different set of features during aggregation.

To evaluate how strongly position $i$ should attend to position $j$, the model measures the directional alignment between the query $q_i$ and the key $k_j$ using an inner product:

When the query vector and the key vector point in similar directions in their shared $d_k$-dimensional subspace, their inner product is positive and large. When they are orthogonal, their inner product is zero. When they point in opposite directions, it is negative. This inner product does not measure superficial orthographic or dictionary similarity; it reflects alignment within an abstract representational space learned through gradient descent to minimize task loss.

However, calculating compatibility solely through raw dot products introduces a numerical instability that worsens as the channel dimension $d_k$ increases. To analyze this behavior, Vaswani et al. examine the statistical properties of the dot product under standard initialization conditions. Assume that the components of $q_i$ and $k_j$ are independent random variables with mean zero and variance one:

The product of two independent, zero-mean unit-variance variables has a mean of zero and a variance of one:

Because the dot product $q_i^T k_j$ is the sum of $d_k$ such independent terms, its mean remains zero, but its variance scales linearly with the dimensionality:

The standard deviation of the raw dot product is therefore $\sqrt{d_k}$. When $d_k$ is small, this dispersion is manageable. But when $d_k$ reaches typical architectural scales such as $64$, the standard deviation reaches $8$; for $d_k = 512$, it reaches approximately $22.6$.

Under such large standard deviations, dot products frequently produce large positive or negative values. When these extreme logits are fed into the softmax function, exponentiation amplifies the gap between the largest score and the remaining scores. The softmax distribution collapses into a near-deterministic one-hot vector, placing almost all probability mass on a single position.

In this saturated regime, the gradients of the softmax function with respect to its inputs approach zero:

This vanishing gradient impedes backpropagation, preventing the projection matrices $W_Q$ and $W_K$ from learning effectively. To neutralize this dimensional growth, Vaswani et al. introduce scaled dot-product attention, dividing the inner product by the square root of the projection dimension:

Dividing by $\sqrt{d_k}$ normalizes the variance of the compatibility scores back to one, preserving a smooth distribution over sequence positions and maintaining non-vanishing gradients throughout training.

Once calculated, the scores for each query position $i$ are converted into attention weights using the softmax function across all available key positions $j \in \{1, \ldots, T\}$:

The scalar $\alpha_{ij}$ represents the proportion of attention that token $i$ assigns to token $j$. Because the softmax operates along the sequence dimension, the weights form a valid probability distribution:

The final representation for position $i$ is obtained by computing the convex combination of all value vectors according to these weights:

The resulting vector $y_i$ is a context-dependent mixture of values. It contains information drawn from tokens across the sentence, weighted by how well their keys satisfied the inquiry formulated by query $q_i$.

A frequent source of confusion when analyzing this pipeline is conflating the role of the projection dimension $d_k$ with the sequence length $T$. The channel dimension $d_k$ determines the capacity of the comparison space; it is along this dimension that inner products are calculated. In contrast, the sequence length $T$ determines the cardinality of the competition pool; it is across these $T$ positions that the softmax normalizes the scores. The score $s_{ij}$ is not an absolute threshold; it is relative to the other scores in row $i$. A position with a positive compatibility score may still receive negligible attention weight if another token in the sentence produces a significantly larger alignment.

It is also important to demystify the retrieval terminology. Although "query", "key", and "value" are intuitive metaphors inherited from database systems, they do not denote discrete symbols or natural language questions. They are continuous, differentiable coordinate vectors produced by linear projections. Their internal geometric structure is not manually authored; it emerges entirely from the interaction between gradient descent, the loss objective, and the architectural constraints of the network.

Finally, relying on a single set of projection matrices would force the model to average all semantic and syntactic relationships into a single attention distribution. In natural language, a verb may simultaneously need to identify its grammatical subject, bind its direct object, and adjust for adverbial tense. To allow the model to track multiple relational types in parallel, Vaswani et al. replicate this functional triad across $h$ independent heads, projecting queries, keys, and values into separate representational subspaces. Within each subspace, the triad executes its scaled dot-product calculus, producing specialized mixtures that are concatenated and projected to form the final multi-head output.