When you first study natural language processing, the journey usually begins with sequential models such as recurrent neural networks. These models read sentences word by word, updating a hidden state at step one, passing it to step two, and continuing until the end of the phrase. While intuitive, this step-by-step chain creates a computational bottleneck: a computer cannot work on the tenth word until it finishes processing the ninth.

To overcome this limitation the transformer takes a different path. Instead of reading tokens sequentially, it consumes an entire sequence of tokens at the same time. At the input stage, raw text is converted into numerical representations: words or sub-words are split into discrete tokens, mapped to numerical identifiers using a fixed vocabulary, and projected through an embedding table into continuous vectors. Because the model processes all tokens at once rather than one after another, it adds positional embeddings to these vectors so that word order is preserved. Finally several of these sequences are grouped into a batch to be processed together.

Once this preparation is complete, the input text lives inside your program as a single three-dimensional grid of numbers, denoted as:

Every letter in this mathematical notation describes a concrete axis:

• $B$ represents the batch size, meaning the number of separate sentences or documents you are feeding into the model at that moment.
• $T$ represents the sequence length or time dimension, which is the number of token positions in each sentence.
• $C$ represents the channel or embedding dimension, which is the size of the feature vector representing each token's meaning and position.

In the original paper Attention Is All You Need (Vaswani et al., 2017, arXiv:1706.03762), the authors formulated these dimensions algebraically. They denoted the embedding size $C$ as $d_{model}$ (set to $512$ in their base architecture) and the sequence length $T$ as $n$. The batch dimension $B$ was treated as an outer grouping for parallel computation.

Everything that happens next in the Transformer is an operation on this three-dimensional tensor. The goal of these operations is to allow the tokens to interact with each other. Before tokens can interact, however, each token must take on three distinct conversational roles:

1. A Query ($Q$), representing what the token is searching for in the rest of the sentence.
2. A Key ($K$), representing a label of what the token contains, which other queries can match against.
3. A Value ($V$), representing the actual information the token contributes if another token finds its key relevant.

To create these three roles the model applies a linear projection to $X$. In a linear projection, every individual token vector of length $C$ is multiplied by a matrix of learned weights. This projection is position-wise: the model applies the exact same transformation matrix to every position across all sequences in the batch. Instead of multiplying $X$ three separate times by three different matrices to get $Q$, $K$, and $V$, it is more straightforward to use a single wider matrix:

Multiplying our input tensor X by this projection matrix yields a combined tensor:

Observe what happened to the dimensions. The batch axis $B$ and the sequence length $T$ did not change; you still have the same number of sentences and the exact same number of token positions. Only the final axis grew: each position now holds $3C$ numbers instead of $C$.

Because these $3C$ channels simply consist of queries, keys, and values placed side by side, we separate this final axis into three equal pieces of size $C$:

Now, each role has its own $(B, T, C)$ tensor.

At this point, you might wonder why we do not immediately calculate attention between all $C$ numbers of $Q$ and all $C$ numbers of $K$. If we did that, each token would produce only a single similarity score with every other token. Language, however, carries multiple relationships at once. In the sentence "The cat sat on the mat because it was tired", the pronoun "it" needs to connect to "cat" to resolve reference, but it also needs to connect to "was tired" to resolve the predicate.

To allow the model to track several relationships at the same time, the Transformer divides the channel axis into $H$ separate groups called attention heads. If each head receives $D$ channels, the total number of channels must satisfy a simple arithmetic constraint:

In Vaswani et al. (2017), the number of heads $H$ was designated as $h$ (set to $8$), and the dimension per head $D$ was designated as $d_k$ (set to $d_k = 512 / 8 = 64$).

This divisibility equation is strict. If your embedding size is $C = 12$ and you choose $H = 3$ heads, each head receives $D = 4$ feature dimensions. But if you have $C = 10$ and attempt to divide it across $H = 3$ heads, you cannot divide $10$ numbers into $3$ equal integer pieces. The computer cannot allocate fractional numbers to a head, so the architecture cannot function without changing $C$ or $H$.

When $C$ is divisible by $H$, we reshape the final dimension of each tensor from $C$ into $(H, D)$:

This reshape does not change any values or create new data. It simply places conceptual boundaries along the channel axis, indicating that the first $D$ numbers belong to Head 1, the next $D$ numbers belong to Head 2, and so forth.

However, the tensor is now in the shape $(B, T, H, D)$. If you read this shape aloud, it says: for each sentence in the batch ($B$), go to each word position ($T$), and examine its heads ($H$).

This ordering presents a problem for attention. Self-attention does not compare different heads belonging to the same word. Instead, it compares all words in the sentence within the same head. To compute Head 1, we need all $T$ words laid out together under Head 1.

To achieve this alignment, the model swaps the second and third axes:

Reading this new shape reveals the logic of the transformation: for each sentence ($B$), go to each head ($H$), and look at the sequence of words ($T$), where each word is described by its head vector of size $D$.

With the tensor arranged as $(B, H, T, D)$, calculating attention becomes direct. For any given head in any sentence, our queries form a matrix of shape $(T, D)$ and our keys form a matrix of shape $(T, D)$. Multiplying queries by transposed keys matches words against words:

The result is a $T \times T$ grid for each head, containing the pairwise connection strength between every token position $i$ and every token position $j$.

The entire journey of the representation through these transformations forms a continuous geometric pipeline:

Every step in this sequence maintains a clear physical meaning. The input grid of token embeddings is expanded to generate distinct conversational roles, divided into smaller subspaces to capture multiple semantic relationships in parallel, and reordered so that tokens can evaluate each other across the entire sentence.