Attention Is All You Need: Understanding Transformers and How Inference Works
Language Transformer
Introduction
In 2017, a research paper from Google Research and Google Brain changed the direction of modern Artificial Intelligence.
The paper was titled “Attention Is All You Need”, written by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. The paper introduced a new neural-network architecture called the Transformer. Instead of processing a sequence step by step using recurrent neural networks, the Transformer relied entirely on attention mechanisms and removed both recurrence and convolution from the architecture.
At the time, the primary problem was sequence-to-sequence learning, particularly machine translation. However, the architecture turned out to be much more general than its original application. Transformers later became the foundation for systems such as BERT, T5, GPT-style language models, vision transformers, multimodal models, and modern large language models.
The most important idea is surprisingly simple:
Instead of forcing a model to process a sequence one element at a time, allow the model to determine which other elements are important through attention.
That change had enormous consequences for both training and inference.
Why Was the Transformer Needed?
Before Transformers, sequence modeling was dominated by architectures such as Recurrent Neural Networks (RNNs), LSTMs, and GRUs, often combined with attention mechanisms.
An RNN processes a sequence sequentially:
Token 1 → Token 2 → Token 3 → Token 4 → Token 5
↓ ↓ ↓ ↓ ↓
State 1 → State 2 → State 3 → State 4 → State 5
The model processes one position and then uses its resulting state when processing the next position.
This creates a major limitation: sequential computation.
If a sequence contains thousands of tokens, the computation naturally develops a dependency chain across those positions.
The Transformer proposed a fundamentally different approach.
Instead of:
Token 1 → Token 2 → Token 3 → Token 4
the model can process the representations together and use attention to determine relationships:
┌───────────────┐
│ Attention │
└───────┬───────┘
│
┌───────────┼───────────┐
▼ ▼ ▼
Token 1 Token 2 Token 3
▲ ▲ ▲
└───────────┴───────────┘
This makes the architecture much more amenable to parallel computation during training.
Google's original paper reported that the Transformer was more parallelizable and required substantially less training time than the recurrent/convolutional approaches it compared against.
The Core Idea: Attention
The central mechanism in the Transformer is self-attention.
Imagine the sentence:
“The animal didn't cross the street because it was tired.”
To understand what “it” refers to, a model needs to consider relationships between words.
Self-attention allows each token representation to interact with other token representations and determine which ones are important.
For a token such as:
it
the model can assign different attention weights to other tokens.
Conceptually:
it
│
├── animal → high relevance
├── street → lower relevance
├── cross → medium relevance
└── tired → contextual relevance
The important point is that attention is not simply looking at the previous token. It creates a mechanism for relating representations across the sequence.
Google's explanation of the Transformer describes this as directly modeling relationships between words, regardless of their distance in the sentence.
Query, Key and Value
The Transformer attention mechanism is built around three representations:
Query (Q), Key (K), and Value (V).
For each input representation, the model produces these vectors using learned linear transformations.
The attention calculation from the original paper is:
Attention(Q,K,V)=softmax(QKTdk)VAttention(Q,K,V) = softmax \left( \frac{QK^T}{\sqrt{d_k}} \right)V
This equation can be understood in three stages.
First, the model calculates the similarity between queries and keys:
QKTQK^T
This determines how strongly different positions relate to one another.
The result is divided by:
dk\sqrt{d_k}
to scale the values.
A softmax function then converts the scores into normalized attention weights.
Finally, those weights are applied to the value vectors.
Conceptually:
Input Representations
│
├────────► Query
│
├────────► Key
│
└────────► Value
│
▼
Attention Scores
│
▼
Softmax
│
▼
Weighted Values
│
▼
New Representation
This mechanism is the heart of the original Transformer.
Why Divide by √dk?
The scaling factor is easy to overlook, but it is important.
The attention mechanism calculates dot products between queries and keys.
As the dimensionality of these vectors increases, the magnitude of their dot products can also increase.
Very large values passed into softmax can cause the probability distribution to become extremely concentrated.
The Transformer therefore scales the dot products by:
dk\sqrt{d_k}
This produces the scaled dot-product attention used in the original architecture.
Self-Attention
The word self in self-attention is important.
Self-attention means that the queries, keys, and values are derived from the same sequence representation.
Suppose we have:
The engineer designed the system
The representation of each word can attend to other words in the same sequence.
Conceptually:
The
↙ ↓ ↘
Engineer ← Attention → System
↖ ↑ ↗
Designed
The resulting representation of a token is therefore contextual.
The meaning of a token is not represented independently. It is influenced by the other tokens that the attention mechanism considers relevant.
Multi-Head Attention
The original Transformer does not rely on a single attention calculation.
It uses Multi-Head Attention.
Instead of asking one attention mechanism to learn every type of relationship, the model creates multiple attention heads.
Conceptually:
Input
│
┌───────────┼───────────┐
▼ ▼ ▼
Head 1 Head 2 Head 3
│ │ │
▼ ▼ ▼
Relationship Relationship Relationship
│ │ │
└───────────┼───────────┘
▼
Concatenate
│
▼
Linear Layer
Different heads can learn different relationships within the representation.
The original paper described multi-head attention as allowing the model to jointly attend to information from different representation subspaces and at different positions.
The Original Transformer Architecture
The original Transformer used an encoder-decoder architecture.
The encoder processes the input sequence, while the decoder generates the output sequence.
A simplified view is:
INPUT
│
▼
┌─────────────┐
│ Encoder │
│ Stack │
└──────┬──────┘
│
▼
Contextual
Representations
│
▼
┌─────────────┐
│ Decoder │
│ Stack │
└──────┬──────┘
│
▼
OUTPUT
The original architecture used six encoder layers and six decoder layers in the base configuration described in the paper.
Each encoder layer contains:
Multi-head self-attention
Position-wise feed-forward network
Each decoder layer contains:
Masked multi-head self-attention
Encoder-decoder attention
Position-wise feed-forward network
Residual connections and layer normalization are used throughout the architecture.
Why Does the Transformer Need Positional Information?
There is an important problem with attention.
If we provide the model with a collection of token representations, attention itself does not inherently tell the model whether a token appeared first, second, or tenth.
But order matters.
Compare:
“Dog bites man.”
with:
“Man bites dog.”
The same words appear, but their order changes the meaning.
The original Transformer addressed this using positional encoding.
The positional information is added to the token embeddings before they enter the Transformer layers.
Conceptually:
Token Embedding
+
Position Encoding
│
▼
Transformer Input
The original paper used sinusoidal positional encodings as one approach, while also discussing learned positional embeddings.
Encoder Self-Attention vs Decoder Self-Attention
There is an important distinction between the encoder and decoder.
The encoder can generally attend to all positions in the input sequence.
The decoder, however, generates the output sequentially.
During generation, the model must not look at future output tokens.
Therefore, the decoder uses masked self-attention.
For example, when generating:
The
the model should not see future tokens.
When generating: The engineer
it can use:
The
but not future tokens.
This creates a triangular attention structure:
Token 1 ✓
Token 2 ✓ ✓
Token 3 ✓ ✓ ✓
Token 4 ✓ ✓ ✓ ✓
Future positions are masked.
This concept becomes extremely important when we move from the original Transformer to modern autoregressive LLM inference.
Training vs Inference
One of the most important concepts when studying Transformers is that training and inference behave very differently. During training, the model can process many positions in parallel. During inference for an autoregressive language model, the model generates tokens one at a time.
This difference has major implications for performance.
Consider:
Prompt: "The data engineer deployed the"
Expected continuation: "application"
After generating: application
the model may generate:
to
Then: production
Then: successfully
The generation process looks like:
Prompt
│
▼
Token 1
│
▼
Token 2
│
▼
Token 3
│
▼
Token 4
This sequential dependency is one of the major challenges of LLM inference.
What Is Inference?
Inference is the process of using a trained model to produce an output for new input.
Training looks like:
Training Data
↓
Model
↓
Loss
↓
Backpropagation
↓
Updated Weights
Inference looks like:
User Input
↓
Tokenizer
↓
Model
↓
Logits
↓
Probability Distribution
↓
Token Selection
↓
Next Token
The important difference is that during inference the model's learned weights are generally fixed. The model is no longer learning. It is using what it learned during training.
How LLM Inference Generates Text
Suppose the prompt is: “The capital of France is”
The tokenizer converts the text into tokens. Conceptually:
"The capital of France is"
↓
Token IDs
↓
Transformer
↓
Logits
↓
Probability Distribution
The model might assign probabilities such as:
The model then selects a token according to the configured decoding strategy.
If it selects: Paris
the sequence becomes: The capital of France is Paris. The model can then run another generation step to determine what comes next.
Autoregressive Generation
Many generative Transformer models use autoregressive generation.
The basic loop is:
Input
↓
Predict next token
↓
Append token
↓
Run model again
↓
Predict next token
↓
Append token
↓
Repeat
In mathematical terms, the model estimates:
P(xt∣x1,x2,...,xt−1)P(x_t|x_1,x_2,...,x_{t-1})
The probability of the next token depends on the tokens that have already been generated.
The probability of an entire sequence can therefore be decomposed as:
P(x1,x2,...,xn)=∏t=1nP(xt∣x1,...,xt−1)P(x_1,x_2,...,x_n) = \prod_{t=1}^{n} P(x_t|x_1,...,x_{t-1})
This is the mathematical foundation behind autoregressive generation.
Why Inference Can Be Expensive
Suppose a model needs to generate 500 tokens. The model may need to perform a generation step for each new token.
Therefore:
500 output tokens
↓
Potentially hundreds of decoding iterations
Even if the GPU can process a huge amount of computation in parallel within each iteration, the dependency between generated tokens remains. This is fundamentally different from training, where many token positions can be processed simultaneously. This is one reason why LLM inference latency and cost have become major engineering challenges.
The KV Cache
One of the most important inference optimizations for Transformer-based language models is the Key-Value cache, commonly called the KV cache.
Remember that attention uses:
Query
Key
Value
During autoregressive generation, previously generated tokens do not need to have their key and value representations recomputed from scratch at every step. Instead, the model can cache previously computed keys and values.
Conceptually:
Previous Tokens
│
▼
┌───────────────┐
│ KV Cache │
│ K1 K2 K3 ... │
│ V1 V2 V3 ... │
└───────┬───────┘
│
▼
New Query
│
▼
Attention
│
▼
Next Token
This substantially reduces repeated computation during autoregressive decoding. However, the KV cache itself consumes memory. As the context length and number of concurrent requests increase, KV-cache memory can become a major inference-system constraint.
Prefill and Decode
Modern LLM inference can be understood as two major phases:
Prefill and Decode.
Prefill
The model processes the initial prompt.
For example:
User:
Explain distributed systems in simple terms.
The model processes the complete input and constructs the necessary internal representations, including the KV cache. This phase can involve substantial parallel computation.
Decode
The model then generates the output token by token.
Explain
→ distributed
→ systems
→ in
→ simple
→ terms
→ ...
Each generated token depends on the previous sequence.
This means decode is much more sequential.
A simplified inference pipeline is therefore:
Prompt
│
▼
┌─────────┐
│ Prefill │
└────┬────┘
│
▼
KV Cache
│
▼
┌─────────┐
│ Decode │
└────┬────┘
│
┌────────┴────────┐
▼ ▼ ▼
Token Token Token
1 2 3
Understanding this distinction is extremely important when optimizing LLM serving infrastructure.
The Inference Bottleneck
The fundamental problem can be summarized as:
Training:
Many tokens → parallel computation
Inference:
Token 1 → Token 2 → Token 3 → Token 4
The second process contains an unavoidable dependency for standard autoregressive generation. This means that simply adding more computational resources does not automatically eliminate generation latency.
Engineers therefore use techniques such as:
KV caching
Batching
Quantization
Tensor parallelism
Pipeline parallelism
Continuous batching
Efficient attention implementations
Speculative decoding
Model compression
Hardware acceleration
to improve inference performance.
Google's Work on Faster Transformer Inference
The original Transformer paper focused primarily on the architecture and machine translation. Google Research later investigated ways to accelerate inference from autoregressive Transformers. One notable technique is speculative decoding.
Google Research published work titled “Fast Inference from Transformers via Speculative Execution”, which describes a method where a smaller and faster model proposes multiple tokens and the larger model verifies them.
The basic idea is:
Large Model
▲
│ verify
│
Small Model ─────────┘
│
├── Token 1
├── Token 2
├── Token 3
└── Token 4
Instead of asking the expensive model to generate exactly one token at a time, a smaller model proposes several candidate tokens.
The larger model can then process and verify those candidates together. If the candidates are accepted, multiple output tokens can effectively be produced from fewer expensive large-model decoding steps.
Google's reported experiments demonstrated 2×–3× acceleration on T5-XXL in the described setup while maintaining the same output distribution.
Speculative Decoding in Simple Terms
Imagine you need to travel 100 kilometers.
Normally, you move:
10 km → stop → calculate next 10 km
10 km → stop → calculate next 10 km
...
A smaller model acts like a fast navigator that predicts several upcoming steps:
Predict:
A → B → C → D
The large model then checks those predictions.
If they are correct enough:
A ✓
B ✓
C ✓
D ✓
the system can move forward quickly.
If the prediction fails:
A ✓
B ✓
C ✗
the large model corrects the sequence and continues. The important property of the method described by Google is that the acceleration does not require changing the large model's output distribution.
Why Speculative Decoding Matters
The cost of LLM inference is increasingly important because models are being used in production systems that may serve millions of requests.
For every request, the infrastructure must consider:
Model Size
+
Context Length
+
Output Length
+
Concurrency
+
GPU Memory
+
GPU Compute
+
Network
+
Latency Requirements
An optimization that reduces the number of expensive large-model decoding iterations can therefore have a significant effect on system throughput and latency.
Google Research has continued to study speculative decoding and described it as a technique for faster and cheaper LLM inference without compromising output quality.
Transformer Attention Complexity
A major challenge of standard self-attention is its relationship with sequence length. If a sequence contains nn tokens, the attention matrix involves interactions between token pairs.
This gives the standard attention mechanism approximately:
O(n2)O(n^2)
attention complexity with respect to sequence length.
For example:
1,000 tokens
≈ 1,000 × 1,000 attention relationships
10,000 tokens
≈ 10,000 × 10,000 attention relationships
The growth is quadratic. This becomes increasingly expensive as context windows become larger.
Google Research has explored several approaches to this problem, including Performers, which use a linear-time and linear-space approximation to softmax attention, and BigBird, which uses sparse attention to reduce the quadratic dependency on sequence length.
Why This Matters for Data Engineers
At first glance, Transformers may appear to be purely an AI topic. They are not. Running modern AI systems requires a substantial data and infrastructure platform.
A production architecture might look like:
Data Sources
│
┌──────────┼──────────┐
▼ ▼ ▼
Documents Events Databases
│ │ │
└──────────┼──────────┘
▼
Data Lake
│
▼
Processing / ETL
│
▼
Training Dataset
│
▼
Model Training
│
▼
Model Artifact
│
▼
Model Registry
│
▼
Inference Service
│
┌─────────┴─────────┐
▼ ▼
Users Applications
The data engineer may therefore be responsible for:
Data ingestion
Data quality
Feature or training-data pipelines
Distributed processing
Storage
Model-serving infrastructure
Monitoring
Cost optimization
Batch inference
Real-time inference
Observability
Understanding Transformer inference helps explain why these systems require significant compute and memory infrastructure.
Transformer Inference Architecture
A simplified production LLM-serving architecture could look like:
Users
│
▼
API Gateway
│
▼
Load Balancer
│
▼
Inference Router
│
┌─────────────┼─────────────┐
▼ ▼ ▼
GPU Node GPU Node GPU Node
│ │ │
▼ ▼ ▼
Model Server Model Server Model Server
│ │ │
└─────────────┼─────────────┘
│
▼
Token Generation
│
▼
Response
Inside each inference server, the important components can include:
Tokenizer
↓
Request Scheduler
↓
Prefill
↓
KV Cache
↓
Decode Loop
↓
Sampling / Decoding
↓
Detokenization
↓
Response
This is where Transformer theory meets real-world cloud and distributed-systems engineering.
What Happens Inside One Generation Step?
Suppose the current context is: "The engineer deployed the application to"
The model receives the current token representation and performs its Transformer computations.
Conceptually:
Tokens
↓
Embeddings
↓
Position Information
↓
Transformer Layers
↓
Self-Attention
↓
Feed-Forward Networks
↓
Final Hidden Representation
↓
LM Head
↓
Logits
↓
Softmax / Sampling
↓
Next Token
The output might be: "production"
The token is appended to the context.The process repeats.That loop is the essence of autoregressive inference.
Temperature, Sampling and Token Selection
The model produces logits, not directly a sentence. These logits are converted into probabilities. A decoding strategy then determines which token is selected.
For example:
Token Probability
production 0.55
development 0.20
AWS 0.10
the 0.05
other 0.10
A greedy decoder chooses: production because it has the highest probability.
Sampling-based approaches can introduce controlled randomness.
Parameters such as temperature affect the sharpness of the probability distribution.
Lower temperature generally makes the output more deterministic, while higher temperature produces more variation.
Transformer vs Traditional RNN
The architectural difference can be summarized as:
The Transformer did not make inference completely parallel.
This is an important distinction.
Training became much more parallelizable, but autoregressive generation still contains sequential dependencies.
That is one reason inference optimization remains an active research area.
The Evolution After the Original Paper
The original Transformer was only the beginning.
The architecture has since evolved into many forms.
Transformer
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Encoder Decoder Encoder-Decoder
│ │ │
▼ ▼ ▼
BERT GPT T5
│ │ │
└─────────────┼─────────────┘
▼
Modern Foundation
Models
Transformers have also expanded beyond text.
Google Research demonstrated the use of Transformer architectures for computer vision with the Vision Transformer (ViT), treating images as sequences of patches.
Other research has focused on longer sequences, efficient attention, adaptive computation, and improved architectures.
The Most Important Lessons From the Paper
The original Attention Is All You Need paper is important not simply because it introduced a new neural network.
Its larger contribution was architectural.
It demonstrated that sequence transduction could be built around attention rather than recurrence or convolution.
The architecture provided three particularly important advantages:
Parallel computation during training: token positions could be processed much more concurrently than in recurrent architectures.
Direct interaction between positions: attention allows representations to interact across the sequence without requiring information to pass through a long chain of recurrent states.
Scalability: the architecture became a strong foundation for increasingly large models and later research.
The original experiments showed strong results on WMT 2014 machine-translation benchmarks, including 28.4 BLEU on English-German and 41.0 BLEU on English-French for the reported Transformer configuration, with the model trained on eight GPUs for the larger setup.
The Bigger Picture: From Attention to LLM Inference
The evolution can be summarized as:
2017
Attention Is All You Need
│
▼
Transformer
│
▼
Large-scale Pretraining
│
▼
Foundation Models
│
▼
Large Language Models
│
▼
Production Inference
│
├── KV Cache
├── Quantization
├── Batching
├── Parallelism
├── Efficient Attention
└── Speculative Decoding
The original Transformer solved a fundamental architectural problem.
Modern AI engineering is now solving another problem:
How can we run these increasingly powerful models quickly, cheaply, and reliably at production scale?
That is why understanding both Transformer architecture and inference architecture is important.
Final Takeaway
The most important idea from “Attention Is All You Need” is not simply the word attention.
It is the architectural shift from:
“Process the sequence step by step”
to:
“Let the model determine which parts of the sequence should interact.”
That seemingly simple change created an architecture that was highly parallelizable during training and capable of modeling relationships across a sequence using self-attention.
But modern LLMs introduce another challenge.
During autoregressive inference, the model still needs to generate output tokens sequentially:
Token → Token → Token → Token
This makes inference a major engineering problem.
Techniques such as KV caching, batching, quantization, efficient attention, parallelism, and speculative decoding are therefore essential for turning Transformer models into practical production systems.
Google's research on speculative decoding demonstrates one particularly interesting direction: use a faster model to propose several tokens and allow the larger model to verify them, potentially reducing expensive decoding work while preserving the desired output distribution.
So the journey from the 2017 Transformer paper to today's AI infrastructure can be viewed as:
Attention → Transformer → Foundation Model → LLM → Efficient Inference → Production AI.
And that is why “Attention Is All You Need” remains one of the most important research papers for anyone working in modern AI, machine learning, data engineering, or cloud infrastructure.
References
Google Research — Attention Is All You Need
Original research paper — Attention Is All You Need
Google Research — Fast Inference from Transformers via Speculative Execution
Google Research — Looking Back at Speculative Decoding
Google Research — Rethinking Attention with Performers
Google Research — BigBird: Transformers for Longer Sequences