Definition
The transformer is a neural-network architecture that processes token representations through repeated blocks. In a typical block, attention mixes information between token positions, and a feed-forward network transforms each position’s representation. Residual connections provide skip paths across sublayers. Normalization helps stabilize activations and optimization in deep networks.
Transformers can process text, images, audio, or other data once those inputs are represented as suitable tokens or patches. Many current foundation models use decoder-only transformers for autoregressive generation, but that is only one form of the architecture.
Simple example
Consider the sentence “The deployment failed because it exceeded the quota”. As token representations pass through transformer blocks, attention can combine information from “it” with earlier positions such as “deployment”. Feed-forward layers then transform each position’s representation. After several layers, the resulting representations can support next-token prediction or another trained task.
Why it matters
Attention patterns determine which positions can interact directly. The context window bounds what the model can process together, while repeated blocks account for much of its compute and parameter count. During autoregressive transformer inference, KV caching reuses keys and values computed for earlier positions instead of recomputing them at every decoding step.
For serving, these choices affect memory use and latency, especially as requests grow longer.
One important nuance
“Transformer” names an architecture family. Encoder-only, encoder-decoder, and decoder-only models use attention differently. Implementations also vary in normalization, positional representation, attention patterns, and feed-forward layers. Architecture alone does not explain a model’s capabilities. Training data, objective, scale, adaptation, and inference configuration also matter.