---
title: "Attention Mechanism"
description: "The core mechanism of the Transformer architecture."
canonical_url: "https://angeloai.vercel.app/docs/concepts/04-attention"
markdown_url: "https://angeloai.vercel.app/docs/concepts/04-attention.md"
last_updated: "2018-10-20"
---

# Attention Mechanism
URL: /docs/concepts/04-attention
LLM index: /llms.txt
Description: The core mechanism of the Transformer architecture.

# Self-Attention & Causal Masking

**Self-Attention** is the core mechanism that allows the model to understand context. It lets every token dynamically decide how much "attention" to pay to every other token in the sequence.

For example, in the sentence `"The animal didn't cross the street because it was tired"`, the attention mechanism allows the word `"it"` to look back at the sequence and strongly attend to `"animal"` rather than `"street"`.

### How it Works
1. **Query (Q), Key (K), Value (V):** Each token creates a Query (what I'm looking for), a Key (what I contain), and a Value (my actual content).
2. **Dot Product:** The model multiplies every token's Query with every other token's Key to calculate an "attention score" (how relevant they are to each other).
3. **Softmax:** The scores are converted to percentages.
4. **Weighted Sum:** The model multiplies these percentages by the Values to get the final contextualized representation.

### Multi-Head Attention
Instead of doing this once, the model splits the embedding into multiple "heads" (e.g., 4 heads in Angelo) and performs attention independently. This allows different heads to look for different types of relationships (e.g., one head tracks grammar, another tracks pronoun references).

### Causal Masking
In a Decoder-Only model, we are predicting the *next* word. If the model could look ahead at future words during training, it would simply cheat and copy the answer. A **Causal Mask** zeroes out all attention scores for future tokens, ensuring token $N$ can only attend to tokens $1$ through $N$.

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).