---
title: "LM Head & Generation"
description: "Projecting hidden states to probabilities and generating text."
canonical_url: "https://angeloai.vercel.app/docs/concepts/07-lm-head"
markdown_url: "https://angeloai.vercel.app/docs/concepts/07-lm-head.md"
last_updated: "2018-10-20"
---

# LM Head & Generation
URL: /docs/concepts/07-lm-head
LLM index: /llms.txt
Description: Projecting hidden states to probabilities and generating text.

# The LM Head, Logits, & Generation

After passing through all the Transformer blocks, the model outputs a final **Hidden State** for the last token in the sequence. This hidden state contains the semantic understanding of everything processed so far.

But we need a word prediction, not a dense vector.

### The LM Head
The Language Modeling (LM) Head is a final Linear layer that projects this hidden state from the `d_model` dimension (e.g., 128) up to the `vocab_size` dimension (e.g., 5,000). 

*Note: In Angelo (and models like GPT-2), the weights of this layer are tied (shared) with the Embedding layer, saving memory.*

### Logits & Softmax
The output of the LM Head is a list of 5,000 numbers called **Logits**. These are raw, unbounded scores (e.g., `[4.2, -1.5, 12.1, ...]`).
To make sense of them, we apply the **Softmax** function, which converts these raw logits into a proper probability distribution that sums to 100%.

### Generation (Temperature & Sampling)
Once we have probabilities (e.g., `"apple": 80%, "banana": 15%, "car": 5%`), we must pick the next token.
- **Greedy Decoding:** Always pick the highest probability (80%). This can lead to repetitive, boring text.
- **Temperature:** A mathematical trick applied to the logits before Softmax. 
  - A Temperature of `1.0` changes nothing. 
  - A Temperature `< 1.0` (e.g., `0.5`) makes the high probabilities even higher (more deterministic). 
  - A Temperature `> 1.0` (e.g., `1.5`) flattens the distribution, making lower-probability words more likely (more creative/random).

Once a token is sampled, it is appended to the input, and the entire process runs again to predict the next token. This is **autoregressive generation**.

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).