---
title: "Tokenization & Vocabulary"
description: "How raw text is converted into numbers the model can understand."
canonical_url: "https://angeloai.vercel.app/docs/concepts/02-tokenization"
markdown_url: "https://angeloai.vercel.app/docs/concepts/02-tokenization.md"
last_updated: "2018-10-20"
---

# Tokenization & Vocabulary
URL: /docs/concepts/02-tokenization
LLM index: /llms.txt
Description: How raw text is converted into numbers the model can understand.

# Tokenization & Vocabulary

Computers cannot process raw text strings like `"The cat sleeps"`. They require numbers.

**Tokenization** is the process of breaking text down into smaller, manageable pieces called **tokens**. A token can be a single character, a subword, or a whole word. 

### Subword Tokenization (BPE)
This is the standard for modern LLMs (including Angelo, which uses Byte-Pair Encoding). It strikes a balance by keeping common words intact (e.g., `"sleep"`) while breaking down rare words into chunks (e.g., `"unbelievable"` → `"un"`, `"believ"`, `"able"`). This allows the model to handle any word without needing an infinite vocabulary.

### Token IDs and Vocabulary
Once split, each unique token is mapped to a specific integer ID in a lookup table called the **Vocabulary**. 

- **Vocabulary Size:** The total number of unique tokens the model knows. 
- For Angelo, this is `5,000`. For models like GPT-4, it can exceed `100,000`. 
- A larger vocabulary means fewer tokens are needed to represent the same text (improving inference speed), but it significantly increases the size of the embedding matrix (requiring more memory).

### Example Pipeline
1. Raw Text: `"The cat sleeps"`
2. Tokens: `["The", " cat", " sleep", "s"]` 
3. IDs: `[464, 3797, 3993, 82]`

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).