---
title: "Embeddings & Position"
description: "Mapping discrete tokens to continuous semantic spaces."
canonical_url: "https://angeloai.vercel.app/docs/concepts/03-embeddings"
markdown_url: "https://angeloai.vercel.app/docs/concepts/03-embeddings.md"
last_updated: "2018-10-20"
---

# Embeddings & Position
URL: /docs/concepts/03-embeddings
LLM index: /llms.txt
Description: Mapping discrete tokens to continuous semantic spaces.

# Embeddings & Positional Encoding

### Word Embeddings
Once we have Token IDs (integers), we still have a problem: integers have implicit mathematical relationships (e.g., 2 is closer to 3 than 1000). But Token ID 2 (`"apple"`) has nothing to do with Token ID 3 (`"the"`). 

The **Embedding Layer** solves this by mapping each Token ID to a high-dimensional vector (a list of floating-point numbers). This vector represents the *semantic meaning* of the token in a continuous space. Words with similar meanings will end up with vectors that are close together in this mathematical space.
- The dimension of this vector is `d_model` (e.g., 128 for Angelo, 4096 for Llama 3).

### Positional Encoding
Because a Transformer processes all tokens in parallel, it has no built-in sense of order. Without positional encoding, `"Dog bites man"` and `"Man bites dog"` would look identical to the model.

**RoPE (Rotary Position Embedding)**
Instead of adding a static position vector to the embedding (like the original 2017 Transformer), modern models like Angelo use RoPE. RoPE *rotates* the token's vector in multi-dimensional space based on its position. 
- It encodes *relative* position (how far apart two words are) rather than absolute position (word #4 vs word #5). 
- This naturally allows the model to generalize to sequence lengths longer than what it was trained on.

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).