--- title: "Model Architecture" description: "Every neural network component in Angelo, from embedding to LM head." canonical_url: "https://angeloai.vercel.app/docs/model-architecture" markdown_url: "https://angeloai.vercel.app/docs/model-architecture.md" last_updated: "2018-10-20" --- # Model Architecture URL: /docs/model-architecture LLM index: /llms.txt Description: Every neural network component in Angelo, from embedding to LM head. # Model Architecture Angelo is a **decoder-only transformer** that generates text by predicting the next token, one step at a time. The current implementation is `angelo-5m` - 1.8M parameters, 6 decoder layers, `d_model = 128`. --- ## End-to-End Flow ``` Input Text ↓ tokenizer/tokinizer.py → token IDs (integers) ↓ embeddings/embedding.py → token vectors (d_model = 128) ↓ transformer/block.py × 6 → contextualized hidden states (RoPE applied in attention) ↓ heads/lm_head.py → logits over vocab (5000 scores) ↓ utils/softmax.py → probabilities ↓ Next Token ``` --- ## 1 - Tokenizer **Files:** `compute/tokenizer/tokinizer.py`, `compute/tokenizer/vocabulary.py` The tokenizer converts raw text into **token IDs** - integers the model can process. Angelo uses a character/subword tokenizer with a 5,000-token vocabulary (set in `1.8m.yaml`). `vocabulary.py` manages the token ↔ integer mapping. `tokinizer.py` handles encoding (text → IDs) and decoding (IDs → text). ```python title="Concept" text = "The cat sleeps." ids = tokenizer.encode(text) # e.g. [464, 3797, 3993, 13] decoded = tokenizer.decode(ids) # "The cat sleeps." ``` --- ## 2 - Embedding Layer **Files:** `compute/embeddings/embedding.py` ### Token Embedding `embedding.py` contains a learned matrix of shape `(vocab_size, d_model)` - that is `(5000, 128)` for the current config. Each token ID is converted to a 128-dimensional dense vector by looking up its row in this matrix. ``` Token ID 464 → [-0.23, 0.81, 0.14, ..., 0.45] ← 128 numbers ``` ### Positional Encoding - RoPE Rotary Position Embeddings (RoPE) are applied within the attention layers, configured with `rope_theta: 10000.0`. RoPE encodes position by rotating query and key vectors in the attention computation. This means: - Position is relative, not absolute - The model generalizes better to longer sequences - No extra learnable parameters needed --- ## 3 - Transformer Block **Files:** `compute/transformer/block.py`, `compute/transformer/attention.py`, `compute/transformer/multi_head_attention.py`, `compute/transformer/feed_forward.py` The decoder is 6 identical blocks stacked in sequence. Each block is documented inline in `block.py`: ``` LayerNorm ↓ Masked Multi-Head Self-Attention ↓ Residual Addition (+) ↓ LayerNorm ↓ Feed-Forward Network ↓ Residual Addition (+) ↓ Output ``` ### Attention `attention.py` - core scaled dot-product attention: ``` Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V ``` `multi_head_attention.py` - splits the input into `n_heads = 4` parallel subspaces, runs attention independently in each, then concatenates and projects the results. The mask ensures each token can only attend to **previous** tokens (causal masking), which is what enables next-token prediction training. ### Feed-Forward Network `feed_forward.py` - each token passes through a two-layer MLP independently with `d_ff = 512`: ``` FFN(x) = (SiLU(xW₁) ⊙ xW₃) W₂ ``` This is the SwiGLU/SiLU variant. The gating (`⊙`) selectively amplifies useful information. `d_ff = 512` is 4× `d_model = 128`. `decoder.py` wires N blocks into the full decoder stack. --- ## 4 - Language Model Head **File:** `compute/heads/lm_head.py` The transformer produces 128-dimensional hidden states. The LM head converts those to **logits** - one score per vocabulary token. ``` Linear(d_model=128, out_features=vocab_size=5000) ``` This is a `128 → 5000` projection. Because `tie_word_embeddings: true` in the config, the LM head **shares weights** with the token embedding matrix - this is why the 1.8M total doesn't add extra parameters for the head. --- ## 5 - Utilities **Files:** `compute/utils/losses.py`, `compute/utils/optimizer.py`, `compute/utils/softmax.py` `softmax.py` - converts logits to probabilities `losses.py` - cross-entropy loss for next-token prediction `optimizer.py` - AdamW optimizer setup and learning rate scheduling --- ## Full Model Assembly **File:** `src/model/angelo-1.8m-v1.py` This file wires all `compute/` components into a single `AngeloGPT` class. The config dataclass in `src/model/config.py` maps from `1.8m.yaml` to Python objects, so the model is fully config-driven. --- ## Architecture Diagram (from docs-old) ``` Input Tokens │ ▼ ┌──────────────────────────┐ │ Token Embedding │ │ + RoPE │ ├──────────────────────────┤ │ ┌────────────────────┐ │ │ │ RMSNorm │ │ │ │ Self-Attention │ │ × 6 layers │ │ Residual + Norm │ │ │ │ SiLU FFN │ │ │ │ Residual │ │ │ └────────────────────┘ │ ├──────────────────────────┤ │ LM Head (tied) │ │ (→ 5000 logits) │ └──────────────────────────┘ │ ▼ Next Token ``` --- ## Current Status | Component | File | Status | | --------- | ---- | ------ | | Tokenizer | `compute/tokenizer/tokinizer.py` | Implemented | | Vocabulary | `compute/tokenizer/vocabulary.py` | Implemented | | Token Embedding | `compute/embeddings/embedding.py` | Implemented | | Positional Encoding (RoPE) | `compute/positional_encoding/` | Scaffolded | | Attention | `compute/transformer/attention.py` | Planned | | Multi-Head Attention | `compute/transformer/multi_head_attention.py` | Planned | | Feed-Forward (SiLU) | `compute/transformer/feed_forward.py` | Planned | | Transformer Block | `compute/transformer/block.py` | Planned | | Decoder Stack | `compute/transformer/decoder.py` | Planned | | LM Head (tied) | `compute/heads/lm_head.py` | Planned | | Full Model | `src/model/angelo-1.8m-v1.py` | Planned | | Training Pipeline | `lifecycle/training/` | Scaffolded | ## Sitemap See the full [sitemap](/sitemap.md) for all pages. Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md). Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).