---
title: "Overview"
description: "Angelo is a decoder-only LLM built entirely from scratch for educational purposes."
canonical_url: "https://angeloai.vercel.app/docs"
markdown_url: "https://angeloai.vercel.app/docs.md"
last_updated: "2018-10-20"
---

# Overview
URL: /docs
LLM index: /llms.txt
Description: Angelo is a decoder-only LLM built entirely from scratch for educational purposes.

# Overview

**Angelo** is a decoder-only language model (LLM) built from scratch for educational
purposes. It implements the same architectural family as GPT, LLaMA, DeepSeek, and Qwen
- every component from tokenizer to training pipeline is built, tested, and documented
here.

> This is not a production model. The goal is to understand how modern LLMs work
> at the subsystem level by implementing each piece from first principles.

---

## Project Evolution

This project evolved through three stages, each driven by a deeper understanding of how
LLMs actually work.

**Stage 1 - Encoder-Decoder Transformer**  
Started by studying the 2017 *Attention Is All You Need* encoder-decoder architecture.
Built out both the encoder and decoder sides to understand how they interact.

**Stage 2 - Embedding Layer**  
Realized a standalone embedding model was a prerequisite. Built a full tokenizer +
vocabulary + embedding matrix pipeline from scratch.

**Stage 3 - Decoder-Only Mini LLM (current)**  
Production models don't train embeddings in isolation - they learn everything jointly
through end-to-end training. The project shifted to building a miniature decoder-only
language model, the same paradigm used by virtually every modern chatbot.

---

## Quick Start

```bash title="terminal"
# Clone
git clone https://github.com/Atnatewoss/angelo-ai.git
cd angelo-ai

# Install
pip install -r core/requirements.txt

# Train the 1.8M model
python core/scripts/finetune.py \
    --config core/configs/model/1.8m.yaml \
    --data core/configs/data/default.yaml
```

---

## Current Model

The active implementation is `angelo-5m`, a 1.8M parameter miniature
decoder-only transformer defined in `core/configs/model/1.8m.yaml`.

| Property | Value |
| -------- | ----- |
| Architecture | Decoder-only transformer |
| Parameters | ~1.8M |
| Layers (N) | 6 |
| Attention heads | 4 |
| d\_model | 128 |
| d\_ff | 512 |
| Vocab size | 5000 |
| Max sequence length | 5000 |
| Normalization | RMSNorm (eps 1e-5) |
| Attention | Scaled dot-product (SDPA) + RoPE |
| Activation | SiLU (SwiGLU family) |
| LM Head | Tied to embedding matrix |

### Parameter breakdown

```
Embedding:  5000 × 128 = 640K
Per layer:  Q/K/V/O (4 × 128²) + FFN (2 × 128 × 512) = 197K
6 layers:   6 × 197K = 1.18M
Total:      ~1.8M  (LM head tied → no extra params)
```

---

## What's in the Docs

| Section | What it covers |
| ------- | -------------- |
| [Project Structure](/docs/project-structure) | Real directory layout and design rationale |
| [Model Architecture](/docs/model-architecture) | Every component, from embedding to LM head |
| [Training](/docs/training) | Pre-training, fine-tuning, checkpointing |
| [Data Pipeline](/docs/data-pipeline) | Tokenization, preprocessing, loading |
| [Evaluation](/docs/evaluation) | Benchmarks, scripts, results format |
| [Concepts](/docs/concepts) | Heads, logits, softmax - the core vocabulary |

---

## Transformer Architecture Families

Angelo implements the **decoder-only** family. For context, here are all three:

| Family | Reads | Generates | Used by |
| ------ | ----- | --------- | ------- |
| Encoder-only | Full sequence (bidirectional) | Embeddings / class labels | BERT, RoBERTa, DeBERTa |
| Encoder-Decoder | Full input, then generates | Sequences conditioned on input | T5, BART, mT5 |
| **Decoder-only** | **Previous tokens (causal)** | **Next token autoregressively** | **GPT, Llama, Claude, Qwen, DeepSeek** |

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).