---
title: "Project Structure"
description: "The real directory layout of Angelo and why it's organized this way."
canonical_url: "https://angeloai.vercel.app/docs/project-structure"
markdown_url: "https://angeloai.vercel.app/docs/project-structure.md"
last_updated: "2018-10-20"
---

# Project Structure
URL: /docs/project-structure
LLM index: /llms.txt
Description: The real directory layout of Angelo and why it's organized this way.

# Project Structure

Angelo follows the same separation of concerns used in production LLM repos
(nanoGPT, LLaMA, LitGPT, OLMo, Megatron-LM). The key split is:

| Layer | What it is |
| ----- | ---------- |
| `compute/` | The actual neural network math - model components |
| `lifecycle/` | End-to-end ML pipeline: data, preprocessing, training, evaluation, inference, deployment |
| `model/` | Orchestrates compute components into a full model assembly |
| `scripts/` | Entry-point commands that run things end-to-end |

---

## Real Directory Layout

```
angelo-ai/
│
├── core/
│   │
│   ├── configs/
│   │   ├── model/
│   │   │   └── 1.8m.yaml          # Active model config (~1.8M params)
│   │   ├── training/
│   │   └── data/
│   │
│   ├── scripts/
│   │   ├── finetune.py            # Training entry point (pretrain + finetune)
│   │   ├── pretrain.py
│   │   ├── generate.py
│   │   └── evaluate.py
│   │
│   └── src/
│       │
│       ├── compute/               # Neural network components
│       │   ├── tokenizer/
│       │   │   ├── tokinizer.py
│       │   │   └── vocabulary.py
│       │   ├── embeddings/
│       │   │   └── embedding.py
│       │   ├── transformer/
│       │   │   ├── attention.py
│       │   │   ├── multi_head_attention.py
│       │   │   ├── feed_forward.py
│       │   │   ├── block.py
│       │   │   ├── decoder.py
│       │   │   └── utils.py
│       │   ├── heads/
│       │   │   └── lm_head.py
│       │   └── utils/
│       │       ├── losses.py
│       │       ├── optimizer.py
│       │       └── softmax.py
│       │
│       ├── model/
│       │   ├── angelo-1.8m-v1.py    # Full model assembly
│       │   └── config.py
│       │
│       ├── lifecycle/
│       │   ├── data/
│       │   │   └── cache/
│       │   ├── preprocessing/
│       │   │   ├── cleaning.py
│       │   │   └── deduplication.py
│       │   ├── training/
│       │   │   ├── pretrain.py
│       │   │   └── finetune.py
│       │   ├── evaluation/
│       │   │   ├── bias.py, coding.py, cost.py, hallucination.py
│       │   │   ├── human_preference.py, language_benchmarks.py
│       │   │   ├── latency.py, math.py, memory.py
│       │   │   ├── reasoning.py, safety.py
│       │   ├── inference/
│       │   │   └── inference.py
│       │   └── deployment/
│       │       └── deploy.py
│       │
│       └── main.py
│
├── checkpoints/
├── evaluation/
├── docs/                          # This documentation site
└── README.md
```

---

## Why This Structure?

### `compute/` - the neural network itself

These are the pure mathematical components. Each subfolder is one
conceptual piece of the model:

```
tokenizer/      → converts text into integers the model processes
embeddings/     → maps token IDs to dense vector representations
transformer/    → the attention + FFN blocks that build understanding
heads/          → converts hidden states into task outputs (LM head)
utils/          → shared math: losses, optimizer, softmax
```

### Why is `heads/` separate from `transformer/`?

The transformer is **task-agnostic** - the same decoder blocks can be
re-used for next-token prediction, classification, or reward modelling.
Only the head changes:

```
heads/
└── lm_head.py     # Linear(d_model → vocab_size) - language modelling
```

### `lifecycle/` - the ML lifecycle pipeline

Each subdirectory covers one stage of the model lifecycle:

```
data/            → data storage and caching
preprocessing/   → cleaning, deduplication, quality filtering
training/        → pretrain and finetune loops
evaluation/      → benchmarks (bias, coding, math, reasoning, safety, etc.)
inference/       → generation loop (forward pass → sample → repeat)
deployment/      → model serving and deployment
```

### `model/` - assembles the full model

`angelo-1.8m-v1.py` wires all the `compute/` components together into a
single `AngeloGPT` model class. `config.py` holds the dataclass that
maps from the YAML config to Python objects.

### `scripts/` - commands that run things

```
finetune.py   → training entry point (handles --mode pretrain|finetune)
pretrain.py   → pretrain-specific runner
generate.py   → text generation with a saved checkpoint
evaluate.py   → benchmark evaluation harness
```

---

## The Active Config

`core/configs/model/1.8m.yaml` is the only model config currently in the
repo. It defines `angelo-5m`, a 1.8M parameter miniature model used for
development and testing:

```yaml title="core/configs/model/1.8m.yaml"
model:
  name: "angelo-5m"
  architecture: "decoder-only"
  vocab_size: 5000
  max_seq_len: 5000
  n_layers: 6
  n_heads: 4
  d_model: 128
  d_ff: 512
  dropout: 0.1
  norm_type: "rmsnorm"
  attention_type: "sdpa"
  rope: true
  rope_theta: 10000.0
  activation: "silu"
  tie_word_embeddings: true
```

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).