---
title: "Training"
description: "How to train Angelo - pretrain, finetune, and checkpoint."
canonical_url: "https://angeloai.vercel.app/docs/training"
markdown_url: "https://angeloai.vercel.app/docs/training.md"
last_updated: "2018-10-20"
---

# Training
URL: /docs/training
LLM index: /llms.txt
Description: How to train Angelo - pretrain, finetune, and checkpoint.

# Training

Angelo's training is driven by a single entry point: `core/scripts/finetune.py`.
It supports both pre-training from scratch and fine-tuning from a checkpoint,
controlled by `--mode`.

---

## Running Training

```bash title="terminal"
# Pre-train from scratch
python core/scripts/finetune.py \
    --config core/configs/model/1.8m.yaml \
    --data core/configs/data/default.yaml \
    --mode pretrain

# Fine-tune from a checkpoint
python core/scripts/finetune.py \
    --config core/configs/model/1.8m.yaml \
    --data core/configs/data/default.yaml \
    --mode finetune \
    --resume checkpoints/v1.0.0/

# Resume a crashed run
python core/scripts/finetune.py \
    --config core/configs/model/1.8m.yaml \
    --data core/configs/data/default.yaml \
    --resume checkpoints/step-500/ \
    --gpu 0
```

### CLI flags

| Flag | Description | Default |
| ---- | ----------- | ------- |
| `--config` | Path to model YAML config | required |
| `--data` | Path to data YAML config | required |
| `--mode` | `pretrain` or `finetune` | `pretrain` |
| `--resume` | Checkpoint path to resume from | `None` |
| `--gpu` | GPU device index | `0` |

---

## Pre-Training

The model is trained on raw text using **next-token prediction** (causal
language modelling). At every position in the sequence the model predicts
the token that comes next; the loss is cross-entropy averaged over all
positions.

### Optimizer

**AdamW** - implemented in `compute/utils/optimizer.py`:

- Cosine learning rate schedule with linear warmup
- Weight decay on all non-bias, non-normalization parameters

### Loss

Cross-entropy loss between the model's logits and the true next tokens -
implemented in `compute/utils/losses.py`.

### Config

The only model config currently in the repo is `1.8m.yaml` (`angelo-5m`):

```yaml title="core/configs/model/1.8m.yaml"
model:
  name: "angelo-5m"
  architecture: "decoder-only"
  vocab_size: 5000
  max_seq_len: 5000
  n_layers: 6
  n_heads: 4
  d_model: 128
  d_ff: 512
  dropout: 0.1
  norm_type: "rmsnorm"
  norm_eps: 1.0e-5
  attention_type: "sdpa"
  rope: true
  rope_theta: 10000.0
  activation: "silu"
  tie_word_embeddings: true
```

---

## Fine-Tuning

Fine-tuning adapts a pre-trained checkpoint to a specific task or dataset.
The same `finetune.py` script handles this - pass `--mode finetune` and
`--resume` to load an existing checkpoint.

### Pipeline

`core/src/lifecycle/training/finetune.py` contains the fine-tuning loop.
`core/src/lifecycle/training/pretrain.py` contains the pre-training loop.

---

## Checkpoints

Checkpoints are saved to `checkpoints/` and contain the complete model
state needed to resume training or run inference:

```
checkpoints/
└── v1.0.0/
    ├── model_weights.bin      # Embedding + decoder + LM head weights
    ├── config.json            # Model architecture config
    ├── tokenizer.json         # Vocabulary and encoding rules
    ├── vocabulary.json        # Token → ID mapping
    └── training_metadata.json # Step, epoch, loss, training config
```

Because `tie_word_embeddings: true`, the LM head shares the embedding
matrix - the checkpoint doesn't store them separately.

---

## Text Generation

```bash title="terminal"
python core/scripts/generate.py \
    --checkpoint checkpoints/v1.0.0/ \
    --prompt "The meaning of life is"
```

Generation is handled by `core/src/lifecycle/inference/`.

---

## Implementation Status

| Component | File | Status |
| --------- | ---- | ------ |
| Training entry point | `core/scripts/finetune.py` | Implemented (scaffold) |
| Pre-training loop | `lifecycle/training/pretrain.py` | Scaffolded |
| Fine-tuning loop | `lifecycle/training/finetune.py` | Scaffolded |
| Optimizer | `compute/utils/optimizer.py` | Planned |
| Loss | `compute/utils/losses.py` | Planned |
| Inference loop | `lifecycle/inference/` | Scaffolded |

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).