---
title: "Data Pipeline"
description: "Tokenization, preprocessing, and data loading for Angelo."
canonical_url: "https://angeloai.vercel.app/docs/data-pipeline"
markdown_url: "https://angeloai.vercel.app/docs/data-pipeline.md"
last_updated: "2018-10-20"
---

# Data Pipeline
URL: /docs/data-pipeline
LLM index: /llms.txt
Description: Tokenization, preprocessing, and data loading for Angelo.

# Data Pipeline

The data pipeline handles everything between raw text files and the
batches of token IDs that the model sees during training.

---

## Pipeline Overview

```
Raw Text Files
      ↓
  Deduplication        → remove near-duplicate documents
      ↓
  Quality Filtering    → language detection, content filtering
      ↓
  Tokenization (BPE)   → encode text to token IDs
      ↓
  Sequence Packing     → pack short sequences into fixed-length chunks
      ↓
  Sharding             → split into fixed-size files for streaming
      ↓
  DataLoader           → batched, prefetched tensors for training
```

---

## Tokenization

Angelo uses a **Byte-Pair Encoding (BPE)** tokenizer with a configurable
vocabulary size (default 50,257 - same as GPT-2).

BPE starts from individual bytes and iteratively merges the most frequent
pairs, building up a vocabulary of common subword units. This means the
tokenizer can handle any Unicode text without an "unknown token".

Tokenizer artifacts are saved alongside each checkpoint:

```
checkpoints/v1.0.0/
├── tokenizer.json      # BPE merge rules + vocabulary
└── vocabulary.json     # token → ID mapping
```

---

## Preprocessing

Inspired by the [Dolma](https://github.com/allenai/dolma) pipeline (used
for OLMo), preprocessing includes:

| Step | What it does |
| ---- | ------------ |
| **Deduplication** | MinHash-based near-duplicate removal across the full corpus |
| **Language filtering** | Keep only documents detected as English (or target language) |
| **Quality filtering** | Remove low-perplexity spam, boilerplate, and malformed text |
| **Toxicity filtering** | Heuristic filtering of harmful content categories |
| **Tokenization** | Encode surviving documents with the BPE tokenizer |
| **Sequence packing** | Concatenate short documents to fill `max_seq_len` without padding waste |
| **Sharding** | Write output as fixed-size binary shards for memory-efficient streaming |

---

## Data Loading

The `DataLoader` is configured for high-throughput GPU training:

- **Multi-worker loading** - configurable `num_workers` for parallel prefetching
- **Sequence packing** - no padding wasted; every position in every batch is a real token
- **Streaming** - shards are read sequentially; the full dataset need not fit in RAM
- **Pinned memory** - `pin_memory=True` for faster host-to-device transfer
- **Prefetching** - next batch prepared while GPU processes the current one

---

## Configuration

```yaml title="core/configs/data/default.yaml"
tokenizer:
  vocab_size: 50257
  path: checkpoints/v1.0.0/tokenizer.json

preprocessing:
  deduplicate: true
  language_filter: en
  quality_filter: true
  min_doc_length: 128

loader:
  num_workers: 4
  sequence_length: 2048
  batch_size: 8
  pin_memory: true
  prefetch_factor: 2
```

---

## Directory Structure

```
lifecycle/data/
└── cache/                  # Intermediate preprocessing outputs
```

```
core/src/lifecycle/data/    # Code
└── cache/
```

```
core/src/lifecycle/preprocessing/   # Preprocessing code
├── cleaning.py
└── deduplication.py
```

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).