---
title: "The Origin Story"
description: "How the Angelo architecture evolved from an Encoder-Decoder into a modern Decoder-only LLM."
canonical_url: "https://angeloai.vercel.app/docs/blog"
markdown_url: "https://angeloai.vercel.app/docs/blog.md"
last_updated: "2018-10-20"
---

# The Origin Story
URL: /docs/blog
LLM index: /llms.txt
Description: How the Angelo architecture evolved from an Encoder-Decoder into a modern Decoder-only LLM.

# The Origin Story

This project did not start with a clear roadmap. It evolved organically through three distinct stages, each driven by a deeper understanding of how modern language models actually work.

### Stage 1 - Encoder-Decoder Transformer

The project originally began as an attempt to understand the **Encoder-Decoder Transformer** architecture introduced in the 2017 paper [*Attention Is All You Need*](https://arxiv.org/abs/1706.03762).

This architecture consists of two symmetric halves:

```
Input ──► [ Encoder ] ──► Context ──► [ Decoder ] ──► Output
```

- The **encoder** reads the full input sequence and produces contextual representations.
- The **decoder** generates the output sequence one token at a time, attending to both the encoder's output and its own previously generated tokens.

It was primarily designed for **sequence-to-sequence tasks** such as machine translation, summarization, and text generation conditioned on an input.

### Stage 2 - Embedding Layer

While implementing the encoder manually, it became clear that a working embedding layer was a prerequisite. The project shifted toward building an **embedding model completely from scratch**, including:

- **Tokenizer** - BPE-based tokenization pipeline
- **Vocabulary** - learned merge operations and token-to-index mapping
- **Token IDs** - integer representations of subword units
- **Embedding matrix** - trainable dense vector lookup
- **Training pipeline** - data loading, optimization, and checkpointing

Embeddings are the foundation of every transformer. They convert discrete tokens into continuous vector representations that the model can process. Without them, no downstream component - attention, feed-forward networks, or otherwise - can function.

### Stage 3 - Mini Decoder-Only LLM

While studying how modern LLMs are actually trained, a key insight emerged: **production models do not train embeddings separately**. Instead, embeddings are learned jointly with the rest of the transformer through end-to-end training (and the separate embedding models that existed were done before transformers were introduced).

This realization shifted the project toward building a **miniature decoder-only language model** from scratch - the same architecture family used by GPT, Claude, Llama, and virtually every modern chatbot.

Without knowing it at the time, this was the natural direction. The original 2017 paper's encoder-decoder design gave way to a simpler, more scalable paradigm that now dominates the field. Angelo is the result of that evolution.

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).