---
title: "Transformer Types"
description: "Understanding Encoder, Decoder, and Encoder-Decoder architectures."
canonical_url: "https://angeloai.vercel.app/docs/concepts/01-transformer-types"
markdown_url: "https://angeloai.vercel.app/docs/concepts/01-transformer-types.md"
last_updated: "2018-10-20"
---

# Transformer Types
URL: /docs/concepts/01-transformer-types
LLM index: /llms.txt
Description: Understanding Encoder, Decoder, and Encoder-Decoder architectures.

# Transformer Types

The **Transformer** is a neural network architecture introduced in 2017. It revolutionized AI by replacing sequential processing (like RNNs and LSTMs) with **Attention**, allowing the model to process all tokens in parallel and directly understand long-range dependencies.

There are three main variations of the Transformer:

### 1. Encoder-Decoder Transformers
**Examples:** T5, BART, Original 2017 Model
Used for sequence-to-sequence tasks (e.g., translation, summarization). 
- The **Encoder** reads the input sequence bidirectionally and builds a deep contextual representation. 
- The **Decoder** generates the output one step at a time, looking at both the Encoder's output (via cross-attention) and what it has generated so far.

### 2. Encoder-Only Transformers
**Examples:** BERT, RoBERTa
Used for understanding text (e.g., classification, sentiment analysis, extractive Q&A). 
- It reads the entire sequence bidirectionally to build deep context for every token simultaneously. 
- Because it sees the "future" tokens during processing, it cannot generate text autoregressively.

### 3. Decoder-Only Transformers
**Examples:** GPT, Llama, Claude, DeepSeek, Angelo
Used for generative language modeling. 
- It predicts the *next* token by only looking at the *previous* tokens (enforced via causal masking). 
- Because it is simple, does not require cross-attention, and scales incredibly well, it has become the standard architecture for modern chatbots and LLMs.

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).