---
title: "Feed-Forward & Activations"
description: "Processing tokens individually using SwiGLU."
canonical_url: "https://angeloai.vercel.app/docs/concepts/05-feed-forward"
markdown_url: "https://angeloai.vercel.app/docs/concepts/05-feed-forward.md"
last_updated: "2018-10-20"
---

# Feed-Forward & Activations
URL: /docs/concepts/05-feed-forward
LLM index: /llms.txt
Description: Processing tokens individually using SwiGLU.

# Feed-Forward Networks & SwiGLU

After the Attention mechanism figures out *context* (how words relate to each other), the representation passes through a **Feed-Forward Network (FFN)**. 

The FFN acts as a memory bank and processing unit for individual tokens. While Attention moves information *between* tokens, the FFN processes the information *within* each token independently. It generally projects the vector to a much higher dimension (`d_ff`), applies an activation function, and projects it back down.

### SwiGLU (Swish-Gated Linear Unit)
Traditional neural networks use simple activation functions like ReLU (which just turns negative numbers to 0) or GELU. 

Modern LLMs like Angelo use **SwiGLU**. SwiGLU splits the input into two paths:
1. One path calculates values.
2. The other path calculates a "gate" (a percentage from 0 to 1). 

They are multiplied together. This gating mechanism allows the network to selectively amplify important features and completely suppress irrelevant ones, leading to significantly better performance and training stability.

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).