---
title: "Normalization & Residuals"
description: "Stabilizing deep networks using RMSNorm and Skip Connections."
canonical_url: "https://angeloai.vercel.app/docs/concepts/06-normalization"
markdown_url: "https://angeloai.vercel.app/docs/concepts/06-normalization.md"
last_updated: "2018-10-20"
---

# Normalization & Residuals
URL: /docs/concepts/06-normalization
LLM index: /llms.txt
Description: Stabilizing deep networks using RMSNorm and Skip Connections.

# Normalization & Residual Connections

### Residual Connections (Skip Connections)
Neural networks suffer from the "vanishing gradient" problem-as the network gets deeper, the learning signal gets too weak by the time it reaches the early layers during training. 

Residual connections solve this by taking the input of a layer and adding it directly to the output: 
`Output = Layer(x) + x`

This creates an "express lane" for gradients to flow uninterrupted through the entire network, allowing for models with hundreds of layers.

### RMSNorm (Root Mean Square Normalization)
As data flows through a deep network, its mathematical scale can grow too large or shrink too small, causing instability. Normalization keeps the numbers grounded.

- Older models used **LayerNorm**, which centers the data (subtracts the mean) and scales it (divides by the variance).
- Modern models (including Angelo and Llama) use **RMSNorm**, which only scales the data (divides by the Root Mean Square) without subtracting the mean. It is computationally faster and performs just as well as LayerNorm.

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).