---
title: "Evaluation"
description: "Running benchmarks and interpreting results for Angelo."
canonical_url: "https://angeloai.vercel.app/docs/evaluation"
markdown_url: "https://angeloai.vercel.app/docs/evaluation.md"
last_updated: "2018-10-20"
---

# Evaluation
URL: /docs/evaluation
LLM index: /llms.txt
Description: Running benchmarks and interpreting results for Angelo.

# Evaluation

Angelo includes a benchmark evaluation script at `core/scripts/evaluate.py`.
Evaluation results are stored under `evaluation/results/` (tracked in git;
raw outputs are gitignored).

---

## Running Evaluation

```bash title="terminal"
# Evaluate on specific benchmarks
python core/scripts/evaluate.py \
    --checkpoint checkpoints/v1.0.0/ \
    --benchmarks hellaswag,mmlu

# Evaluate on all supported benchmarks
python core/scripts/evaluate.py \
    --checkpoint checkpoints/v1.0.0/ \
    --benchmarks all

# Save results to disk
python core/scripts/evaluate.py \
    --checkpoint checkpoints/v1.0.0/ \
    --benchmarks all \
    --output evaluation/results/v1.0.0.json
```

---

## Supported Benchmarks

Targets the Open LLM Leaderboard suite:

| Benchmark | Task | Metric |
| --------- | ---- | ------ |
| HellaSwag | Common sense completion | Accuracy |
| MMLU | 57-subject academic knowledge | Accuracy |
| ARC-Challenge | Science reasoning | Accuracy |
| WinoGrande | Coreference resolution | Accuracy |
| TruthfulQA | Truthfulness | MC2 |
| GSM8K | Grade-school math | Accuracy |
| HumanEval | Code generation | Pass@1 |
| PIQA | Physical intuition | Accuracy |
| BoolQ | Reading comprehension | Accuracy |

---

## Results Format

```json title="evaluation/results/v1.0.0.json"
{
  "checkpoint": "checkpoints/v1.0.0",
  "model": "angelo-5m",
  "params": "1.8M",
  "results": {
    "hellaswag": { "score": null, "status": "not_implemented" },
    "mmlu":      { "score": null, "status": "not_implemented" }
  }
}
```

---

## Interpreting Scores

For a 1.8M parameter model, scores significantly below chance indicate
the model hasn't converged. Scores near random baseline (25–33% for
4-choice tasks) indicate early but real learning.

| Score range | What it means |
| ----------- | ------------- |
| < 25% | Below random - something is wrong |
| 25–35% | Near random - early training |
| 40–55% | Learning - comparable to small trained models |
| > 60% | Strong for this scale |

> Note: At 1.8M parameters, Angelo is far smaller than any Open LLM
> Leaderboard model. The benchmarks here are used as learning tools,
> not competitive comparisons.

---

## Implementation Status

| Component | File | Status |
| --------- | ---- | ------ |
| Evaluation entry point | `core/scripts/evaluate.py` | Scaffold |
| Benchmark harness | `lifecycle/evaluation/` | Scaffolded |
| Results format | `evaluation/results/` | Planned |

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.
Docs-scoped sitemap: [/docs/sitemap.md](/docs/sitemap.md).
Well-known sitemap: [/.well-known/sitemap.md](/.well-known/sitemap.md).