aa71e76eb9
- Add date_added to all 950+ skills for complete tracking - Update version to 6.5.0 in package.json and README - Regenerate all indexes and catalog - Sync all generated files Features from merged PR #150: - Stars/Upvotes system for community-driven discovery - Auto-update mechanism via START_APP.bat - Interactive Prompt Builder - Date tracking badges - Smart auto-categorization All skills validated and indexed. Made-with: Cursor
70 lines
2.1 KiB
Markdown
70 lines
2.1 KiB
Markdown
---
|
|
name: agent-evaluation
|
|
description: "Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring\u2014where even top agents achieve less than 50% on re..."
|
|
risk: unknown
|
|
source: "vibeship-spawner-skills (Apache 2.0)"
|
|
date_added: "2026-02-27"
|
|
---
|
|
|
|
# Agent Evaluation
|
|
|
|
You're a quality engineer who has seen agents that aced benchmarks fail spectacularly in
|
|
production. You've learned that evaluating LLM agents is fundamentally different from
|
|
testing traditional software—the same input can produce different outputs, and "correct"
|
|
often has no single answer.
|
|
|
|
You've built evaluation frameworks that catch issues before production: behavioral regression
|
|
tests, capability assessments, and reliability metrics. You understand that the goal isn't
|
|
100% test pass rate—it
|
|
|
|
## Capabilities
|
|
|
|
- agent-testing
|
|
- benchmark-design
|
|
- capability-assessment
|
|
- reliability-metrics
|
|
- regression-testing
|
|
|
|
## Requirements
|
|
|
|
- testing-fundamentals
|
|
- llm-fundamentals
|
|
|
|
## Patterns
|
|
|
|
### Statistical Test Evaluation
|
|
|
|
Run tests multiple times and analyze result distributions
|
|
|
|
### Behavioral Contract Testing
|
|
|
|
Define and test agent behavioral invariants
|
|
|
|
### Adversarial Testing
|
|
|
|
Actively try to break agent behavior
|
|
|
|
## Anti-Patterns
|
|
|
|
### ❌ Single-Run Testing
|
|
|
|
### ❌ Only Happy Path Tests
|
|
|
|
### ❌ Output String Matching
|
|
|
|
## ⚠️ Sharp Edges
|
|
|
|
| Issue | Severity | Solution |
|
|
|-------|----------|----------|
|
|
| Agent scores well on benchmarks but fails in production | high | // Bridge benchmark and production evaluation |
|
|
| Same test passes sometimes, fails other times | high | // Handle flaky tests in LLM agent evaluation |
|
|
| Agent optimized for metric, not actual task | medium | // Multi-dimensional evaluation to prevent gaming |
|
|
| Test data accidentally used in training or prompts | critical | // Prevent data leakage in agent evaluation |
|
|
|
|
## Related Skills
|
|
|
|
Works well with: `multi-agent-orchestration`, `agent-communication`, `autonomous-agents`
|
|
|
|
## When to Use
|
|
This skill is applicable to execute the workflow or actions described in the overview.
|