2c72babd52
- mkdocs.yml: Material theme with dark/light mode, search, tabs, sitemap - scripts/generate-docs.py: auto-generates docs from all SKILL.md files - docs/index.md: landing page with domain overview and quick install - docs/getting-started.md: installation guide for Claude Code, Codex, OpenClaw - docs/skills/: 170 skill pages + 9 domain index pages - .github/workflows/static.yml: MkDocs build + GitHub Pages deploy - .gitignore: exclude site/ build output Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
118 lines
3.2 KiB
Markdown
118 lines
3.2 KiB
Markdown
---
|
|
title: "Fix Failing or Flaky Tests"
|
|
description: "Fix Failing or Flaky Tests - Claude Code skill from the Engineering - Core domain."
|
|
---
|
|
|
|
# Fix Failing or Flaky Tests
|
|
|
|
**Domain:** Engineering - Core | **Skill:** `fix` | **Source:** [`engineering-team/playwright-pro/skills/fix/SKILL.md`](https://github.com/alirezarezvani/claude-skills/tree/main/engineering-team/playwright-pro/skills/fix/SKILL.md)
|
|
|
|
---
|
|
|
|
|
|
# Fix Failing or Flaky Tests
|
|
|
|
Diagnose and fix a Playwright test that fails or passes intermittently using a systematic taxonomy.
|
|
|
|
## Input
|
|
|
|
`$ARGUMENTS` contains:
|
|
- A test file path: `e2e/login.spec.ts`
|
|
- A test name: `"should redirect after login"`
|
|
- A description: `"the checkout test fails in CI but passes locally"`
|
|
|
|
## Steps
|
|
|
|
### 1. Reproduce the Failure
|
|
|
|
Run the test to capture the error:
|
|
|
|
```bash
|
|
npx playwright test <file> --reporter=list
|
|
```
|
|
|
|
If the test passes, it's likely flaky. Run burn-in:
|
|
|
|
```bash
|
|
npx playwright test <file> --repeat-each=10 --reporter=list
|
|
```
|
|
|
|
If it still passes, try with parallel workers:
|
|
|
|
```bash
|
|
npx playwright test --fully-parallel --workers=4 --repeat-each=5
|
|
```
|
|
|
|
### 2. Capture Trace
|
|
|
|
Run with full tracing:
|
|
|
|
```bash
|
|
npx playwright test <file> --trace=on --retries=0
|
|
```
|
|
|
|
Read the trace output. Use `/debug` to analyze trace files if available.
|
|
|
|
### 3. Categorize the Failure
|
|
|
|
Load `flaky-taxonomy.md` from this skill directory.
|
|
|
|
Every failing test falls into one of four categories:
|
|
|
|
| Category | Symptom | Diagnosis |
|
|
|---|---|---|
|
|
| **Timing/Async** | Fails intermittently everywhere | `--repeat-each=20` reproduces locally |
|
|
| **Test Isolation** | Fails in suite, passes alone | `--workers=1 --grep "test name"` passes |
|
|
| **Environment** | Fails in CI, passes locally | Compare CI vs local screenshots/traces |
|
|
| **Infrastructure** | Random, no pattern | Error references browser internals |
|
|
|
|
### 4. Apply Targeted Fix
|
|
|
|
**Timing/Async:**
|
|
- Replace `waitForTimeout()` with web-first assertions
|
|
- Add `await` to missing Playwright calls
|
|
- Wait for specific network responses before asserting
|
|
- Use `toBeVisible()` before interacting with elements
|
|
|
|
**Test Isolation:**
|
|
- Remove shared mutable state between tests
|
|
- Create test data per-test via API or fixtures
|
|
- Use unique identifiers (timestamps, random strings) for test data
|
|
- Check for database state leaks
|
|
|
|
**Environment:**
|
|
- Match viewport sizes between local and CI
|
|
- Account for font rendering differences in screenshots
|
|
- Use `docker` locally to match CI environment
|
|
- Check for timezone-dependent assertions
|
|
|
|
**Infrastructure:**
|
|
- Increase timeout for slow CI runners
|
|
- Add retries in CI config (`retries: 2`)
|
|
- Check for browser OOM (reduce parallel workers)
|
|
- Ensure browser dependencies are installed
|
|
|
|
### 5. Verify the Fix
|
|
|
|
Run the test 10 times to confirm stability:
|
|
|
|
```bash
|
|
npx playwright test <file> --repeat-each=10 --reporter=list
|
|
```
|
|
|
|
All 10 must pass. If any fail, go back to step 3.
|
|
|
|
### 6. Prevent Recurrence
|
|
|
|
Suggest:
|
|
- Add to CI with `retries: 2` if not already
|
|
- Enable `trace: 'on-first-retry'` in config
|
|
- Add the fix pattern to project's test conventions doc
|
|
|
|
## Output
|
|
|
|
- Root cause category and specific issue
|
|
- The fix applied (with diff)
|
|
- Verification result (10/10 passes)
|
|
- Prevention recommendation
|