Files
claude-skills-reference/engineering-team/playwright-pro/skills/fix/flaky-taxonomy.md
Alireza Rezvani d33d03da50 feat: add playwright-pro plugin — production-grade Playwright testing toolkit (#254)
Complete Claude Code plugin with:
- 9 skills (/pw:init, generate, review, fix, migrate, coverage, testrail, browserstack, report)
- 3 specialized agents (test-architect, test-debugger, migration-planner)
- 55 test case templates across 11 categories (auth, CRUD, checkout, search, forms, dashboard, settings, onboarding, notifications, API, accessibility)
- TestRail MCP server (TypeScript) — 8 tools for bidirectional sync
- BrowserStack MCP server (TypeScript) — 7 tools for cross-browser testing
- Smart hooks (auto-validate tests, auto-detect Playwright projects)
- 6 curated reference docs (golden rules, locators, assertions, fixtures, pitfalls, flaky tests)
- Leverages Claude Code built-ins (/batch, /debug, Explore subagent)
- Zero-config for core features; TestRail/BrowserStack via env vars
- Both TypeScript and JavaScript support throughout

Co-authored-by: Leo <leo@openclaw.ai>
2026-03-05 13:50:05 +01:00

135 lines
3.4 KiB
Markdown

# Flaky Test Taxonomy
## Decision Tree
```
Test is flaky
├── Fails locally with --repeat-each=20?
│ ├── YES → TIMING / ASYNC
│ │ ├── Missing await? → Add await
│ │ ├── waitForTimeout? → Replace with assertion
│ │ ├── Race condition? → Wait for specific event
│ │ └── Animation? → Wait for animation end or disable
│ │
│ └── NO → Continue...
├── Passes alone, fails in suite?
│ ├── YES → TEST ISOLATION
│ │ ├── Shared variable? → Make per-test
│ │ ├── Database state? → Reset per-test
│ │ ├── localStorage? → Clear in beforeEach
│ │ └── Cookie leak? → Use isolated contexts
│ │
│ └── NO → Continue...
├── Fails in CI, passes locally?
│ ├── YES → ENVIRONMENT
│ │ ├── Viewport? → Set explicit size
│ │ ├── Fonts? → Use Docker locally
│ │ ├── Timezone? → Use UTC everywhere
│ │ └── Network? → Mock external services
│ │
│ └── NO → INFRASTRUCTURE
│ ├── Browser crash? → Reduce workers
│ ├── OOM? → Limit parallel tests
│ ├── DNS? → Add retry config
│ └── File system? → Use unique temp dirs
```
## Common Fixes by Category
### Timing / Async
**Missing await:**
```typescript
// BAD — race condition
page.goto('/dashboard');
expect(page.getByText('Welcome')).toBeVisible();
// GOOD
await page.goto('/dashboard');
await expect(page.getByText('Welcome')).toBeVisible();
```
**Clicking before visible:**
```typescript
// BAD — element may not be ready
await page.getByRole('button', { name: 'Submit' }).click();
// GOOD — ensure visible first
const submitBtn = page.getByRole('button', { name: 'Submit' });
await expect(submitBtn).toBeVisible();
await submitBtn.click();
```
**Race with network:**
```typescript
// BAD — data might not be loaded
await page.goto('/users');
await expect(page.getByRole('table')).toBeVisible();
// GOOD — wait for API response
const responsePromise = page.waitForResponse('**/api/users');
await page.goto('/users');
await responsePromise;
await expect(page.getByRole('table')).toBeVisible();
```
### Test Isolation
**Shared state fix:**
```typescript
// BAD — tests share userId
let userId: string;
test('create', async () => { userId = '123'; });
test('read', async () => { /* uses userId */ });
// GOOD — each test is independent
test('read user', async ({ request }) => {
const response = await request.post('/api/users', { data: { name: 'Test' } });
const { id } = await response.json();
// Use id within this test
});
```
**localStorage cleanup:**
```typescript
test.beforeEach(async ({ page }) => {
await page.goto('/');
await page.evaluate(() => localStorage.clear());
});
```
### Environment
**Explicit viewport:**
```typescript
test.use({ viewport: { width: 1280, height: 720 } });
```
**Timezone-safe dates:**
```typescript
// BAD
expect(dateText).toBe('March 5, 2026');
// GOOD — timezone independent
expect(dateText).toMatch(/\d{1,2}\/\d{1,2}\/\d{4}/);
```
### Infrastructure
**Retry config:**
```typescript
// playwright.config.ts
export default defineConfig({
retries: process.env.CI ? 2 : 0,
workers: process.env.CI ? 2 : undefined,
});
```
**Increase timeout for CI:**
```typescript
test.setTimeout(60_000); // 60s for slow CI
```