chore: sync cross-platform indexes, regenerate docs, fix plugin.json counts
- Codex CLI: 174 skills synced, 11 new symlinks - Gemini CLI: 262 items synced, 11 new - engineering plugin.json: 33 → 35 skills - engineering-team plugin.json: 28 → 29 skills - Docs regenerated: 261 pages (214 skills + 25 agents + 22 commands) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Reza Rezvani
@@ -453,6 +453,18 @@ Provide these details for architecture design:
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
| Skill | Relationship |
|
||||
|-------|-------------|
|
||||
| `engineering-team/aws-solution-architect` | AWS equivalent — same 6-step workflow, different services |
|
||||
| `engineering-team/gcp-cloud-architect` | GCP equivalent — completes the cloud trifecta |
|
||||
| `engineering-team/senior-devops` | Broader DevOps scope — pipelines, monitoring, containerization |
|
||||
| `engineering/terraform-patterns` | IaC implementation — use for Terraform modules targeting Azure |
|
||||
| `engineering/ci-cd-pipeline-builder` | Pipeline construction — automates Azure DevOps and GitHub Actions |
|
||||
|
||||
---
|
||||
|
||||
## Reference Documentation
|
||||
|
||||
| Document | Contents |
|
||||
|
||||
@@ -420,6 +420,32 @@ Provide these details for architecture design:
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
| Anti-Pattern | Why It Fails | Better Approach |
|
||||
|---|---|---|
|
||||
| Using default VPC for production | No isolation, shared firewall rules | Create custom VPC with private subnets |
|
||||
| Over-provisioning GKE node pools | Wasted cost on idle capacity | Use GKE Autopilot or cluster autoscaler |
|
||||
| Storing secrets in environment variables | Visible in Cloud Console, logs | Use Secret Manager with Workload Identity |
|
||||
| Ignoring sustained use discounts | Missing 20-30% automatic savings | Right-size VMs for consistent baseline usage |
|
||||
| Single-region deployment for SaaS | One region outage = full downtime | Multi-region with Cloud Load Balancing |
|
||||
| BigQuery on-demand for heavy workloads | Unpredictable costs at scale | Use BigQuery slots (flat-rate) for consistent workloads |
|
||||
| Running Cloud Functions for long tasks | 9-minute timeout, cold starts | Use Cloud Run for tasks > 60 seconds |
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
| Skill | Relationship |
|
||||
|-------|-------------|
|
||||
| `engineering-team/aws-solution-architect` | AWS equivalent — same 6-step workflow, different services |
|
||||
| `engineering-team/azure-cloud-architect` | Azure equivalent — completes the cloud trifecta |
|
||||
| `engineering-team/senior-devops` | Broader DevOps scope — pipelines, monitoring, containerization |
|
||||
| `engineering/terraform-patterns` | IaC implementation — use for Terraform modules targeting GCP |
|
||||
| `engineering/ci-cd-pipeline-builder` | Pipeline construction — automates Cloud Build and deployment |
|
||||
|
||||
---
|
||||
|
||||
## Reference Documentation
|
||||
|
||||
| Document | Contents |
|
||||
|
||||
@@ -71,353 +71,71 @@ python scripts/vulnerability_scanner.py --target web --scope full
|
||||
python scripts/vulnerability_scanner.py --target api --scope quick --json
|
||||
```
|
||||
|
||||
### A01:2021 — Broken Access Control
|
||||
### Quick Reference
|
||||
|
||||
**Test Procedures:**
|
||||
1. Attempt horizontal privilege escalation: access another user's resources by changing IDs
|
||||
2. Test vertical escalation: access admin endpoints with regular user tokens
|
||||
3. Verify CORS configuration — check `Access-Control-Allow-Origin` for wildcards
|
||||
4. Test forced browsing to admin pages (`/admin`, `/api/admin`, `/debug`)
|
||||
5. Modify JWT claims (`role`, `is_admin`) and replay tokens
|
||||
|
||||
**What to Look For:**
|
||||
- Missing authorization checks on API endpoints
|
||||
- Predictable resource IDs (sequential integers vs. UUIDs)
|
||||
- Client-side only access controls (hidden UI elements without server checks)
|
||||
- CORS misconfigurations allowing arbitrary origins
|
||||
|
||||
### A02:2021 — Cryptographic Failures
|
||||
|
||||
**Test Procedures:**
|
||||
1. Check TLS version — reject anything below TLS 1.2
|
||||
2. Verify password hashing: bcrypt/scrypt/argon2 with adequate cost factor
|
||||
3. Look for sensitive data in URLs (tokens in query params get logged)
|
||||
4. Check for hardcoded encryption keys in source code
|
||||
5. Test for weak random number generation (Math.random() for tokens)
|
||||
|
||||
**What to Look For:**
|
||||
- MD5/SHA1 used for password hashing
|
||||
- Secrets in environment variables without encryption at rest
|
||||
- Missing `Strict-Transport-Security` header
|
||||
- Self-signed certificates in production
|
||||
|
||||
### A03:2021 — Injection
|
||||
|
||||
**Test Procedures:**
|
||||
1. SQL injection: test all input fields with `' OR 1=1--` and time-based payloads
|
||||
2. NoSQL injection: test with `{"$gt": ""}` and `{"$ne": null}` in JSON bodies
|
||||
3. Command injection: test inputs with `; whoami` and backtick substitution
|
||||
4. LDAP injection: test with `*)(uid=*))(|(uid=*`
|
||||
5. Template injection: test with `{{7*7}}` and `${7*7}`
|
||||
|
||||
**What to Look For:**
|
||||
- String concatenation in SQL queries
|
||||
- User input passed to `eval()`, `exec()`, `os.system()`
|
||||
- Unparameterized ORM queries
|
||||
- Template engines rendering user input without sandboxing
|
||||
|
||||
### A04:2021 — Insecure Design
|
||||
|
||||
**Test Procedures:**
|
||||
1. Review business logic flows for abuse scenarios (e.g., negative quantities in carts)
|
||||
2. Check rate limiting on sensitive operations (login, password reset, OTP)
|
||||
3. Test multi-step flows for state manipulation (skip payment step)
|
||||
4. Verify security questions aren't guessable
|
||||
|
||||
**What to Look For:**
|
||||
- Missing rate limits on authentication endpoints
|
||||
- Business logic that trusts client-side calculations
|
||||
- Lack of account lockout after failed attempts
|
||||
- Missing CAPTCHA on public-facing forms
|
||||
|
||||
### A05:2021 — Security Misconfiguration
|
||||
|
||||
**Test Procedures:**
|
||||
1. Check for default credentials on admin panels
|
||||
2. Verify unnecessary HTTP methods are disabled (TRACE, DELETE on public endpoints)
|
||||
3. Check error handling — stack traces should never leak to users
|
||||
4. Review HTTP security headers (CSP, X-Frame-Options, X-Content-Type-Options)
|
||||
5. Check directory listing is disabled
|
||||
|
||||
**What to Look For:**
|
||||
- Debug mode enabled in production
|
||||
- Default admin:admin credentials
|
||||
- Verbose error messages with stack traces
|
||||
- Missing security headers
|
||||
|
||||
### A06:2021 — Vulnerable and Outdated Components
|
||||
|
||||
**Test Procedures:**
|
||||
1. Run dependency audit against known CVE databases
|
||||
2. Check for end-of-life frameworks and libraries
|
||||
3. Verify transitive dependency versions
|
||||
4. Check for known vulnerable versions (e.g., Log4j 2.0-2.14.1)
|
||||
| # | Category | Key Tests |
|
||||
|---|----------|-----------|
|
||||
| A01 | Broken Access Control | IDOR, vertical escalation, CORS, JWT claim manipulation, forced browsing |
|
||||
| A02 | Cryptographic Failures | TLS version, password hashing, hardcoded keys, weak PRNG |
|
||||
| A03 | Injection | SQLi, NoSQLi, command injection, template injection, XSS |
|
||||
| A04 | Insecure Design | Rate limiting, business logic abuse, multi-step flow bypass |
|
||||
| A05 | Security Misconfiguration | Default credentials, debug mode, security headers, directory listing |
|
||||
| A06 | Vulnerable Components | Dependency audit (npm/pip/go), EOL checks, known CVEs |
|
||||
| A07 | Auth Failures | Brute force, session cookie flags, session invalidation, MFA bypass |
|
||||
| A08 | Integrity Failures | Unsafe deserialization, SRI checks, CI/CD pipeline integrity |
|
||||
| A09 | Logging Failures | Auth event logging, sensitive data in logs, alerting thresholds |
|
||||
| A10 | SSRF | Internal IP access, cloud metadata endpoints, DNS rebinding |
|
||||
|
||||
```bash
|
||||
# Audit a package manifest
|
||||
# Audit dependencies
|
||||
python scripts/dependency_auditor.py --file package.json --severity high
|
||||
python scripts/dependency_auditor.py --file requirements.txt --json
|
||||
```
|
||||
|
||||
### A07:2021 — Identification and Authentication Failures
|
||||
|
||||
**Test Procedures:**
|
||||
1. Test brute force protection on login endpoints
|
||||
2. Check password policy enforcement (minimum length, complexity)
|
||||
3. Verify session invalidation on logout and password change
|
||||
4. Test "remember me" token security (HttpOnly, Secure, SameSite flags)
|
||||
5. Check multi-factor authentication bypass paths
|
||||
|
||||
**What to Look For:**
|
||||
- Sessions that persist after logout
|
||||
- Missing `HttpOnly` and `Secure` flags on session cookies
|
||||
- Password reset tokens that don't expire
|
||||
- Username enumeration via different error messages
|
||||
|
||||
### A08:2021 — Software and Data Integrity Failures
|
||||
|
||||
**Test Procedures:**
|
||||
1. Check for unsigned updates or deployment artifacts
|
||||
2. Verify CI/CD pipeline integrity (signed commits, protected branches)
|
||||
3. Test deserialization endpoints with crafted payloads
|
||||
4. Check for SRI (Subresource Integrity) on CDN-loaded scripts
|
||||
|
||||
**What to Look For:**
|
||||
- Unsafe deserialization of user input (pickle, Java serialization)
|
||||
- Missing integrity checks on downloaded artifacts
|
||||
- CI/CD pipelines running untrusted code
|
||||
- CDN scripts without SRI hashes
|
||||
|
||||
### A09:2021 — Security Logging and Monitoring Failures
|
||||
|
||||
**Test Procedures:**
|
||||
1. Verify authentication events are logged (success and failure)
|
||||
2. Check that logs don't contain sensitive data (passwords, tokens, PII)
|
||||
3. Test alerting thresholds (do 50 failed logins trigger an alert?)
|
||||
4. Verify log integrity — can an attacker tamper with logs?
|
||||
|
||||
**What to Look For:**
|
||||
- Missing audit trail for admin actions
|
||||
- Passwords or tokens appearing in logs
|
||||
- No alerting on suspicious patterns
|
||||
- Logs stored without integrity protection
|
||||
|
||||
### A10:2021 — Server-Side Request Forgery (SSRF)
|
||||
|
||||
**Test Procedures:**
|
||||
1. Test URL input fields with internal addresses (`http://169.254.169.254/` for cloud metadata)
|
||||
2. Check for open redirect chains that reach internal services
|
||||
3. Test with DNS rebinding payloads
|
||||
4. Verify allowlist validation on outbound requests
|
||||
|
||||
**What to Look For:**
|
||||
- User-controlled URLs passed to `fetch()`, `requests.get()`, `curl`
|
||||
- Missing allowlist on outbound HTTP requests
|
||||
- Ability to reach cloud metadata endpoints (AWS, GCP, Azure)
|
||||
- PDF generators or screenshot services that fetch arbitrary URLs
|
||||
See [owasp_top_10_checklist.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering-team/security-pen-testing/references/owasp_top_10_checklist.md) for detailed test procedures, code patterns to detect, remediation steps, and CVSS scoring guidance for each category.
|
||||
|
||||
---
|
||||
|
||||
## Static Analysis
|
||||
|
||||
### CodeQL Custom Rules
|
||||
**Recommended tools:** CodeQL (custom queries for project-specific patterns), Semgrep (rule-based scanning with auto-fix), ESLint security plugins (`eslint-plugin-security`, `eslint-plugin-no-unsanitized`).
|
||||
|
||||
Write custom CodeQL queries for project-specific vulnerability patterns:
|
||||
Key patterns to detect: SQL injection via string concatenation, hardcoded JWT secrets, unsafe YAML/pickle deserialization, missing security middleware (e.g., Express without Helmet).
|
||||
|
||||
```ql
|
||||
/**
|
||||
* Detect SQL injection via string concatenation
|
||||
*/
|
||||
import python
|
||||
import semmle.python.dataflow.new.DataFlow
|
||||
|
||||
from Call call, StringFormatting fmt
|
||||
where
|
||||
call.getFunc().getName() = "execute" and
|
||||
fmt = call.getArg(0) and
|
||||
exists(DataFlow::Node source |
|
||||
source.asExpr() instanceof Name and
|
||||
DataFlow::localFlow(source, DataFlow::exprNode(fmt.getAnOperand()))
|
||||
)
|
||||
select call, "Potential SQL injection: user input flows into execute()"
|
||||
```
|
||||
|
||||
### Semgrep Custom Rules
|
||||
|
||||
Create project-specific Semgrep rules:
|
||||
|
||||
```yaml
|
||||
rules:
|
||||
- id: hardcoded-jwt-secret
|
||||
pattern: |
|
||||
jwt.encode($PAYLOAD, "...", ...)
|
||||
message: "JWT signed with hardcoded secret"
|
||||
severity: ERROR
|
||||
languages: [python]
|
||||
|
||||
- id: unsafe-yaml-load
|
||||
pattern: yaml.load($DATA)
|
||||
fix: yaml.safe_load($DATA)
|
||||
message: "Use yaml.safe_load() to prevent arbitrary code execution"
|
||||
severity: WARNING
|
||||
languages: [python]
|
||||
|
||||
- id: express-no-helmet
|
||||
pattern: |
|
||||
const app = express();
|
||||
...
|
||||
app.listen(...)
|
||||
pattern-not: |
|
||||
const app = express();
|
||||
...
|
||||
app.use(helmet(...));
|
||||
...
|
||||
app.listen(...)
|
||||
message: "Express app missing helmet middleware for security headers"
|
||||
severity: WARNING
|
||||
languages: [javascript, typescript]
|
||||
```
|
||||
|
||||
### ESLint Security Plugins
|
||||
|
||||
Recommended configuration:
|
||||
|
||||
```json
|
||||
{
|
||||
"plugins": ["security", "no-unsanitized"],
|
||||
"extends": ["plugin:security/recommended"],
|
||||
"rules": {
|
||||
"security/detect-object-injection": "error",
|
||||
"security/detect-non-literal-regexp": "warn",
|
||||
"security/detect-unsafe-regex": "error",
|
||||
"security/detect-buffer-noassert": "error",
|
||||
"security/detect-eval-with-expression": "error",
|
||||
"no-unsanitized/method": "error",
|
||||
"no-unsanitized/property": "error"
|
||||
}
|
||||
}
|
||||
```
|
||||
See [attack_patterns.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering-team/security-pen-testing/references/attack_patterns.md) for code patterns and detection payloads across injection types.
|
||||
|
||||
---
|
||||
|
||||
## Dependency Vulnerability Scanning
|
||||
|
||||
### Ecosystem-Specific Commands
|
||||
**Ecosystem commands:** `npm audit`, `pip audit`, `govulncheck ./...`, `bundle audit check`
|
||||
|
||||
**CVE Triage Workflow:**
|
||||
1. **Collect** — Run ecosystem audit tools, aggregate findings
|
||||
2. **Deduplicate** — Group by CVE ID across direct and transitive deps
|
||||
3. **Prioritize** — Critical + exploitable + reachable = fix immediately
|
||||
4. **Remediate** — Upgrade, patch, or mitigate with compensating controls
|
||||
5. **Verify** — Rerun audit to confirm fix, update lock files
|
||||
|
||||
```bash
|
||||
# Node.js
|
||||
npm audit --json | jq '.vulnerabilities | to_entries[] | select(.value.severity == "critical")'
|
||||
|
||||
# Python
|
||||
pip audit --format json --desc
|
||||
safety check --json
|
||||
|
||||
# Go
|
||||
govulncheck ./...
|
||||
|
||||
# Ruby
|
||||
bundle audit check --update
|
||||
```
|
||||
|
||||
### CVE Triage Workflow
|
||||
|
||||
1. **Collect**: Run ecosystem audit tools, aggregate findings
|
||||
2. **Deduplicate**: Group by CVE ID across direct and transitive deps
|
||||
3. **Score**: Use CVSS base score + environmental adjustments
|
||||
4. **Prioritize**: Critical + exploitable + reachable = fix immediately
|
||||
5. **Remediate**: Upgrade, patch, or mitigate with compensating controls
|
||||
6. **Verify**: Rerun audit to confirm fix, update lock files
|
||||
|
||||
```bash
|
||||
# Use the dependency auditor for automated triage
|
||||
python scripts/dependency_auditor.py --file package.json --severity critical --json
|
||||
```
|
||||
|
||||
### Known Vulnerable Patterns
|
||||
|
||||
| Package | Vulnerable Versions | CVE | Impact |
|
||||
|---------|-------------------|-----|--------|
|
||||
| log4j-core | 2.0 - 2.14.1 | CVE-2021-44228 | RCE via JNDI injection |
|
||||
| lodash | < 4.17.21 | CVE-2021-23337 | Prototype pollution |
|
||||
| axios | < 1.6.0 | CVE-2023-45857 | CSRF token exposure |
|
||||
| pillow | < 9.3.0 | CVE-2022-45198 | DoS via crafted image |
|
||||
| express | < 4.19.2 | CVE-2024-29041 | Open redirect |
|
||||
|
||||
---
|
||||
|
||||
## Secret Scanning
|
||||
|
||||
### TruffleHog Patterns
|
||||
**Tools:** TruffleHog (git history + filesystem), Gitleaks (regex-based with custom rules).
|
||||
|
||||
```bash
|
||||
# Scan git history for secrets
|
||||
# Scan git history for verified secrets
|
||||
trufflehog git file://. --only-verified --json
|
||||
|
||||
# Scan filesystem (no git history)
|
||||
# Scan filesystem
|
||||
trufflehog filesystem . --json
|
||||
```
|
||||
|
||||
### Gitleaks Configuration
|
||||
|
||||
```toml
|
||||
# .gitleaks.toml
|
||||
title = "Custom Gitleaks Config"
|
||||
|
||||
[[rules]]
|
||||
id = "aws-access-key"
|
||||
description = "AWS Access Key ID"
|
||||
regex = '''AKIA[0-9A-Z]{16}'''
|
||||
tags = ["aws", "credentials"]
|
||||
|
||||
[[rules]]
|
||||
id = "generic-api-key"
|
||||
description = "Generic API Key"
|
||||
regex = '''(?i)(api[_-]?key|apikey)\s*[:=]\s*['\"][a-zA-Z0-9]{20,}['\"]'''
|
||||
tags = ["api", "key"]
|
||||
|
||||
[[rules]]
|
||||
id = "private-key"
|
||||
description = "Private Key Header"
|
||||
regex = '''-----BEGIN (RSA|EC|DSA|OPENSSH) PRIVATE KEY-----'''
|
||||
tags = ["private-key"]
|
||||
|
||||
[allowlist]
|
||||
paths = ['''\.test\.''', '''_test\.go''', '''mock''', '''fixture''']
|
||||
```
|
||||
|
||||
### Pre-commit Hook Integration
|
||||
|
||||
```yaml
|
||||
# .pre-commit-config.yaml
|
||||
repos:
|
||||
- repo: https://github.com/gitleaks/gitleaks
|
||||
rev: v8.18.0
|
||||
hooks:
|
||||
- id: gitleaks
|
||||
|
||||
- repo: https://github.com/trufflesecurity/trufflehog
|
||||
rev: v3.63.0
|
||||
hooks:
|
||||
- id: trufflehog
|
||||
args: ["git", "file://.", "--since-commit", "HEAD", "--only-verified"]
|
||||
```
|
||||
|
||||
### CI Integration (GitHub Actions)
|
||||
|
||||
```yaml
|
||||
name: Secret Scan
|
||||
on: [push, pull_request]
|
||||
jobs:
|
||||
scan:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- uses: trufflesecurity/trufflehog@main
|
||||
with:
|
||||
extra_args: --only-verified
|
||||
```
|
||||
**Integration points:** Pre-commit hooks (gitleaks, trufflehog), CI/CD gates (GitHub Actions with `trufflesecurity/trufflehog@main`). Configure `.gitleaks.toml` for custom rules (AWS keys, API keys, private key headers) and allowlists for test fixtures.
|
||||
|
||||
---
|
||||
|
||||
@@ -425,252 +143,45 @@ jobs:
|
||||
|
||||
### Authentication Bypass
|
||||
|
||||
**JWT Manipulation:**
|
||||
1. Decode token at jwt.io — inspect claims without verification
|
||||
2. Change `alg` to `none` and remove signature: `eyJ...payload.`
|
||||
3. Change `alg` from RS256 to HS256 and sign with the public key
|
||||
4. Modify claims (`role: "admin"`, `exp: 9999999999`) and re-sign with weak secrets
|
||||
5. Test key confusion: HMAC signed with RSA public key bytes
|
||||
|
||||
**Session Fixation:**
|
||||
1. Obtain a session token before authentication
|
||||
2. Authenticate — check if the session ID changes
|
||||
3. If the same session ID persists, the app is vulnerable to session fixation
|
||||
- **JWT manipulation:** Change `alg` to `none`, RS256-to-HS256 confusion, claim modification (`role: "admin"`, `exp: 9999999999`)
|
||||
- **Session fixation:** Check if session ID changes after authentication
|
||||
|
||||
### Authorization Flaws
|
||||
|
||||
**IDOR (Insecure Direct Object Reference):**
|
||||
```
|
||||
GET /api/users/123/profile → 200 (your profile)
|
||||
GET /api/users/124/profile → 200 (someone else's profile — IDOR!)
|
||||
GET /api/users/124/profile → 403 (properly protected)
|
||||
```
|
||||
- **IDOR/BOLA:** Change resource IDs in every endpoint — test read, update, delete across users
|
||||
- **BFLA:** Regular user tries admin endpoints (expect 403)
|
||||
- **Mass assignment:** Add privileged fields (`role`, `is_admin`) to update requests
|
||||
|
||||
Test pattern: Change numeric IDs, UUIDs, slugs in every endpoint. Use Burp Intruder or a simple script to iterate.
|
||||
### Rate Limiting & GraphQL
|
||||
|
||||
**BOLA (Broken Object Level Authorization):**
|
||||
Same as IDOR but specifically in REST APIs. Test every CRUD operation:
|
||||
- Can user A read user B's resource?
|
||||
- Can user A update user B's resource?
|
||||
- Can user A delete user B's resource?
|
||||
- **Rate limiting:** Rapid-fire requests to auth endpoints; expect 429 after threshold
|
||||
- **GraphQL:** Test introspection (should be disabled in prod), query depth attacks, batch mutations bypassing rate limits
|
||||
|
||||
**BFLA (Broken Function Level Authorization):**
|
||||
```
|
||||
# Regular user tries admin endpoints
|
||||
POST /api/admin/users → Should be 403
|
||||
DELETE /api/admin/users/123 → Should be 403
|
||||
PUT /api/settings/global → Should be 403
|
||||
```
|
||||
|
||||
### Rate Limiting Validation
|
||||
|
||||
Test rate limits on critical endpoints:
|
||||
```bash
|
||||
# Rapid-fire login attempts
|
||||
for i in $(seq 1 100); do
|
||||
curl -s -o /dev/null -w "%{http_code}" \
|
||||
-X POST https://target.com/api/login \
|
||||
-d '{"email":"test@test.com","password":"wrong"}';
|
||||
done
|
||||
# Expect: 429 after threshold (typically 5-10 attempts)
|
||||
```
|
||||
|
||||
### Mass Assignment Detection
|
||||
|
||||
```bash
|
||||
# Try adding admin fields to a regular update request
|
||||
PUT /api/users/profile
|
||||
{
|
||||
"name": "Normal User",
|
||||
"email": "user@test.com",
|
||||
"role": "admin", # mass assignment attempt
|
||||
"is_verified": true, # mass assignment attempt
|
||||
"subscription": "enterprise" # mass assignment attempt
|
||||
}
|
||||
```
|
||||
|
||||
### GraphQL-Specific Testing
|
||||
|
||||
**Introspection Query:**
|
||||
```graphql
|
||||
{
|
||||
__schema {
|
||||
types { name fields { name type { name } } }
|
||||
}
|
||||
}
|
||||
```
|
||||
Introspection should be **disabled in production**.
|
||||
|
||||
**Query Depth Attack:**
|
||||
```graphql
|
||||
{
|
||||
user(id: 1) {
|
||||
friends {
|
||||
friends {
|
||||
friends {
|
||||
friends { # Keep nesting until server crashes
|
||||
name
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Batching Attack:**
|
||||
```json
|
||||
[
|
||||
{"query": "mutation { login(user:\"admin\", pass:\"password1\") { token } }"},
|
||||
{"query": "mutation { login(user:\"admin\", pass:\"password2\") { token } }"},
|
||||
{"query": "mutation { login(user:\"admin\", pass:\"password3\") { token } }"}
|
||||
]
|
||||
```
|
||||
Batch mutations can bypass rate limiting if counted as a single request.
|
||||
See [attack_patterns.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering-team/security-pen-testing/references/attack_patterns.md) for complete JWT manipulation payloads, IDOR testing methodology, BFLA endpoint lists, GraphQL introspection/depth/batch attack patterns, and rate limiting bypass techniques.
|
||||
|
||||
---
|
||||
|
||||
## Web Vulnerability Testing
|
||||
|
||||
### XSS (Cross-Site Scripting)
|
||||
| Vulnerability | Key Tests |
|
||||
|--------------|-----------|
|
||||
| **XSS** | Reflected (script/img/svg payloads), Stored (persistent fields), DOM-based (innerHTML + location.hash) |
|
||||
| **CSRF** | Replay without token (expect 403), cross-session token replay, check SameSite cookie attribute |
|
||||
| **SQL Injection** | Error-based (`' OR 1=1--`), union-based enumeration, time-based blind (`SLEEP(5)`), boolean-based blind |
|
||||
| **SSRF** | Internal IPs, cloud metadata endpoints (AWS/GCP/Azure), IPv6/hex/decimal encoding bypasses |
|
||||
| **Path Traversal** | [`etc/passwd`](https://github.com/alirezarezvani/claude-skills/tree/main/../etc/passwd), URL encoding, double encoding bypasses |
|
||||
|
||||
**Reflected XSS Test Payloads** (non-destructive):
|
||||
```
|
||||
<script>alert(document.domain)</script>
|
||||
"><img src=x onerror=alert(document.domain)>
|
||||
javascript:alert(document.domain)
|
||||
<svg onload=alert(document.domain)>
|
||||
'-alert(document.domain)-'
|
||||
</script><script>alert(document.domain)</script>
|
||||
```
|
||||
|
||||
**Stored XSS**: Submit payloads in persistent fields (comments, profiles, messages), then check if they render for other users.
|
||||
|
||||
**DOM-Based XSS**: Look for `innerHTML`, `document.write()`, `eval()` operating on `location.hash`, `location.search`, or `document.referrer`.
|
||||
|
||||
### CSRF Token Validation
|
||||
|
||||
1. Capture a legitimate request with CSRF token
|
||||
2. Replay the request without the token — should fail (403)
|
||||
3. Replay with a token from a different session — should fail
|
||||
4. Check if token changes per request or is static per session
|
||||
5. Verify `SameSite` cookie attribute is set to `Strict` or `Lax`
|
||||
|
||||
### SQL Injection
|
||||
|
||||
**Detection Payloads** (safe, non-destructive):
|
||||
```
|
||||
' OR '1'='1
|
||||
' OR '1'='1' --
|
||||
" OR "1"="1
|
||||
1 OR 1=1
|
||||
' UNION SELECT NULL--
|
||||
' AND SLEEP(5)-- (time-based blind)
|
||||
' AND 1=1-- (boolean-based blind)
|
||||
```
|
||||
|
||||
**Union-Based Enumeration** (authorized testing only):
|
||||
```sql
|
||||
' UNION SELECT 1,2,3-- -- Find column count
|
||||
' UNION SELECT table_name,2,3 FROM information_schema.tables--
|
||||
' UNION SELECT column_name,2,3 FROM information_schema.columns WHERE table_name='users'--
|
||||
```
|
||||
|
||||
**Time-Based Blind:**
|
||||
```sql
|
||||
' AND IF(1=1, SLEEP(5), 0)-- -- MySQL
|
||||
' AND pg_sleep(5)-- -- PostgreSQL
|
||||
' WAITFOR DELAY '0:0:5'-- -- MSSQL
|
||||
```
|
||||
|
||||
### SSRF Detection
|
||||
|
||||
**Payloads for SSRF testing:**
|
||||
```
|
||||
http://127.0.0.1
|
||||
http://localhost
|
||||
http://169.254.169.254/latest/meta-data/ (AWS metadata)
|
||||
http://metadata.google.internal/ (GCP metadata)
|
||||
http://169.254.169.254/metadata/instance (Azure metadata)
|
||||
http://[::1] (IPv6 localhost)
|
||||
http://0x7f000001 (hex encoding)
|
||||
http://2130706433 (decimal encoding)
|
||||
```
|
||||
|
||||
### Path Traversal
|
||||
|
||||
```
|
||||
GET /api/files?name=../../../etc/passwd
|
||||
GET /api/files?name=....//....//....//etc/passwd
|
||||
GET /api/files?name=%2e%2e%2f%2e%2e%2f%2e%2e%2fetc%2fpasswd
|
||||
GET /api/files?name=..%252f..%252f..%252fetc%252fpasswd (double encoding)
|
||||
```
|
||||
See [attack_patterns.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering-team/security-pen-testing/references/attack_patterns.md) for complete test payloads (XSS filter bypasses, context-specific XSS, SQL injection per database engine, SSRF bypass techniques, and DOM-based XSS source/sink pairs).
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure Security
|
||||
|
||||
### Misconfigured Cloud Storage
|
||||
|
||||
**S3 Bucket Checks:**
|
||||
```bash
|
||||
# Check for public read access
|
||||
aws s3 ls s3://target-bucket --no-sign-request
|
||||
|
||||
# Check bucket policy
|
||||
aws s3api get-bucket-policy --bucket target-bucket
|
||||
|
||||
# Check ACL
|
||||
aws s3api get-bucket-acl --bucket target-bucket
|
||||
```
|
||||
|
||||
**Common Bucket Name Patterns:**
|
||||
```
|
||||
{company}-backup, {company}-dev, {company}-staging
|
||||
{company}-assets, {company}-uploads, {company}-logs
|
||||
```
|
||||
|
||||
### HTTP Security Headers
|
||||
|
||||
Required headers and expected values:
|
||||
|
||||
| Header | Expected Value |
|
||||
|--------|---------------|
|
||||
| `Strict-Transport-Security` | `max-age=31536000; includeSubDomains; preload` |
|
||||
| `Content-Security-Policy` | Restrictive policy, no `unsafe-inline` or `unsafe-eval` |
|
||||
| `X-Content-Type-Options` | `nosniff` |
|
||||
| `X-Frame-Options` | `DENY` or `SAMEORIGIN` |
|
||||
| `Referrer-Policy` | `strict-origin-when-cross-origin` |
|
||||
| `Permissions-Policy` | Restrict camera, microphone, geolocation |
|
||||
| `X-XSS-Protection` | `0` (deprecated, CSP is preferred) |
|
||||
|
||||
### TLS Configuration
|
||||
|
||||
```bash
|
||||
# Check TLS version and cipher suites
|
||||
nmap --script ssl-enum-ciphers -p 443 target.com
|
||||
|
||||
# Quick check with testssl.sh
|
||||
./testssl.sh target.com
|
||||
|
||||
# Check certificate expiry
|
||||
echo | openssl s_client -connect target.com:443 2>/dev/null | openssl x509 -noout -dates
|
||||
```
|
||||
|
||||
**Reject:** TLS 1.0, TLS 1.1, RC4, DES, 3DES, MD5 in cipher suites, CBC mode ciphers (BEAST), export-grade ciphers.
|
||||
|
||||
### Open Port Scanning
|
||||
|
||||
```bash
|
||||
# Quick top-1000 ports
|
||||
nmap -sV target.com
|
||||
|
||||
# Full port scan
|
||||
nmap -p- -sV target.com
|
||||
|
||||
# Common dangerous open ports
|
||||
# 21 (FTP), 23 (Telnet), 445 (SMB), 3389 (RDP), 6379 (Redis), 27017 (MongoDB)
|
||||
```
|
||||
**Key checks:**
|
||||
- **Cloud storage:** S3 bucket public access (`aws s3 ls s3://bucket --no-sign-request`), bucket policies, ACLs
|
||||
- **HTTP security headers:** HSTS, CSP (no `unsafe-inline`/`unsafe-eval`), X-Content-Type-Options, X-Frame-Options, Referrer-Policy
|
||||
- **TLS configuration:** `nmap --script ssl-enum-ciphers -p 443 target.com` or `testssl.sh` — reject TLS 1.0/1.1, RC4, 3DES, export-grade ciphers
|
||||
- **Port scanning:** `nmap -sV target.com` — flag dangerous open ports (FTP/21, Telnet/23, Redis/6379, MongoDB/27017)
|
||||
|
||||
---
|
||||
|
||||
@@ -719,26 +230,11 @@ python scripts/pentest_report_generator.py --findings findings.json --format jso
|
||||
|
||||
## Responsible Disclosure Workflow
|
||||
|
||||
Responsible disclosure is **mandatory** for any vulnerability found during authorized testing or independent research. See `references/responsible_disclosure.md` for full templates.
|
||||
Responsible disclosure is **mandatory** for any vulnerability found during authorized testing. Standard timeline: report on day 1, follow up at day 7, status update at day 30, public disclosure at day 90.
|
||||
|
||||
### Timeline
|
||||
**Key principles:** Never exploit beyond proof of concept, encrypt all communications, do not access real user data, document everything with timestamps.
|
||||
|
||||
| Day | Action |
|
||||
|-----|--------|
|
||||
| 0 | Discovery — document finding with evidence |
|
||||
| 1 | Report to vendor via security contact or bug bounty program |
|
||||
| 7 | Follow up if no acknowledgment received |
|
||||
| 30 | Request status update and remediation timeline |
|
||||
| 60 | Second follow-up — offer technical assistance |
|
||||
| 90 | Public disclosure (with or without fix, per industry standard) |
|
||||
|
||||
### Key Principles
|
||||
|
||||
1. **Never exploit beyond proof of concept** — demonstrate impact without causing damage
|
||||
2. **Encrypt all communications** — PGP/GPG for email, secure channels for details
|
||||
3. **Do not access, modify, or exfiltrate real user data** — use your own test accounts
|
||||
4. **Document everything** — timestamps, screenshots, request/response pairs
|
||||
5. **Respect the vendor's timeline** — extend deadline if they're actively working on a fix
|
||||
See [responsible_disclosure.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering-team/security-pen-testing/references/responsible_disclosure.md) for full disclosure timelines (standard 90-day, accelerated 30-day, extended 120-day), communication templates, legal considerations, bug bounty program integration, and CVE request process.
|
||||
|
||||
---
|
||||
|
||||
@@ -792,47 +288,7 @@ python scripts/pentest_report_generator.py --findings findings.json --format md
|
||||
|
||||
### Workflow 3: CI/CD Security Gate
|
||||
|
||||
Automated security checks that run on every pull request:
|
||||
|
||||
```yaml
|
||||
# .github/workflows/security-gate.yml
|
||||
name: Security Gate
|
||||
on: [pull_request]
|
||||
jobs:
|
||||
security:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
# Secret scanning
|
||||
- name: Scan for secrets
|
||||
uses: trufflesecurity/trufflehog@main
|
||||
with:
|
||||
extra_args: --only-verified
|
||||
|
||||
# Dependency audit
|
||||
- name: Audit dependencies
|
||||
run: |
|
||||
npm audit --audit-level=high
|
||||
pip audit --desc
|
||||
|
||||
# SAST
|
||||
- name: Static analysis
|
||||
uses: returntocorp/semgrep-action@v1
|
||||
with:
|
||||
config: >-
|
||||
p/security-audit
|
||||
p/secrets
|
||||
p/owasp-top-ten
|
||||
|
||||
# Security headers check (staging only)
|
||||
- name: Check security headers
|
||||
if: github.base_ref == 'staging'
|
||||
run: |
|
||||
curl -sI $STAGING_URL | python scripts/vulnerability_scanner.py --target web --scope quick
|
||||
```
|
||||
Automated security checks on every PR: secret scanning (TruffleHog), dependency audit (`npm audit`, `pip audit`), SAST (Semgrep with `p/security-audit`, `p/owasp-top-ten`), and security headers check on staging.
|
||||
|
||||
**Gate Policy**: Block merge on critical/high findings. Warn on medium. Log low/info.
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ python scripts/pipeline_generator.py ./app --platform=github --stages=build,test
|
||||
python scripts/terraform_scaffolder.py ./infra --provider=aws --module=ecs-service --verbose
|
||||
|
||||
# Script 3: Deployment Manager — orchestrates container deployments with rollback support
|
||||
python scripts/deployment_manager.py deploy --env=production --image=app:1.2.3 --strategy=blue-green
|
||||
python3 scripts/deployment_manager.py ./deploy --verbose --json
|
||||
```
|
||||
|
||||
## Core Capabilities
|
||||
@@ -281,6 +281,54 @@ kubectl get pods -n production -l app=myapp
|
||||
curl -sf https://app.example.com/healthz || echo "ROLLBACK FAILED — escalate"
|
||||
```
|
||||
|
||||
## Multi-Cloud Cross-References
|
||||
|
||||
Use these companion skills for cloud-specific deep dives:
|
||||
|
||||
| Skill | Cloud | Use When |
|
||||
|-------|-------|----------|
|
||||
| **aws-solution-architect** | AWS | ECS/EKS, Lambda, VPC design, cost optimization |
|
||||
| **azure-cloud-architect** | Azure | AKS, App Service, Virtual Networks, Azure DevOps |
|
||||
| **gcp-cloud-architect** | GCP | GKE, Cloud Run, VPC, Cloud Build *(coming soon)* |
|
||||
|
||||
**Multi-cloud vs single-cloud decision:**
|
||||
- **Single-cloud** (default) — lower operational complexity, deeper managed-service integration, better cost leverage with committed-use discounts
|
||||
- **Multi-cloud** — required when mandated by compliance/data residency, acquiring companies on different clouds, or needing best-of-breed services across providers (e.g., AWS for compute + GCP for ML)
|
||||
- **Hybrid** — on-prem + cloud; use when regulated workloads must stay on-prem while burst/non-sensitive workloads run in the cloud
|
||||
|
||||
> Start single-cloud. Add a second cloud only when there is a concrete business or compliance driver — not for theoretical redundancy.
|
||||
|
||||
---
|
||||
|
||||
## Cloud-Agnostic IaC
|
||||
|
||||
### Terraform / OpenTofu (Default Choice)
|
||||
|
||||
Terraform (or its open-source fork OpenTofu) is the recommended IaC tool for most teams:
|
||||
- Single language (HCL) across AWS, Azure, GCP, and 3,000+ providers
|
||||
- State management with remote backends (S3, GCS, Azure Blob)
|
||||
- Plan-before-apply workflow prevents drift surprises
|
||||
- Cross-reference **terraform-patterns** for module structure, state isolation, and CI/CD integration
|
||||
|
||||
### Pulumi (Programming Language IaC)
|
||||
|
||||
Choose Pulumi when the team strongly prefers TypeScript, Python, Go, or C# over HCL:
|
||||
- Full programming language — loops, conditionals, unit tests native
|
||||
- Same cloud provider coverage as Terraform
|
||||
- Easier onboarding for dev teams that resist learning HCL
|
||||
|
||||
### When to Use Cloud-Native IaC
|
||||
|
||||
| Tool | Use When |
|
||||
|------|----------|
|
||||
| **CloudFormation** | AWS-only shop; need native AWS support (StackSets, Service Catalog) |
|
||||
| **Bicep** | Azure-only shop; simpler syntax than ARM templates |
|
||||
| **Cloud Deployment Manager** | GCP-only; rare — most GCP teams prefer Terraform |
|
||||
|
||||
> **Rule of thumb:** Use Terraform/OpenTofu unless you are 100% committed to a single cloud AND the cloud-native tool offers a feature Terraform cannot replicate (e.g., AWS Service Catalog integration).
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
Check the comprehensive troubleshooting section in `references/deployment_strategies.md`.
|
||||
|
||||
@@ -424,6 +424,89 @@ app.use((req, res, next) => {
|
||||
|
||||
---
|
||||
|
||||
## OWASP Top 10 Quick-Check
|
||||
|
||||
Rapid 15-minute assessment — run through each category and note pass/fail. For deep-dive testing, hand off to the **security-pen-testing** skill.
|
||||
|
||||
| # | Category | One-Line Check |
|
||||
|---|----------|----------------|
|
||||
| A01 | Broken Access Control | Verify role checks on every endpoint; test horizontal privilege escalation |
|
||||
| A02 | Cryptographic Failures | Confirm TLS 1.2+ everywhere; no secrets in logs or source |
|
||||
| A03 | Injection | Run parameterized query audit; check ORM raw-query usage |
|
||||
| A04 | Insecure Design | Review threat model exists for critical flows |
|
||||
| A05 | Security Misconfiguration | Check default credentials removed; error pages generic |
|
||||
| A06 | Vulnerable Components | Run `vulnerability_assessor.py`; zero critical/high CVEs |
|
||||
| A07 | Auth Failures | Verify MFA on admin; brute-force protection active |
|
||||
| A08 | Software & Data Integrity | Confirm CI/CD pipeline signs artifacts; no unsigned deps |
|
||||
| A09 | Logging & Monitoring | Validate audit logs capture auth events; alerts configured |
|
||||
| A10 | SSRF | Test internal URL filters; block metadata endpoints (169.254.169.254) |
|
||||
|
||||
> **Deep dive needed?** Hand off to `security-pen-testing` for full OWASP Testing Guide coverage.
|
||||
|
||||
---
|
||||
|
||||
## Secret Scanning Tools
|
||||
|
||||
Choose the right scanner for each stage of your workflow:
|
||||
|
||||
| Tool | Best For | Language | Pre-commit | CI/CD | Custom Rules |
|
||||
|------|----------|----------|:----------:|:-----:|:------------:|
|
||||
| **gitleaks** | CI pipelines, full-repo scans | Go | Yes | Yes | TOML regexes |
|
||||
| **detect-secrets** | Pre-commit hooks, incremental | Python | Yes | Partial | Plugin-based |
|
||||
| **truffleHog** | Deep history scans, entropy | Go | No | Yes | Regex + entropy |
|
||||
|
||||
**Recommended setup:** Use `detect-secrets` as a pre-commit hook (catches secrets before they enter history) and `gitleaks` in CI (catches anything that slips through).
|
||||
|
||||
```bash
|
||||
# detect-secrets pre-commit hook (.pre-commit-config.yaml)
|
||||
- repo: https://github.com/Yelp/detect-secrets
|
||||
rev: v1.4.0
|
||||
hooks:
|
||||
- id: detect-secrets
|
||||
args: ['--baseline', '.secrets.baseline']
|
||||
|
||||
# gitleaks in GitHub Actions
|
||||
- name: gitleaks
|
||||
uses: gitleaks/gitleaks-action@v2
|
||||
env:
|
||||
GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Supply Chain Security
|
||||
|
||||
Protect against dependency and artifact tampering with SBOM generation, artifact signing, and SLSA compliance.
|
||||
|
||||
**SBOM Generation:**
|
||||
- **syft** — generates SBOMs from container images or source dirs (SPDX, CycloneDX formats)
|
||||
- **cyclonedx-cli** — CycloneDX-native tooling; merge multiple SBOMs for mono-repos
|
||||
|
||||
```bash
|
||||
# Generate SBOM from container image
|
||||
syft packages ghcr.io/org/app:latest -o cyclonedx-json > sbom.json
|
||||
```
|
||||
|
||||
**Artifact Signing (Sigstore/cosign):**
|
||||
```bash
|
||||
# Sign a container image (keyless via OIDC)
|
||||
cosign sign ghcr.io/org/app:latest
|
||||
# Verify signature
|
||||
cosign verify ghcr.io/org/app:latest --certificate-identity=ci@org.com --certificate-oidc-issuer=https://token.actions.githubusercontent.com
|
||||
```
|
||||
|
||||
**SLSA Levels Overview:**
|
||||
| Level | Requirement | What It Proves |
|
||||
|-------|-------------|----------------|
|
||||
| 1 | Build process documented | Provenance exists |
|
||||
| 2 | Hosted build service, signed provenance | Tamper-resistant provenance |
|
||||
| 3 | Hardened build platform, non-falsifiable provenance | Tamper-proof build |
|
||||
| 4 | Two-party review, hermetic builds | Maximum supply-chain assurance |
|
||||
|
||||
> **Cross-references:** `security-pen-testing` (vulnerability exploitation testing), `dependency-auditor` (license and CVE audit for dependencies).
|
||||
|
||||
---
|
||||
|
||||
## Reference Documentation
|
||||
|
||||
| Document | Description |
|
||||
|
||||
@@ -150,6 +150,254 @@ Additional scripts: `framework_adapter.py` (convert between frameworks), `metric
|
||||
|
||||
---
|
||||
|
||||
## Spec-First Workflow
|
||||
|
||||
TDD is most effective when driven by a written spec. The flow:
|
||||
|
||||
1. **Write or receive a spec** — stored in `specs/<feature>.md`
|
||||
2. **Extract acceptance criteria** — each criterion becomes one or more test cases
|
||||
3. **Write failing tests (RED)** — one test per acceptance criterion
|
||||
4. **Implement minimal code (GREEN)** — satisfy each test in order
|
||||
5. **Refactor** — clean up while all tests stay green
|
||||
|
||||
### Spec Directory Convention
|
||||
|
||||
```
|
||||
project/
|
||||
├── specs/
|
||||
│ ├── user-auth.md # Feature spec with acceptance criteria
|
||||
│ ├── payment-processing.md
|
||||
│ └── notification-system.md
|
||||
├── tests/
|
||||
│ ├── test_user_auth.py # Tests derived from specs/user-auth.md
|
||||
│ ├── test_payments.py
|
||||
│ └── test_notifications.py
|
||||
└── src/
|
||||
```
|
||||
|
||||
### Extracting Tests from Specs
|
||||
|
||||
Each acceptance criterion in a spec maps to at least one test:
|
||||
|
||||
| Spec Criterion | Test Case |
|
||||
|---------------|-----------|
|
||||
| "User can log in with valid credentials" | `test_login_valid_credentials_returns_token` |
|
||||
| "Invalid password returns 401" | `test_login_invalid_password_returns_401` |
|
||||
| "Account locks after 5 failed attempts" | `test_login_locks_after_five_failures` |
|
||||
|
||||
**Tip:** Number your acceptance criteria in the spec. Reference the number in the test docstring for traceability (`# AC-3: Account locks after 5 failed attempts`).
|
||||
|
||||
> **Cross-reference:** See `engineering/spec-driven-workflow` for the full spec methodology, including spec templates and review checklists.
|
||||
|
||||
---
|
||||
|
||||
## Red-Green-Refactor Examples Per Language
|
||||
|
||||
### TypeScript / Jest
|
||||
|
||||
```typescript
|
||||
// test/cart.test.ts
|
||||
describe("Cart", () => {
|
||||
describe("addItem", () => {
|
||||
it("should add a new item to an empty cart", () => {
|
||||
const cart = new Cart();
|
||||
cart.addItem({ id: "sku-1", name: "Widget", price: 9.99, qty: 1 });
|
||||
|
||||
expect(cart.items).toHaveLength(1);
|
||||
expect(cart.items[0].id).toBe("sku-1");
|
||||
});
|
||||
|
||||
it("should increment quantity when adding an existing item", () => {
|
||||
const cart = new Cart();
|
||||
cart.addItem({ id: "sku-1", name: "Widget", price: 9.99, qty: 1 });
|
||||
cart.addItem({ id: "sku-1", name: "Widget", price: 9.99, qty: 2 });
|
||||
|
||||
expect(cart.items).toHaveLength(1);
|
||||
expect(cart.items[0].qty).toBe(3);
|
||||
});
|
||||
|
||||
it("should throw when quantity is zero or negative", () => {
|
||||
const cart = new Cart();
|
||||
expect(() =>
|
||||
cart.addItem({ id: "sku-1", name: "Widget", price: 9.99, qty: 0 })
|
||||
).toThrow("Quantity must be positive");
|
||||
});
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Python / Pytest (Advanced Patterns)
|
||||
|
||||
```python
|
||||
# tests/conftest.py — shared fixtures
|
||||
import pytest
|
||||
from app.db import create_engine, Session
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def db_engine():
|
||||
engine = create_engine("sqlite:///:memory:")
|
||||
yield engine
|
||||
engine.dispose()
|
||||
|
||||
@pytest.fixture
|
||||
def db_session(db_engine):
|
||||
session = Session(bind=db_engine)
|
||||
yield session
|
||||
session.rollback()
|
||||
session.close()
|
||||
|
||||
# tests/test_pricing.py — parametrize for multiple cases
|
||||
import pytest
|
||||
from app.pricing import calculate_discount
|
||||
|
||||
@pytest.mark.parametrize("subtotal, expected_discount", [
|
||||
(50.0, 0.0), # Below threshold — no discount
|
||||
(100.0, 5.0), # 5% tier
|
||||
(250.0, 25.0), # 10% tier
|
||||
(500.0, 75.0), # 15% tier
|
||||
])
|
||||
def test_calculate_discount(subtotal, expected_discount):
|
||||
assert calculate_discount(subtotal) == pytest.approx(expected_discount)
|
||||
```
|
||||
|
||||
### Go — Table-Driven Tests
|
||||
|
||||
```go
|
||||
// cart_test.go
|
||||
package cart
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestApplyDiscount(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
subtotal float64
|
||||
want float64
|
||||
}{
|
||||
{"no discount below threshold", 50.0, 0.0},
|
||||
{"5 percent tier", 100.0, 5.0},
|
||||
{"10 percent tier", 250.0, 25.0},
|
||||
{"15 percent tier", 500.0, 75.0},
|
||||
{"zero subtotal", 0.0, 0.0},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := ApplyDiscount(tt.subtotal)
|
||||
if got != tt.want {
|
||||
t.Errorf("ApplyDiscount(%v) = %v, want %v", tt.subtotal, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Bounded Autonomy Rules
|
||||
|
||||
When generating tests autonomously, follow these rules to decide when to stop and ask the user:
|
||||
|
||||
### Stop and Ask When
|
||||
|
||||
- **Ambiguous requirements** — the spec or user story has conflicting or unclear acceptance criteria
|
||||
- **Missing edge cases** — you cannot determine boundary values without domain knowledge (e.g., max allowed transaction amount)
|
||||
- **Test count exceeds 50** — large test suites need human review before committing; present a summary and ask which areas to prioritize
|
||||
- **External dependencies unclear** — the feature relies on third-party APIs or services with undocumented behavior
|
||||
- **Security-sensitive logic** — authentication, authorization, encryption, or payment flows require human sign-off on test scenarios
|
||||
|
||||
### Continue Autonomously When
|
||||
|
||||
- **Clear spec with numbered acceptance criteria** — each criterion maps directly to tests
|
||||
- **Straightforward CRUD operations** — create, read, update, delete with well-defined models
|
||||
- **Well-defined API contracts** — OpenAPI spec or typed interfaces available
|
||||
- **Pure functions** — deterministic input/output with no side effects
|
||||
- **Existing test patterns** — the codebase already has similar tests to follow
|
||||
|
||||
---
|
||||
|
||||
## Property-Based Testing
|
||||
|
||||
Property-based testing generates random inputs to verify invariants instead of relying on hand-picked examples. Use it when the input space is large and the expected behavior can be described as a property.
|
||||
|
||||
### Python — Hypothesis
|
||||
|
||||
```python
|
||||
from hypothesis import given, strategies as st
|
||||
from app.serializers import serialize, deserialize
|
||||
|
||||
@given(st.text())
|
||||
def test_roundtrip_serialization(data):
|
||||
"""Serialization followed by deserialization returns the original."""
|
||||
assert deserialize(serialize(data)) == data
|
||||
|
||||
@given(st.integers(), st.integers())
|
||||
def test_addition_is_commutative(a, b):
|
||||
assert a + b == b + a
|
||||
```
|
||||
|
||||
### TypeScript — fast-check
|
||||
|
||||
```typescript
|
||||
import fc from "fast-check";
|
||||
import { encode, decode } from "./codec";
|
||||
|
||||
test("encode/decode roundtrip", () => {
|
||||
fc.assert(
|
||||
fc.property(fc.string(), (input) => {
|
||||
expect(decode(encode(input))).toBe(input);
|
||||
})
|
||||
);
|
||||
});
|
||||
```
|
||||
|
||||
### When to Use Property-Based Over Example-Based
|
||||
|
||||
| Use Property-Based | Example |
|
||||
|-------------------|---------|
|
||||
| Data transformations | Serialize/deserialize roundtrips |
|
||||
| Mathematical properties | Commutativity, associativity, idempotency |
|
||||
| Encoding/decoding | Base64, URL encoding, compression |
|
||||
| Sorting and filtering | Output is sorted, length preserved |
|
||||
| Parser correctness | Valid input always parses without error |
|
||||
|
||||
---
|
||||
|
||||
## Mutation Testing
|
||||
|
||||
Mutation testing modifies your production code (creates "mutants") and checks whether your tests catch the changes. If a mutant survives (tests still pass), your tests have a gap that coverage alone cannot reveal.
|
||||
|
||||
### Tools
|
||||
|
||||
| Language | Tool | Command |
|
||||
|----------|------|---------|
|
||||
| TypeScript/JavaScript | **Stryker** | `npx stryker run` |
|
||||
| Python | **mutmut** | `mutmut run --paths-to-mutate=src/` |
|
||||
| Java | **PIT** | `mvn org.pitest:pitest-maven:mutationCoverage` |
|
||||
|
||||
### Why Mutation Testing Matters
|
||||
|
||||
- **100% line coverage != good tests** — coverage tells you code was executed, not that it was verified
|
||||
- **Catches weak assertions** — tests that run code but assert nothing meaningful
|
||||
- **Finds missing boundary tests** — mutants that change `<` to `<=` expose off-by-one gaps
|
||||
- **Quantifiable quality metric** — mutation score (% mutants killed) is a stronger signal than coverage %
|
||||
|
||||
**Recommendation:** Run mutation testing on critical paths (auth, payments, data processing) even if overall coverage is high. Target 85%+ mutation score on P0 modules.
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
| Skill | Relationship |
|
||||
|-------|-------------|
|
||||
| `engineering/spec-driven-workflow` | Spec → acceptance criteria → test extraction pipeline |
|
||||
| `engineering-team/focused-fix` | Phase 5 (Verify) uses TDD to confirm the fix with a regression test |
|
||||
| `engineering-team/senior-qa` | Broader QA strategy; TDD is one layer in the test pyramid |
|
||||
| `engineering-team/code-reviewer` | Review generated tests for assertion quality and coverage completeness |
|
||||
| `engineering-team/senior-fullstack` | Project scaffolders include testing infrastructure compatible with TDD workflows |
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
| Scope | Details |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Agent Designer - Multi-Agent System Architecture — Agent Skill for Codex & OpenClaw"
|
||||
description: "Agent Designer - Multi-Agent System Architecture. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
description: "Use when the user asks to design multi-agent systems, create agent architectures, define agent communication patterns, or build autonomous agent. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
---
|
||||
|
||||
# Agent Designer - Multi-Agent System Architecture
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "API Test Suite Builder — Agent Skill for Codex & OpenClaw"
|
||||
description: "API Test Suite Builder. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
description: "Use when the user asks to generate API tests, create integration test suites, test REST endpoints, or build contract tests. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
---
|
||||
|
||||
# API Test Suite Builder
|
||||
|
||||
@@ -44,9 +44,6 @@ The Browser Automation skill provides comprehensive tools and knowledge for buil
|
||||
|
||||
### 1. Web Scraping Patterns
|
||||
|
||||
#### DOM Extraction with CSS Selectors
|
||||
CSS selectors are the primary tool for element targeting. Prefer them over XPath for readability and performance.
|
||||
|
||||
**Selector priority (most to least reliable):**
|
||||
1. `data-testid`, `data-id`, or custom data attributes — stable across redesigns
|
||||
2. `#id` selectors — unique but may change between deploys
|
||||
@@ -54,365 +51,70 @@ CSS selectors are the primary tool for element targeting. Prefer them over XPath
|
||||
4. Class-based: `.product-card`, `.price` — brittle if classes are generated (e.g., CSS modules)
|
||||
5. Positional: `nth-child()`, `nth-of-type()` — last resort, breaks on layout changes
|
||||
|
||||
**Compound selectors for precision:**
|
||||
```python
|
||||
# Product cards within a specific container
|
||||
page.query_selector_all("div.search-results > article.product-card")
|
||||
Use XPath only when CSS cannot express the relationship (e.g., ancestor traversal, text-based selection).
|
||||
|
||||
# Price inside a product card (scoped)
|
||||
card.query_selector("span[data-field='price']")
|
||||
|
||||
# Links with specific text content
|
||||
page.locator("a", has_text="Next Page")
|
||||
```
|
||||
|
||||
#### XPath for Complex Traversal
|
||||
Use XPath only when CSS cannot express the relationship:
|
||||
```python
|
||||
# Find element by text content (XPath strength)
|
||||
page.locator("//td[contains(text(), 'Total')]/following-sibling::td[1]")
|
||||
|
||||
# Navigate up the DOM tree
|
||||
page.locator("//span[@class='price']/ancestor::div[@class='product']")
|
||||
```
|
||||
|
||||
#### Pagination Patterns
|
||||
- **Next-button pagination**: Click "Next" until disabled or absent
|
||||
- **URL-based pagination**: Increment `?page=N` or `&offset=N` in URL
|
||||
- **Infinite scroll**: Scroll to bottom, wait for new content, repeat until no change
|
||||
- **Load-more button**: Click button, wait for DOM mutation, repeat
|
||||
|
||||
#### Infinite Scroll Handling
|
||||
```python
|
||||
async def scroll_to_bottom(page, max_scrolls=50, pause_ms=1500):
|
||||
previous_height = 0
|
||||
for i in range(max_scrolls):
|
||||
current_height = await page.evaluate("document.body.scrollHeight")
|
||||
if current_height == previous_height:
|
||||
break
|
||||
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
|
||||
await page.wait_for_timeout(pause_ms)
|
||||
previous_height = current_height
|
||||
return i + 1 # number of scrolls performed
|
||||
```
|
||||
**Pagination strategies:** next-button, URL-based (`?page=N`), infinite scroll, load-more button. See [data_extraction_recipes.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering/browser-automation/references/data_extraction_recipes.md) for complete pagination handlers and scroll patterns.
|
||||
|
||||
### 2. Form Filling & Multi-Step Workflows
|
||||
|
||||
#### Login Flows
|
||||
```python
|
||||
async def login(page, url, username, password):
|
||||
await page.goto(url)
|
||||
await page.fill("input[name='username']", username)
|
||||
await page.fill("input[name='password']", password)
|
||||
await page.click("button[type='submit']")
|
||||
# Wait for navigation to complete (post-login redirect)
|
||||
await page.wait_for_url("**/dashboard**")
|
||||
```
|
||||
Break multi-step forms into discrete functions per step. Each function fills fields, clicks "Next"/"Continue", and waits for the next step to load (URL change or DOM element).
|
||||
|
||||
#### Multi-Page Forms
|
||||
Break multi-step forms into discrete functions per step. Each function:
|
||||
1. Fills the fields for that step
|
||||
2. Clicks the "Next" or "Continue" button
|
||||
3. Waits for the next step to load (URL change or DOM element)
|
||||
|
||||
```python
|
||||
async def fill_step_1(page, data):
|
||||
await page.fill("#first-name", data["first_name"])
|
||||
await page.fill("#last-name", data["last_name"])
|
||||
await page.select_option("#country", data["country"])
|
||||
await page.click("button:has-text('Continue')")
|
||||
await page.wait_for_selector("#step-2-form")
|
||||
|
||||
async def fill_step_2(page, data):
|
||||
await page.fill("#address", data["address"])
|
||||
await page.fill("#city", data["city"])
|
||||
await page.click("button:has-text('Continue')")
|
||||
await page.wait_for_selector("#step-3-form")
|
||||
```
|
||||
|
||||
#### File Uploads
|
||||
```python
|
||||
# Single file
|
||||
await page.set_input_files("input[type='file']", "/path/to/file.pdf")
|
||||
|
||||
# Multiple files
|
||||
await page.set_input_files("input[type='file']", [
|
||||
"/path/to/file1.pdf",
|
||||
"/path/to/file2.pdf"
|
||||
])
|
||||
|
||||
# Drag-and-drop upload zones (no visible input element)
|
||||
async with page.expect_file_chooser() as fc_info:
|
||||
await page.click("div.upload-zone")
|
||||
file_chooser = await fc_info.value
|
||||
await file_chooser.set_files("/path/to/file.pdf")
|
||||
```
|
||||
|
||||
#### Dropdown and Select Handling
|
||||
```python
|
||||
# Native <select> element
|
||||
await page.select_option("#country", value="US")
|
||||
await page.select_option("#country", label="United States")
|
||||
|
||||
# Custom dropdown (div-based)
|
||||
await page.click("div.dropdown-trigger")
|
||||
await page.click("div.dropdown-option:has-text('United States')")
|
||||
```
|
||||
Key patterns: login flows, multi-page forms, file uploads (including drag-and-drop zones), native and custom dropdown handling. See [playwright_browser_api.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering/browser-automation/references/playwright_browser_api.md) for complete API reference on `fill()`, `select_option()`, `set_input_files()`, and `expect_file_chooser()`.
|
||||
|
||||
### 3. Screenshot & PDF Capture
|
||||
|
||||
#### Screenshot Strategies
|
||||
```python
|
||||
# Full page (scrolls automatically)
|
||||
await page.screenshot(path="full-page.png", full_page=True)
|
||||
- **Full page:** `await page.screenshot(path="full.png", full_page=True)`
|
||||
- **Element:** `await page.locator("div.chart").screenshot(path="chart.png")`
|
||||
- **PDF (Chromium only):** `await page.pdf(path="out.pdf", format="A4", print_background=True)`
|
||||
- **Visual regression:** Take screenshots at known states, store baselines in version control with naming: `{page}_{viewport}_{state}.png`
|
||||
|
||||
# Viewport only (what's visible)
|
||||
await page.screenshot(path="viewport.png")
|
||||
|
||||
# Specific element
|
||||
element = page.locator("div.chart-container")
|
||||
await element.screenshot(path="chart.png")
|
||||
|
||||
# With custom viewport for consistency
|
||||
context = await browser.new_context(viewport={"width": 1920, "height": 1080})
|
||||
```
|
||||
|
||||
#### PDF Generation
|
||||
```python
|
||||
# Only works in Chromium
|
||||
await page.pdf(
|
||||
path="output.pdf",
|
||||
format="A4",
|
||||
margin={"top": "1cm", "right": "1cm", "bottom": "1cm", "left": "1cm"},
|
||||
print_background=True
|
||||
)
|
||||
```
|
||||
|
||||
#### Visual Regression Baselines
|
||||
Take screenshots at known states and compare pixel-by-pixel. Store baselines in version control. Use naming conventions: `{page}_{viewport}_{state}.png`.
|
||||
See [playwright_browser_api.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering/browser-automation/references/playwright_browser_api.md) for full screenshot/PDF options.
|
||||
|
||||
### 4. Structured Data Extraction
|
||||
|
||||
#### Tables to JSON
|
||||
```python
|
||||
async def extract_table(page, selector):
|
||||
headers = await page.eval_on_selector_all(
|
||||
f"{selector} thead th",
|
||||
"elements => elements.map(e => e.textContent.trim())"
|
||||
)
|
||||
rows = await page.eval_on_selector_all(
|
||||
f"{selector} tbody tr",
|
||||
"""rows => rows.map(row => {
|
||||
return Array.from(row.querySelectorAll('td'))
|
||||
.map(cell => cell.textContent.trim())
|
||||
})"""
|
||||
)
|
||||
return [dict(zip(headers, row)) for row in rows]
|
||||
```
|
||||
Core extraction patterns:
|
||||
- **Tables to JSON** — Extract `<thead>` headers and `<tbody>` rows into dictionaries
|
||||
- **Listings to arrays** — Map repeating card elements using a field-selector map (supports `::attr()` for attributes)
|
||||
- **Nested/threaded data** — Recursive extraction for comments with replies, category trees
|
||||
|
||||
#### Listings to Arrays
|
||||
```python
|
||||
async def extract_listings(page, container_sel, field_map):
|
||||
"""
|
||||
field_map example: {"title": "h3.title", "price": "span.price", "url": "a::attr(href)"}
|
||||
"""
|
||||
items = []
|
||||
cards = await page.query_selector_all(container_sel)
|
||||
for card in cards:
|
||||
item = {}
|
||||
for field, sel in field_map.items():
|
||||
if "::attr(" in sel:
|
||||
attr_sel, attr_name = sel.split("::attr(")
|
||||
attr_name = attr_name.rstrip(")")
|
||||
el = await card.query_selector(attr_sel)
|
||||
item[field] = await el.get_attribute(attr_name) if el else None
|
||||
else:
|
||||
el = await card.query_selector(sel)
|
||||
item[field] = (await el.text_content()).strip() if el else None
|
||||
items.append(item)
|
||||
return items
|
||||
```
|
||||
|
||||
#### Nested Data Extraction
|
||||
For threaded content (comments with replies), use recursive extraction:
|
||||
```python
|
||||
async def extract_comments(page, parent_selector):
|
||||
comments = []
|
||||
elements = await page.query_selector_all(f"{parent_selector} > .comment")
|
||||
for el in elements:
|
||||
text = await (await el.query_selector(".comment-body")).text_content()
|
||||
author = await (await el.query_selector(".author")).text_content()
|
||||
replies = await extract_comments(el, ".replies")
|
||||
comments.append({
|
||||
"author": author.strip(),
|
||||
"text": text.strip(),
|
||||
"replies": replies
|
||||
})
|
||||
return comments
|
||||
```
|
||||
See [data_extraction_recipes.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering/browser-automation/references/data_extraction_recipes.md) for complete extraction functions, price parsing, data cleaning utilities, and output format helpers (JSON, CSV, JSONL).
|
||||
|
||||
### 5. Cookie & Session Management
|
||||
|
||||
#### Save and Restore Sessions
|
||||
```python
|
||||
import json
|
||||
- **Save/restore cookies:** `context.cookies()` and `context.add_cookies()`
|
||||
- **Full storage state** (cookies + localStorage): `context.storage_state(path="state.json")` to save, `browser.new_context(storage_state="state.json")` to restore
|
||||
|
||||
# Save cookies after login
|
||||
cookies = await context.cookies()
|
||||
with open("session.json", "w") as f:
|
||||
json.dump(cookies, f)
|
||||
|
||||
# Restore session in new context
|
||||
with open("session.json", "r") as f:
|
||||
cookies = json.load(f)
|
||||
context = await browser.new_context()
|
||||
await context.add_cookies(cookies)
|
||||
```
|
||||
|
||||
#### Storage State (Cookies + Local Storage)
|
||||
```python
|
||||
# Save full state (cookies + localStorage + sessionStorage)
|
||||
await context.storage_state(path="state.json")
|
||||
|
||||
# Restore full state
|
||||
context = await browser.new_context(storage_state="state.json")
|
||||
```
|
||||
|
||||
**Best practice:** Save state after login, reuse across scraping sessions. Check session validity before starting a long job — make a lightweight request to a protected page and verify you are not redirected to login.
|
||||
**Best practice:** Save state after login, reuse across scraping sessions. Check session validity before starting a long job — make a lightweight request to a protected page and verify you are not redirected to login. See [playwright_browser_api.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering/browser-automation/references/playwright_browser_api.md) for cookie and storage state API details.
|
||||
|
||||
### 6. Anti-Detection Patterns
|
||||
|
||||
Modern websites detect automation through multiple vectors. Address all of them:
|
||||
Modern websites detect automation through multiple vectors. Apply these in priority order:
|
||||
|
||||
#### User Agent Rotation
|
||||
Never use the default Playwright user agent. Rotate through real browser user agents:
|
||||
```python
|
||||
USER_AGENTS = [
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
]
|
||||
```
|
||||
1. **WebDriver flag removal** — Remove `navigator.webdriver = true` via init script (critical)
|
||||
2. **Custom user agent** — Rotate through real browser UAs; never use the default headless UA
|
||||
3. **Realistic viewport** — Set 1920x1080 or similar real-world dimensions (default 800x600 is a red flag)
|
||||
4. **Request throttling** — Add `random.uniform()` delays between actions
|
||||
5. **Proxy support** — Per-browser or per-context proxy configuration
|
||||
|
||||
#### Viewport and Screen Size
|
||||
Set realistic viewport dimensions. The default 800x600 is a red flag:
|
||||
```python
|
||||
context = await browser.new_context(
|
||||
viewport={"width": 1920, "height": 1080},
|
||||
screen={"width": 1920, "height": 1080},
|
||||
user_agent=random.choice(USER_AGENTS),
|
||||
)
|
||||
```
|
||||
|
||||
#### WebDriver Flag Removal
|
||||
Playwright sets `navigator.webdriver = true`. Remove it:
|
||||
```python
|
||||
await page.add_init_script("""
|
||||
Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
|
||||
""")
|
||||
```
|
||||
|
||||
#### Request Throttling
|
||||
Add human-like delays between actions:
|
||||
```python
|
||||
import random
|
||||
|
||||
async def human_delay(min_ms=500, max_ms=2000):
|
||||
delay = random.randint(min_ms, max_ms)
|
||||
await page.wait_for_timeout(delay)
|
||||
```
|
||||
|
||||
#### Proxy Support
|
||||
```python
|
||||
browser = await playwright.chromium.launch(
|
||||
proxy={"server": "http://proxy.example.com:8080"}
|
||||
)
|
||||
# Or per-context:
|
||||
context = await browser.new_context(
|
||||
proxy={"server": "http://proxy.example.com:8080",
|
||||
"username": "user", "password": "pass"}
|
||||
)
|
||||
```
|
||||
See [anti_detection_patterns.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering/browser-automation/references/anti_detection_patterns.md) for the complete stealth stack: navigator property hardening, WebGL/canvas fingerprint evasion, behavioral simulation (mouse movement, typing speed, scroll patterns), proxy rotation strategies, and detection self-test URLs.
|
||||
|
||||
### 7. Dynamic Content Handling
|
||||
|
||||
#### SPA Rendering
|
||||
SPAs render content client-side. Wait for the actual content, not the page load:
|
||||
```python
|
||||
await page.goto(url)
|
||||
# Wait for the data to render, not just the shell
|
||||
await page.wait_for_selector("div.product-list article", state="attached")
|
||||
```
|
||||
- **SPA rendering:** Wait for content selectors (`wait_for_selector`), not the page load event
|
||||
- **AJAX/Fetch waiting:** Use `page.expect_response("**/api/data*")` to intercept and wait for specific API calls
|
||||
- **Shadow DOM:** Playwright pierces open Shadow DOM with `>>` operator: `page.locator("custom-element >> .inner-class")`
|
||||
- **Lazy-loaded images:** Scroll elements into view with `scroll_into_view_if_needed()` to trigger loading
|
||||
|
||||
#### AJAX / Fetch Waiting
|
||||
Intercept and wait for specific API calls:
|
||||
```python
|
||||
async with page.expect_response("**/api/products*") as response_info:
|
||||
await page.click("button.load-more")
|
||||
response = await response_info.value
|
||||
data = await response.json() # You can use the API data directly
|
||||
```
|
||||
|
||||
#### Shadow DOM Traversal
|
||||
```python
|
||||
# Playwright pierces open Shadow DOM automatically with >>
|
||||
await page.locator("custom-element >> .inner-class").click()
|
||||
```
|
||||
|
||||
#### Lazy-Loaded Images
|
||||
Scroll elements into view to trigger lazy loading:
|
||||
```python
|
||||
images = await page.query_selector_all("img[data-src]")
|
||||
for img in images:
|
||||
await img.scroll_into_view_if_needed()
|
||||
await page.wait_for_timeout(200)
|
||||
```
|
||||
See [playwright_browser_api.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering/browser-automation/references/playwright_browser_api.md) for wait strategies, network interception, and Shadow DOM details.
|
||||
|
||||
### 8. Error Handling & Retry Logic
|
||||
|
||||
#### Retry Decorator Pattern
|
||||
```python
|
||||
import asyncio
|
||||
- **Retry with backoff:** Wrap page interactions in retry logic with exponential backoff (e.g., 1s, 2s, 4s)
|
||||
- **Fallback selectors:** On `TimeoutError`, try alternative selectors before failing
|
||||
- **Error-state screenshots:** Capture `page.screenshot(path="error-state.png")` on unexpected failures for debugging
|
||||
- **Rate limit detection:** Check for HTTP 429 responses and respect `Retry-After` headers
|
||||
|
||||
async def with_retry(coro_factory, max_retries=3, backoff_base=2):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
return await coro_factory()
|
||||
except Exception as e:
|
||||
if attempt == max_retries - 1:
|
||||
raise
|
||||
wait = backoff_base ** attempt
|
||||
print(f"Attempt {attempt + 1} failed: {e}. Retrying in {wait}s...")
|
||||
await asyncio.sleep(wait)
|
||||
```
|
||||
|
||||
#### Handling Common Failures
|
||||
```python
|
||||
from playwright.async_api import TimeoutError as PlaywrightTimeout
|
||||
|
||||
try:
|
||||
await page.click("button.submit", timeout=5000)
|
||||
except PlaywrightTimeout:
|
||||
# Element did not appear — page structure may have changed
|
||||
# Try fallback selector
|
||||
await page.click("[type='submit']", timeout=5000)
|
||||
except Exception as e:
|
||||
# Network error, browser crash, etc.
|
||||
await page.screenshot(path="error-state.png")
|
||||
raise
|
||||
```
|
||||
|
||||
#### Rate Limit Detection
|
||||
```python
|
||||
async def check_rate_limit(response):
|
||||
if response.status == 429:
|
||||
retry_after = response.headers.get("retry-after", "60")
|
||||
wait_seconds = int(retry_after)
|
||||
print(f"Rate limited. Waiting {wait_seconds}s...")
|
||||
await asyncio.sleep(wait_seconds)
|
||||
return True
|
||||
return False
|
||||
```
|
||||
See [anti_detection_patterns.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering/browser-automation/references/anti_detection_patterns.md) for the complete exponential backoff implementation and rate limiter class.
|
||||
|
||||
## Workflows
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Database Designer - POWERFUL Tier Skill — Agent Skill for Codex & OpenClaw"
|
||||
description: "Database Designer - POWERFUL Tier Skill. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
description: "Use when the user asks to design database schemas, plan data migrations, optimize queries, choose between SQL and NoSQL, or model data relationships. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
---
|
||||
|
||||
# Database Designer - POWERFUL Tier Skill
|
||||
@@ -70,6 +70,229 @@ A comprehensive database design skill that provides expert-level analysis, optim
|
||||
4. **Validate inputs**: Prevent SQL injection attacks
|
||||
5. **Regular security updates**: Keep database software current
|
||||
|
||||
## Query Generation Patterns
|
||||
|
||||
### SELECT with JOINs
|
||||
|
||||
```sql
|
||||
-- INNER JOIN: only matching rows
|
||||
SELECT o.id, c.name, o.total
|
||||
FROM orders o
|
||||
INNER JOIN customers c ON c.id = o.customer_id;
|
||||
|
||||
-- LEFT JOIN: all left rows, NULLs for non-matches
|
||||
SELECT c.name, COUNT(o.id) AS order_count
|
||||
FROM customers c
|
||||
LEFT JOIN orders o ON o.customer_id = c.id
|
||||
GROUP BY c.name;
|
||||
|
||||
-- Self-join: hierarchical data (employees/managers)
|
||||
SELECT e.name AS employee, m.name AS manager
|
||||
FROM employees e
|
||||
LEFT JOIN employees m ON m.id = e.manager_id;
|
||||
```
|
||||
|
||||
### Common Table Expressions (CTEs)
|
||||
|
||||
```sql
|
||||
-- Recursive CTE for org chart
|
||||
WITH RECURSIVE org AS (
|
||||
SELECT id, name, manager_id, 1 AS depth
|
||||
FROM employees WHERE manager_id IS NULL
|
||||
UNION ALL
|
||||
SELECT e.id, e.name, e.manager_id, o.depth + 1
|
||||
FROM employees e INNER JOIN org o ON o.id = e.manager_id
|
||||
)
|
||||
SELECT * FROM org ORDER BY depth, name;
|
||||
```
|
||||
|
||||
### Window Functions
|
||||
|
||||
```sql
|
||||
-- ROW_NUMBER for pagination / dedup
|
||||
SELECT *, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY created_at DESC) AS rn
|
||||
FROM orders;
|
||||
|
||||
-- RANK with gaps, DENSE_RANK without gaps
|
||||
SELECT name, score, RANK() OVER (ORDER BY score DESC) AS rank FROM leaderboard;
|
||||
|
||||
-- LAG/LEAD for comparing adjacent rows
|
||||
SELECT date, revenue,
|
||||
revenue - LAG(revenue) OVER (ORDER BY date) AS daily_change
|
||||
FROM daily_sales;
|
||||
```
|
||||
|
||||
### Aggregation Patterns
|
||||
|
||||
```sql
|
||||
-- FILTER clause (PostgreSQL) for conditional aggregation
|
||||
SELECT
|
||||
COUNT(*) AS total,
|
||||
COUNT(*) FILTER (WHERE status = 'active') AS active,
|
||||
AVG(amount) FILTER (WHERE amount > 0) AS avg_positive
|
||||
FROM accounts;
|
||||
|
||||
-- GROUPING SETS for multi-level rollups
|
||||
SELECT region, product, SUM(revenue)
|
||||
FROM sales
|
||||
GROUP BY GROUPING SETS ((region, product), (region), ());
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Patterns
|
||||
|
||||
### Up/Down Migration Scripts
|
||||
|
||||
Every migration must have a reversible counterpart. Name files with a timestamp prefix for ordering:
|
||||
|
||||
```
|
||||
migrations/
|
||||
├── 20260101_000001_create_users.up.sql
|
||||
├── 20260101_000001_create_users.down.sql
|
||||
├── 20260115_000002_add_users_email_index.up.sql
|
||||
└── 20260115_000002_add_users_email_index.down.sql
|
||||
```
|
||||
|
||||
### Zero-Downtime Migrations (Expand/Contract)
|
||||
|
||||
Use the expand-contract pattern to avoid locking or breaking running code:
|
||||
|
||||
1. **Expand** — add the new column/table (nullable, with default)
|
||||
2. **Migrate data** — backfill in batches; dual-write from application
|
||||
3. **Transition** — application reads from new column; stop writing to old
|
||||
4. **Contract** — drop old column in a follow-up migration
|
||||
|
||||
### Data Backfill Strategies
|
||||
|
||||
```sql
|
||||
-- Batch update to avoid long-running locks
|
||||
UPDATE users SET email_normalized = LOWER(email)
|
||||
WHERE id IN (SELECT id FROM users WHERE email_normalized IS NULL LIMIT 5000);
|
||||
-- Repeat in a loop until 0 rows affected
|
||||
```
|
||||
|
||||
### Rollback Procedures
|
||||
|
||||
- Always test the `down.sql` in staging before deploying `up.sql` to production
|
||||
- Keep rollback window short — if the contract step has run, rollback requires a new forward migration
|
||||
- For irreversible changes (dropping columns with data), take a logical backup first
|
||||
|
||||
---
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Indexing Strategies
|
||||
|
||||
| Index Type | Use Case | Example |
|
||||
|------------|----------|---------|
|
||||
| **B-tree** (default) | Equality, range, ORDER BY | `CREATE INDEX idx_users_email ON users(email);` |
|
||||
| **GIN** | Full-text search, JSONB, arrays | `CREATE INDEX idx_docs_body ON docs USING gin(to_tsvector('english', body));` |
|
||||
| **GiST** | Geometry, range types, nearest-neighbor | `CREATE INDEX idx_locations ON places USING gist(coords);` |
|
||||
| **Partial** | Subset of rows (reduce size) | `CREATE INDEX idx_active ON users(email) WHERE active = true;` |
|
||||
| **Covering** | Index-only scans | `CREATE INDEX idx_cov ON orders(customer_id) INCLUDE (total, created_at);` |
|
||||
|
||||
### EXPLAIN Plan Reading
|
||||
|
||||
```sql
|
||||
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT) SELECT ...;
|
||||
```
|
||||
|
||||
Key signals to watch:
|
||||
- **Seq Scan** on large tables — missing index
|
||||
- **Nested Loop** with high row estimates — consider hash/merge join or add index
|
||||
- **Buffers shared read** much higher than **hit** — working set exceeds memory
|
||||
|
||||
### N+1 Query Detection
|
||||
|
||||
Symptoms: application issues one query per row (e.g., fetching related records in a loop).
|
||||
|
||||
Fixes:
|
||||
- Use `JOIN` or subquery to fetch in one round-trip
|
||||
- ORM eager loading (`select_related` / `includes` / `with`)
|
||||
- DataLoader pattern for GraphQL resolvers
|
||||
|
||||
### Connection Pooling
|
||||
|
||||
| Tool | Protocol | Best For |
|
||||
|------|----------|----------|
|
||||
| **PgBouncer** | PostgreSQL | Transaction/statement pooling, low overhead |
|
||||
| **ProxySQL** | MySQL | Query routing, read/write splitting |
|
||||
| **Built-in pool** (HikariCP, SQLAlchemy pool) | Any | Application-level pooling |
|
||||
|
||||
**Rule of thumb:** Set pool size to `(2 * CPU cores) + disk spindles`. For cloud SSDs, start with `2 * vCPUs` and tune.
|
||||
|
||||
### Read Replicas and Query Routing
|
||||
|
||||
- Route all `SELECT` queries to replicas; writes to primary
|
||||
- Account for replication lag (typically <1s for async, 0 for sync)
|
||||
- Use `pg_last_wal_replay_lsn()` to detect lag before reading critical data
|
||||
|
||||
---
|
||||
|
||||
## Multi-Database Decision Matrix
|
||||
|
||||
| Criteria | PostgreSQL | MySQL | SQLite | SQL Server |
|
||||
|----------|-----------|-------|--------|------------|
|
||||
| **Best for** | Complex queries, JSONB, extensions | Web apps, read-heavy workloads | Embedded, dev/test, edge | Enterprise .NET stacks |
|
||||
| **JSON support** | Excellent (JSONB + GIN) | Good (JSON type) | Minimal | Good (OPENJSON) |
|
||||
| **Replication** | Streaming, logical | Group replication, InnoDB cluster | N/A | Always On AG |
|
||||
| **Licensing** | Open source (PostgreSQL License) | Open source (GPL) / commercial | Public domain | Commercial |
|
||||
| **Max practical size** | Multi-TB | Multi-TB | ~1 TB (single-writer) | Multi-TB |
|
||||
|
||||
**When to choose:**
|
||||
- **PostgreSQL** — default choice for new projects; best extensibility and standards compliance
|
||||
- **MySQL** — existing MySQL ecosystem; simple read-heavy web applications
|
||||
- **SQLite** — mobile apps, CLI tools, unit test databases, IoT/edge
|
||||
- **SQL Server** — mandated by enterprise policy; deep .NET/Azure integration
|
||||
|
||||
### NoSQL Considerations
|
||||
|
||||
| Database | Model | Use When |
|
||||
|----------|-------|----------|
|
||||
| **MongoDB** | Document | Schema flexibility, rapid prototyping, content management |
|
||||
| **Redis** | Key-value / cache | Session store, rate limiting, leaderboards, pub/sub |
|
||||
| **DynamoDB** | Wide-column | Serverless AWS apps, single-digit-ms latency at any scale |
|
||||
|
||||
> Use SQL as default. Reach for NoSQL only when the access pattern clearly benefits from it.
|
||||
|
||||
---
|
||||
|
||||
## Sharding & Replication
|
||||
|
||||
### Horizontal vs Vertical Partitioning
|
||||
|
||||
- **Vertical partitioning**: Split columns across tables (e.g., separate BLOB columns). Reduces I/O for narrow queries.
|
||||
- **Horizontal partitioning (sharding)**: Split rows across databases/servers. Required when a single node cannot hold the dataset or handle the throughput.
|
||||
|
||||
### Sharding Strategies
|
||||
|
||||
| Strategy | How It Works | Pros | Cons |
|
||||
|----------|-------------|------|------|
|
||||
| **Hash** | `shard = hash(key) % N` | Even distribution | Resharding is expensive |
|
||||
| **Range** | Shard by date or ID range | Simple, good for time-series | Hot spots on latest shard |
|
||||
| **Geographic** | Shard by user region | Data locality, compliance | Cross-region queries are hard |
|
||||
|
||||
### Replication Patterns
|
||||
|
||||
| Pattern | Consistency | Latency | Use Case |
|
||||
|---------|------------|---------|----------|
|
||||
| **Synchronous** | Strong | Higher write latency | Financial transactions |
|
||||
| **Asynchronous** | Eventual | Low write latency | Read-heavy web apps |
|
||||
| **Semi-synchronous** | At-least-one replica confirmed | Moderate | Balance of safety and speed |
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
- **sql-database-assistant** — query writing, optimization, and debugging for day-to-day SQL work
|
||||
- **database-schema-designer** — ERD modeling, normalization analysis, and schema generation
|
||||
- **migration-architect** — large-scale migration planning across database engines or major schema overhauls
|
||||
- **senior-backend** — application-layer patterns (connection pooling, ORM best practices)
|
||||
- **senior-devops** — infrastructure provisioning for database clusters and replicas
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
Effective database design requires balancing multiple competing concerns: performance, scalability, maintainability, and business requirements. This skill provides the tools and knowledge to make informed decisions throughout the database lifecycle, from initial schema design through production optimization and evolution.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Database Schema Designer — Agent Skill for Codex & OpenClaw"
|
||||
description: "Database Schema Designer. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
description: "Use when the user asks to create ERD diagrams, normalize database schemas, design table relationships, or plan schema migrations. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
---
|
||||
|
||||
# Database Schema Designer
|
||||
|
||||
@@ -87,3 +87,185 @@ python3 scripts/env_auditor.py /path/to/repo --json
|
||||
2. Keep dev env files local and gitignored.
|
||||
3. Enforce detection in CI before merge.
|
||||
4. Re-test application paths immediately after credential rotation.
|
||||
|
||||
---
|
||||
|
||||
## Cloud Secret Store Integration
|
||||
|
||||
Production applications should never read secrets from `.env` files or environment variables baked into container images. Use a dedicated secret store instead.
|
||||
|
||||
### Provider Comparison
|
||||
|
||||
| Provider | Best For | Key Feature |
|
||||
|----------|----------|-------------|
|
||||
| **HashiCorp Vault** | Multi-cloud / hybrid | Dynamic secrets, policy engine, pluggable backends |
|
||||
| **AWS Secrets Manager** | AWS-native workloads | Native Lambda/ECS/EKS integration, automatic RDS rotation |
|
||||
| **Azure Key Vault** | Azure-native workloads | Managed HSM, Azure AD RBAC, certificate management |
|
||||
| **GCP Secret Manager** | GCP-native workloads | IAM-based access, automatic replication, versioning |
|
||||
|
||||
### Selection Guidance
|
||||
|
||||
- **Single cloud provider** — use the cloud-native secret manager. It integrates tightly with IAM, reduces operational overhead, and costs less than self-hosting.
|
||||
- **Multi-cloud or hybrid** — use HashiCorp Vault. It provides a uniform API across environments and supports dynamic secret generation (database credentials, cloud IAM keys) that expire automatically.
|
||||
- **Kubernetes-heavy** — combine External Secrets Operator with any backend above to sync secrets into K8s `Secret` objects without hardcoding.
|
||||
|
||||
### Application Access Patterns
|
||||
|
||||
1. **SDK/API pull** — application fetches secret at startup or on-demand via provider SDK.
|
||||
2. **Sidecar injection** — a sidecar container (e.g., Vault Agent) writes secrets to a shared volume or injects them as environment variables.
|
||||
3. **Init container** — a Kubernetes init container fetches secrets before the main container starts.
|
||||
4. **CSI driver** — secrets mount as a filesystem volume via the Secrets Store CSI Driver.
|
||||
|
||||
> **Cross-reference:** See `engineering/secrets-vault-manager` for production vault infrastructure patterns, HA deployment, and disaster recovery procedures.
|
||||
|
||||
---
|
||||
|
||||
## Secret Rotation Workflow
|
||||
|
||||
Stale secrets are a liability. Rotation ensures that even if a credential leaks, its useful lifetime is bounded.
|
||||
|
||||
### Phase 1: Detection
|
||||
|
||||
- Track secret creation and expiry dates in your secret store metadata.
|
||||
- Set alerts at 30, 14, and 7 days before expiry.
|
||||
- Use `scripts/env_auditor.py` to flag secrets with no recorded rotation date.
|
||||
|
||||
### Phase 2: Rotation
|
||||
|
||||
1. **Generate** a new credential (API key, database password, certificate).
|
||||
2. **Deploy** the new credential to all consumers (apps, services, pipelines) in parallel.
|
||||
3. **Verify** each consumer can authenticate using the new credential.
|
||||
4. **Revoke** the old credential only after all consumers are confirmed healthy.
|
||||
5. **Update** metadata with the new rotation timestamp and next rotation date.
|
||||
|
||||
### Phase 3: Automation
|
||||
|
||||
- **AWS Secrets Manager** — use built-in Lambda-based rotation for RDS, Redshift, and DocumentDB.
|
||||
- **HashiCorp Vault** — configure dynamic secrets with TTLs; credentials are generated on-demand and auto-expire.
|
||||
- **Azure Key Vault** — use Event Grid notifications to trigger rotation functions.
|
||||
- **GCP Secret Manager** — use Pub/Sub notifications tied to Cloud Functions for rotation logic.
|
||||
|
||||
### Emergency Rotation Checklist
|
||||
|
||||
When a secret is confirmed leaked:
|
||||
|
||||
1. **Immediately revoke** the compromised credential at the provider level.
|
||||
2. Generate and deploy a replacement credential to all consumers.
|
||||
3. Audit access logs for unauthorized usage during the exposure window.
|
||||
4. Scan git history, CI logs, and artifact registries for the leaked value.
|
||||
5. File an incident report documenting scope, timeline, and remediation steps.
|
||||
6. Review and tighten detection controls to prevent recurrence.
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Secret Injection
|
||||
|
||||
Secrets in CI/CD pipelines require careful handling to avoid exposure in logs, artifacts, or pull request contexts.
|
||||
|
||||
### GitHub Actions
|
||||
|
||||
- Use **repository secrets** or **environment secrets** via `${{ secrets.SECRET_NAME }}`.
|
||||
- Prefer **OIDC federation** (`aws-actions/configure-aws-credentials` with `role-to-assume`) over long-lived access keys.
|
||||
- Environment secrets with required reviewers add approval gates for production deployments.
|
||||
- GitHub automatically masks secrets in logs, but avoid `echo` or `toJSON()` on secret values.
|
||||
|
||||
### GitLab CI
|
||||
|
||||
- Store secrets as **CI/CD variables** with the `masked` and `protected` flags enabled.
|
||||
- Use **HashiCorp Vault integration** (`secrets:vault`) for dynamic secret injection without storing values in GitLab.
|
||||
- Scope variables to specific environments (`production`, `staging`) to enforce least privilege.
|
||||
|
||||
### Universal Patterns
|
||||
|
||||
- **Never echo or print** secret values in pipeline output, even for debugging.
|
||||
- **Use short-lived tokens** (OIDC, STS AssumeRole) instead of static credentials wherever possible.
|
||||
- **Restrict PR access** — do not expose secrets to pipelines triggered by forks or untrusted branches.
|
||||
- **Rotate CI secrets** on the same schedule as application secrets; pipeline credentials are attack vectors too.
|
||||
- **Audit pipeline logs** periodically for accidental secret exposure that masking may have missed.
|
||||
|
||||
---
|
||||
|
||||
## Pre-Commit Secret Detection
|
||||
|
||||
Catching secrets before they reach version control is the most cost-effective defense. Two leading tools cover this space.
|
||||
|
||||
### gitleaks
|
||||
|
||||
```toml
|
||||
# .gitleaks.toml — minimal configuration
|
||||
[extend]
|
||||
useDefault = true
|
||||
|
||||
[[rules]]
|
||||
id = "custom-internal-token"
|
||||
description = "Internal service token pattern"
|
||||
regex = '''INTERNAL_TOKEN_[A-Za-z0-9]{32}'''
|
||||
secretGroup = 0
|
||||
```
|
||||
|
||||
- Install: `brew install gitleaks` or download from GitHub releases.
|
||||
- Pre-commit hook: `gitleaks git --pre-commit --staged`
|
||||
- Baseline scanning: `gitleaks detect --source . --report-path gitleaks-report.json`
|
||||
- Manage false positives in `.gitleaksignore` (one fingerprint per line).
|
||||
|
||||
### detect-secrets
|
||||
|
||||
```bash
|
||||
# Generate baseline
|
||||
detect-secrets scan --all-files > .secrets.baseline
|
||||
|
||||
# Pre-commit hook (via pre-commit framework)
|
||||
# .pre-commit-config.yaml
|
||||
repos:
|
||||
- repo: https://github.com/Yelp/detect-secrets
|
||||
rev: v1.5.0
|
||||
hooks:
|
||||
- id: detect-secrets
|
||||
args: ['--baseline', '.secrets.baseline']
|
||||
```
|
||||
|
||||
- Supports **custom plugins** for organization-specific patterns.
|
||||
- Audit workflow: `detect-secrets audit .secrets.baseline` interactively marks true/false positives.
|
||||
|
||||
### False Positive Management
|
||||
|
||||
- Maintain `.gitleaksignore` or `.secrets.baseline` in version control so the whole team shares exclusions.
|
||||
- Review false positive lists during security audits — patterns may mask real leaks over time.
|
||||
- Prefer tightening regex patterns over broadly ignoring files.
|
||||
|
||||
---
|
||||
|
||||
## Audit Logging
|
||||
|
||||
Knowing who accessed which secret and when is critical for incident investigation and compliance.
|
||||
|
||||
### Cloud-Native Audit Trails
|
||||
|
||||
| Provider | Service | What It Captures |
|
||||
|----------|---------|-----------------|
|
||||
| **AWS** | CloudTrail | Every `GetSecretValue`, `DescribeSecret`, `RotateSecret` API call |
|
||||
| **Azure** | Activity Log + Diagnostic Logs | Key Vault access events, including caller identity and IP |
|
||||
| **GCP** | Cloud Audit Logs | Data access logs for Secret Manager with principal and timestamp |
|
||||
| **Vault** | Audit Backend | Full request/response logging (file, syslog, or socket backend) |
|
||||
|
||||
### Alerting Strategy
|
||||
|
||||
- Alert on **access from unknown IP ranges** or service accounts outside the expected set.
|
||||
- Alert on **bulk secret reads** (more than N secrets accessed within a time window).
|
||||
- Alert on **access outside deployment windows** when no CI/CD pipeline is running.
|
||||
- Feed audit logs into your SIEM (Splunk, Datadog, Elastic) for correlation with other security events.
|
||||
- Review audit logs quarterly as part of access recertification.
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
This skill covers env hygiene and secret detection. For deeper coverage of related domains, see:
|
||||
|
||||
| Skill | Path | Relationship |
|
||||
|-------|------|-------------|
|
||||
| **Secrets Vault Manager** | `engineering/secrets-vault-manager` | Production vault infrastructure, HA deployment, DR |
|
||||
| **Senior SecOps** | `engineering/senior-secops` | Security operations perspective, incident response |
|
||||
| **CI/CD Pipeline Builder** | `engineering/ci-cd-pipeline-builder` | Pipeline architecture, secret injection patterns |
|
||||
| **Infrastructure as Code** | `engineering/infrastructure-as-code` | Terraform/Pulumi secret backend configuration |
|
||||
| **Container Orchestration** | `engineering/container-orchestration` | Kubernetes secret mounting, sealed secrets |
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
---
|
||||
title: "Engineering - POWERFUL Skills — Agent Skills & Codex Plugins"
|
||||
description: "46 engineering - powerful skills — advanced agent-native skill and Claude Code plugin for AI agent design, infrastructure, and automation. Works with Claude Code, Codex CLI, Gemini CLI, and OpenClaw."
|
||||
description: "48 engineering - powerful skills — advanced agent-native skill and Claude Code plugin for AI agent design, infrastructure, and automation. Works with Claude Code, Codex CLI, Gemini CLI, and OpenClaw."
|
||||
---
|
||||
|
||||
<div class="domain-header" markdown>
|
||||
|
||||
# :material-rocket-launch: Engineering - POWERFUL
|
||||
|
||||
<p class="domain-count">46 skills in this domain</p>
|
||||
<p class="domain-count">48 skills in this domain</p>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "PR Review Expert — Agent Skill for Codex & OpenClaw"
|
||||
description: "PR Review Expert. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
description: "Use when the user asks to review pull requests, analyze code changes, check for security issues in PRs, or assess code quality of diffs. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
---
|
||||
|
||||
# PR Review Expert
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "RAG Architect — Agent Skill for Codex & OpenClaw"
|
||||
description: "RAG Architect - POWERFUL. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
description: "Use when the user asks to design RAG pipelines, optimize retrieval strategies, choose embedding models, implement vector search, or build knowledge. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
---
|
||||
|
||||
# RAG Architect
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Release Manager — Agent Skill for Codex & OpenClaw"
|
||||
description: "Release Manager. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
description: "Use when the user asks to plan releases, manage changelogs, coordinate deployments, create release branches, or automate versioning. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
---
|
||||
|
||||
# Release Manager
|
||||
|
||||
@@ -45,185 +45,32 @@ If the spec is not written, reviewed, and approved, implementation does not begi
|
||||
|
||||
Every spec follows this structure. No sections are optional — if a section does not apply, write "N/A — [reason]" so reviewers know it was considered, not forgotten.
|
||||
|
||||
### 1. Title and Context
|
||||
### Mandatory Sections
|
||||
|
||||
```markdown
|
||||
# Spec: [Feature Name]
|
||||
| # | Section | Key Rules |
|
||||
|---|---------|-----------|
|
||||
| 1 | **Title and Metadata** | Author, date, status (Draft/In Review/Approved/Superseded), reviewers |
|
||||
| 2 | **Context** | Why this feature exists. 2-4 paragraphs with evidence (metrics, tickets). |
|
||||
| 3 | **Functional Requirements** | RFC 2119 keywords (MUST/SHOULD/MAY). Numbered FR-N. Each is atomic and testable. |
|
||||
| 4 | **Non-Functional Requirements** | Performance, security, accessibility, scalability, reliability — all with measurable thresholds. |
|
||||
| 5 | **Acceptance Criteria** | Given/When/Then format. Every AC references at least one FR-* or NFR-*. |
|
||||
| 6 | **Edge Cases** | Numbered EC-N. Cover failure modes for every external dependency. |
|
||||
| 7 | **API Contracts** | TypeScript-style interfaces. Cover success and error responses. |
|
||||
| 8 | **Data Models** | Table format with field, type, constraints. Every entity from requirements must have a model. |
|
||||
| 9 | **Out of Scope** | Explicit exclusions with reasons. Prevents scope creep during implementation. |
|
||||
|
||||
**Author:** [name]
|
||||
**Date:** [ISO 8601]
|
||||
**Status:** Draft | In Review | Approved | Superseded
|
||||
**Reviewers:** [list]
|
||||
**Related specs:** [links]
|
||||
|
||||
## Context
|
||||
|
||||
[Why does this feature exist? What problem does it solve? What is the business
|
||||
motivation? Include links to user research, support tickets, or metrics that
|
||||
justify this work. 2-4 paragraphs maximum.]
|
||||
```
|
||||
|
||||
### 2. Functional Requirements (RFC 2119)
|
||||
|
||||
Use RFC 2119 keywords precisely:
|
||||
### RFC 2119 Keywords
|
||||
|
||||
| Keyword | Meaning |
|
||||
|---------|---------|
|
||||
| **MUST** | Absolute requirement. Failing this means the implementation is non-conformant. |
|
||||
| **MUST NOT** | Absolute prohibition. Doing this means the implementation is broken. |
|
||||
| **SHOULD** | Recommended. May be omitted with documented justification. |
|
||||
| **SHOULD NOT** | Discouraged. May be included with documented justification. |
|
||||
| **MAY** | Optional. Purely at the implementer's discretion. |
|
||||
| **MUST** | Absolute requirement. Non-conformant without it. |
|
||||
| **MUST NOT** | Absolute prohibition. |
|
||||
| **SHOULD** | Recommended. Omit only with documented justification. |
|
||||
| **MAY** | Optional. Implementer's discretion. |
|
||||
|
||||
```markdown
|
||||
## Functional Requirements
|
||||
See [spec_format_guide.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering/spec-driven-workflow/references/spec_format_guide.md) for the complete template with section-by-section examples, good/bad requirement patterns, and feature-type templates (CRUD, Integration, Migration).
|
||||
|
||||
- FR-1: The system MUST authenticate users via OAuth 2.0 PKCE flow.
|
||||
- FR-2: The system MUST reject tokens older than 24 hours.
|
||||
- FR-3: The system SHOULD support refresh token rotation.
|
||||
- FR-4: The system MAY cache user profiles for up to 5 minutes.
|
||||
- FR-5: The system MUST NOT store plaintext passwords under any circumstance.
|
||||
```
|
||||
|
||||
Number every requirement. Use `FR-` prefix. Each requirement is a single, testable statement.
|
||||
|
||||
### 3. Non-Functional Requirements
|
||||
|
||||
```markdown
|
||||
## Non-Functional Requirements
|
||||
|
||||
### Performance
|
||||
- NFR-P1: Login flow MUST complete in < 500ms (p95) under normal load.
|
||||
- NFR-P2: Token validation MUST complete in < 50ms (p99).
|
||||
|
||||
### Security
|
||||
- NFR-S1: All tokens MUST be transmitted over TLS 1.2+.
|
||||
- NFR-S2: The system MUST rate-limit login attempts to 5/minute per IP.
|
||||
|
||||
### Accessibility
|
||||
- NFR-A1: Login form MUST meet WCAG 2.1 AA standards.
|
||||
- NFR-A2: Error messages MUST be announced to screen readers.
|
||||
|
||||
### Scalability
|
||||
- NFR-SC1: The system SHOULD handle 10,000 concurrent sessions.
|
||||
|
||||
### Reliability
|
||||
- NFR-R1: The authentication service MUST maintain 99.9% uptime.
|
||||
```
|
||||
|
||||
### 4. Acceptance Criteria (Given/When/Then)
|
||||
|
||||
Every functional requirement maps to one or more acceptance criteria. Use Gherkin syntax:
|
||||
|
||||
```markdown
|
||||
## Acceptance Criteria
|
||||
|
||||
### AC-1: Successful login (FR-1)
|
||||
Given a user with valid credentials
|
||||
When they submit the login form with correct email and password
|
||||
Then they receive a valid access token
|
||||
And they are redirected to the dashboard
|
||||
And the login event is logged with timestamp and IP
|
||||
|
||||
### AC-2: Expired token rejection (FR-2)
|
||||
Given a user with an access token issued 25 hours ago
|
||||
When they make an API request with that token
|
||||
Then they receive a 401 Unauthorized response
|
||||
And the response body contains error code "TOKEN_EXPIRED"
|
||||
And they are NOT redirected (API clients handle their own flow)
|
||||
|
||||
### AC-3: Rate limiting (NFR-S2)
|
||||
Given an IP address that has made 5 failed login attempts in the last minute
|
||||
When a 6th login attempt arrives from that IP
|
||||
Then the request is rejected with 429 Too Many Requests
|
||||
And the response includes a Retry-After header
|
||||
```
|
||||
|
||||
### 5. Edge Cases and Error Scenarios
|
||||
|
||||
```markdown
|
||||
## Edge Cases
|
||||
|
||||
- EC-1: User submits login form with empty email → Show validation error, do not hit API.
|
||||
- EC-2: OAuth provider is down → Show "Service temporarily unavailable", retry after 30s.
|
||||
- EC-3: User has account but no password (social-only) → Redirect to social login.
|
||||
- EC-4: Concurrent login from two devices → Both sessions are valid (no single-session enforcement).
|
||||
- EC-5: Token expires mid-request → Complete the current request, return warning header.
|
||||
```
|
||||
|
||||
### 6. API Contracts
|
||||
|
||||
Define request/response shapes using TypeScript-style notation:
|
||||
|
||||
```markdown
|
||||
## API Contracts
|
||||
|
||||
### POST /api/auth/login
|
||||
Request:
|
||||
```typescript
|
||||
interface LoginRequest {
|
||||
email: string; // MUST be valid email format
|
||||
password: string; // MUST be 8-128 characters
|
||||
rememberMe?: boolean; // Default: false
|
||||
}
|
||||
```
|
||||
|
||||
Success Response (200):
|
||||
```typescript
|
||||
interface LoginResponse {
|
||||
accessToken: string; // JWT, expires in 24h
|
||||
refreshToken: string; // Opaque, expires in 30d
|
||||
expiresIn: number; // Seconds until access token expires
|
||||
user: {
|
||||
id: string;
|
||||
email: string;
|
||||
displayName: string;
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
Error Response (401):
|
||||
```typescript
|
||||
interface AuthError {
|
||||
error: "INVALID_CREDENTIALS" | "TOKEN_EXPIRED" | "ACCOUNT_LOCKED";
|
||||
message: string;
|
||||
retryAfter?: number; // Seconds, present for rate-limited responses
|
||||
}
|
||||
```
|
||||
```
|
||||
|
||||
### 7. Data Models
|
||||
|
||||
```markdown
|
||||
## Data Models
|
||||
|
||||
### User
|
||||
| Field | Type | Constraints |
|
||||
|-------|------|-------------|
|
||||
| id | UUID | Primary key, auto-generated |
|
||||
| email | string | Unique, max 255 chars, valid email format |
|
||||
| passwordHash | string | bcrypt, never exposed via API |
|
||||
| createdAt | timestamp | UTC, immutable |
|
||||
| lastLoginAt | timestamp | UTC, updated on each login |
|
||||
| loginAttempts | integer | Reset to 0 on successful login |
|
||||
| lockedUntil | timestamp | Null if not locked |
|
||||
```
|
||||
|
||||
### 8. Out of Scope
|
||||
|
||||
Explicit exclusions prevent scope creep:
|
||||
|
||||
```markdown
|
||||
## Out of Scope
|
||||
|
||||
- OS-1: Multi-factor authentication (separate spec: SPEC-042)
|
||||
- OS-2: Social login providers beyond Google and GitHub
|
||||
- OS-3: Admin impersonation of user accounts
|
||||
- OS-4: Password complexity rules beyond minimum length (deferred to v2)
|
||||
- OS-5: Session management UI (users cannot see/revoke active sessions yet)
|
||||
```
|
||||
|
||||
If someone asks for an out-of-scope item during implementation, point them to this section. Do not build it.
|
||||
See [acceptance_criteria_patterns.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering/spec-driven-workflow/references/acceptance_criteria_patterns.md) for a full pattern library of Given/When/Then criteria across authentication, CRUD, search, file upload, payment, notification, and accessibility scenarios.
|
||||
|
||||
---
|
||||
|
||||
@@ -416,107 +263,7 @@ Use `engineering/spec-driven-workflow` for:
|
||||
|
||||
## Examples
|
||||
|
||||
### Full Spec: User Password Reset
|
||||
|
||||
```markdown
|
||||
# Spec: Password Reset Flow
|
||||
|
||||
**Author:** Engineering Team
|
||||
**Date:** 2026-03-25
|
||||
**Status:** Approved
|
||||
|
||||
## Context
|
||||
|
||||
Users who forget their passwords currently have no self-service recovery option.
|
||||
Support receives ~200 password reset requests per week, costing approximately
|
||||
8 hours of support time. This feature eliminates that burden entirely.
|
||||
|
||||
## Functional Requirements
|
||||
|
||||
- FR-1: The system MUST allow users to request a password reset via email.
|
||||
- FR-2: The system MUST send a reset link that expires after 1 hour.
|
||||
- FR-3: The system MUST invalidate all previous reset links when a new one is requested.
|
||||
- FR-4: The system MUST enforce minimum password length of 8 characters on reset.
|
||||
- FR-5: The system MUST NOT reveal whether an email exists in the system.
|
||||
- FR-6: The system SHOULD log all reset attempts for audit purposes.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### AC-1: Request reset (FR-1, FR-5)
|
||||
Given a user on the password reset page
|
||||
When they enter any email address and submit
|
||||
Then they see "If an account exists, a reset link has been sent"
|
||||
And the response is identical whether the email exists or not
|
||||
|
||||
### AC-2: Valid reset link (FR-2)
|
||||
Given a user who received a reset email 30 minutes ago
|
||||
When they click the reset link
|
||||
Then they see the password reset form
|
||||
|
||||
### AC-3: Expired reset link (FR-2)
|
||||
Given a user who received a reset email 2 hours ago
|
||||
When they click the reset link
|
||||
Then they see "This link has expired. Please request a new one."
|
||||
|
||||
### AC-4: Previous links invalidated (FR-3)
|
||||
Given a user who requested two reset emails
|
||||
When they click the link from the first email
|
||||
Then they see "This link is no longer valid."
|
||||
|
||||
## Edge Cases
|
||||
|
||||
- EC-1: User submits reset for non-existent email → Same success message (FR-5).
|
||||
- EC-2: User clicks reset link twice → Second click shows "already used" if password was changed.
|
||||
- EC-3: Email delivery fails → Log error, do not retry automatically.
|
||||
- EC-4: User requests reset while already logged in → Allow it, do not force logout.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- OS-1: Security questions as alternative reset method.
|
||||
- OS-2: SMS-based password reset.
|
||||
- OS-3: Admin-initiated password reset (separate spec).
|
||||
```
|
||||
|
||||
### Extracted Test Cases (from above spec)
|
||||
|
||||
```python
|
||||
# Generated by test_extractor.py --framework pytest
|
||||
|
||||
class TestPasswordReset:
|
||||
def test_ac1_request_reset_existing_email(self):
|
||||
"""AC-1: Request reset with existing email shows generic message."""
|
||||
# Given a user on the password reset page
|
||||
# When they enter a registered email and submit
|
||||
# Then they see "If an account exists, a reset link has been sent"
|
||||
raise NotImplementedError("Implement this test")
|
||||
|
||||
def test_ac1_request_reset_nonexistent_email(self):
|
||||
"""AC-1: Request reset with unknown email shows same generic message."""
|
||||
# Given a user on the password reset page
|
||||
# When they enter an unregistered email and submit
|
||||
# Then they see identical response to existing email case
|
||||
raise NotImplementedError("Implement this test")
|
||||
|
||||
def test_ac2_valid_reset_link(self):
|
||||
"""AC-2: Reset link works within expiry window."""
|
||||
raise NotImplementedError("Implement this test")
|
||||
|
||||
def test_ac3_expired_reset_link(self):
|
||||
"""AC-3: Reset link rejected after 1 hour."""
|
||||
raise NotImplementedError("Implement this test")
|
||||
|
||||
def test_ac4_previous_links_invalidated(self):
|
||||
"""AC-4: Old reset links stop working when new one is requested."""
|
||||
raise NotImplementedError("Implement this test")
|
||||
|
||||
def test_ec1_nonexistent_email_same_response(self):
|
||||
"""EC-1: Non-existent email produces identical response."""
|
||||
raise NotImplementedError("Implement this test")
|
||||
|
||||
def test_ec2_reset_link_used_twice(self):
|
||||
"""EC-2: Already-used reset link shows appropriate message."""
|
||||
raise NotImplementedError("Implement this test")
|
||||
```
|
||||
A complete worked example (Password Reset spec with extracted test cases) is available in [spec_format_guide.md](https://github.com/alirezarezvani/claude-skills/tree/main/engineering/spec-driven-workflow/references/spec_format_guide.md#full-example-password-reset). It demonstrates all 9 sections, requirement numbering, acceptance criteria, edge cases, and the corresponding pytest stubs generated by `test_extractor.py`.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -464,6 +464,259 @@ Flag these without being asked:
|
||||
|
||||
---
|
||||
|
||||
## Multi-Cloud Provider Configuration
|
||||
|
||||
When a single root module must provision across AWS, Azure, and GCP simultaneously.
|
||||
|
||||
### Provider Aliasing Pattern
|
||||
|
||||
```hcl
|
||||
terraform {
|
||||
required_providers {
|
||||
aws = {
|
||||
source = "hashicorp/aws"
|
||||
version = "~> 5.0"
|
||||
}
|
||||
azurerm = {
|
||||
source = "hashicorp/azurerm"
|
||||
version = "~> 3.0"
|
||||
}
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "~> 5.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "aws" {
|
||||
region = var.aws_region
|
||||
}
|
||||
|
||||
provider "azurerm" {
|
||||
features {}
|
||||
subscription_id = var.azure_subscription_id
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
project = var.gcp_project_id
|
||||
region = var.gcp_region
|
||||
}
|
||||
```
|
||||
|
||||
### Shared Variables Across Providers
|
||||
|
||||
```hcl
|
||||
variable "environment" {
|
||||
description = "Environment name used across all providers"
|
||||
type = string
|
||||
validation {
|
||||
condition = contains(["dev", "staging", "prod"], var.environment)
|
||||
error_message = "Must be dev, staging, or prod."
|
||||
}
|
||||
}
|
||||
|
||||
locals {
|
||||
common_tags = {
|
||||
environment = var.environment
|
||||
managed_by = "terraform"
|
||||
project = var.project_name
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### When to Use Multi-Cloud
|
||||
|
||||
- **Yes**: Regulatory requirements mandate data residency across providers, or the org has existing workloads on multiple clouds.
|
||||
- **No**: "Avoiding vendor lock-in" alone is not sufficient justification. Multi-cloud doubles operational complexity. Prefer single-cloud unless there is a concrete business requirement.
|
||||
|
||||
---
|
||||
|
||||
## OpenTofu Compatibility
|
||||
|
||||
OpenTofu is an open-source fork of Terraform maintained by the Linux Foundation under the MPL 2.0 license.
|
||||
|
||||
### Migration from Terraform to OpenTofu
|
||||
|
||||
```bash
|
||||
# 1. Install OpenTofu
|
||||
brew install opentofu # macOS
|
||||
snap install --classic tofu # Linux
|
||||
|
||||
# 2. Replace the binary — state files are compatible
|
||||
tofu init # Re-initializes with OpenTofu
|
||||
tofu plan # Identical plan output
|
||||
tofu apply # Same apply workflow
|
||||
```
|
||||
|
||||
### License Considerations
|
||||
|
||||
| | Terraform (1.6+) | OpenTofu |
|
||||
|---|---|---|
|
||||
| **License** | BSL 1.1 (source-available) | MPL 2.0 (open-source) |
|
||||
| **Commercial use** | Restricted for competing products | Unrestricted |
|
||||
| **Community governance** | HashiCorp | Linux Foundation |
|
||||
|
||||
### Feature Parity
|
||||
|
||||
OpenTofu tracks Terraform 1.6.x features. Key additions unique to OpenTofu:
|
||||
- Client-side state encryption (`tofu init -encryption`)
|
||||
- Early variable/locals evaluation
|
||||
- Provider-defined functions
|
||||
|
||||
### When to Choose OpenTofu
|
||||
|
||||
- You need a fully open-source license for your supply chain.
|
||||
- You want client-side state encryption without Terraform Cloud.
|
||||
- Otherwise, either tool works — the HCL syntax and provider ecosystem are identical.
|
||||
|
||||
---
|
||||
|
||||
## Infracost Integration
|
||||
|
||||
Infracost estimates cloud costs from Terraform code before resources are provisioned.
|
||||
|
||||
### PR Workflow
|
||||
|
||||
```bash
|
||||
# Show cost breakdown for current code
|
||||
infracost breakdown --path .
|
||||
|
||||
# Compare cost difference between current branch and main
|
||||
infracost diff --path . --compare-to infracost-base.json
|
||||
```
|
||||
|
||||
### GitHub Actions Cost Comment
|
||||
|
||||
```yaml
|
||||
# .github/workflows/infracost.yml
|
||||
name: Infracost
|
||||
on: [pull_request]
|
||||
|
||||
jobs:
|
||||
cost:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: infracost/actions/setup@v3
|
||||
with:
|
||||
api-key: ${{ secrets.INFRACOST_API_KEY }}
|
||||
- run: infracost breakdown --path ./terraform --format json --out-file /tmp/infracost.json
|
||||
- run: infracost comment github --path /tmp/infracost.json --repo $GITHUB_REPOSITORY --pull-request ${{ github.event.pull_request.number }} --github-token ${{ secrets.GITHUB_TOKEN }} --behavior update
|
||||
```
|
||||
|
||||
### Budget Thresholds and Cost Policy
|
||||
|
||||
```yaml
|
||||
# infracost.yml — policy file
|
||||
version: 0.1
|
||||
policies:
|
||||
- path: "*"
|
||||
max_monthly_cost: "5000" # Fail PR if estimated cost exceeds $5,000/month
|
||||
max_cost_increase: "500" # Fail PR if cost increase exceeds $500/month
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Import Existing Infrastructure
|
||||
|
||||
Bring manually-created resources under Terraform management.
|
||||
|
||||
### terraform import Workflow
|
||||
|
||||
```bash
|
||||
# 1. Write the resource block first (empty body is fine)
|
||||
# main.tf:
|
||||
# resource "aws_s3_bucket" "legacy" {}
|
||||
|
||||
# 2. Import the resource into state
|
||||
terraform import aws_s3_bucket.legacy my-existing-bucket-name
|
||||
|
||||
# 3. Run plan to see attribute diff
|
||||
terraform plan
|
||||
|
||||
# 4. Fill in the resource block until plan shows no changes
|
||||
```
|
||||
|
||||
### Bulk Import with Config Generation (Terraform 1.5+)
|
||||
|
||||
```bash
|
||||
# Generate HCL for imported resources
|
||||
terraform plan -generate-config-out=generated.tf
|
||||
|
||||
# Review generated.tf, then move resources into proper files
|
||||
```
|
||||
|
||||
### Common Pitfalls
|
||||
|
||||
- **Resource drift after import**: The imported resource may have attributes Terraform does not manage. Run `terraform plan` immediately and resolve every diff.
|
||||
- **State manipulation**: Use `terraform state mv` to rename or reorganize. Use `terraform state rm` to remove without destroying. Always back up state before manipulation: `terraform state pull > backup.tfstate`.
|
||||
- **Sensitive defaults**: Imported resources may expose secrets in state. Restrict state access and enable encryption.
|
||||
|
||||
---
|
||||
|
||||
## Terragrunt Patterns
|
||||
|
||||
Terragrunt is a thin wrapper around Terraform that provides DRY configuration for multi-environment setups.
|
||||
|
||||
### Root terragrunt.hcl (Shared Config)
|
||||
|
||||
```hcl
|
||||
# terragrunt.hcl (root)
|
||||
remote_state {
|
||||
backend = "s3"
|
||||
generate = {
|
||||
path = "backend.tf"
|
||||
if_exists = "overwrite_terragrunt"
|
||||
}
|
||||
config = {
|
||||
bucket = "my-org-terraform-state"
|
||||
key = "${path_relative_to_include()}/terraform.tfstate"
|
||||
region = "us-east-1"
|
||||
encrypt = true
|
||||
dynamodb_table = "terraform-locks"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Child terragrunt.hcl (Environment Override)
|
||||
|
||||
```hcl
|
||||
# prod/vpc/terragrunt.hcl
|
||||
include "root" {
|
||||
path = find_in_parent_folders()
|
||||
}
|
||||
|
||||
terraform {
|
||||
source = "../../modules/vpc"
|
||||
}
|
||||
|
||||
inputs = {
|
||||
environment = "prod"
|
||||
cidr_block = "10.0.0.0/16"
|
||||
}
|
||||
```
|
||||
|
||||
### Dependencies Between Modules
|
||||
|
||||
```hcl
|
||||
# prod/eks/terragrunt.hcl
|
||||
dependency "vpc" {
|
||||
config_path = "../vpc"
|
||||
}
|
||||
|
||||
inputs = {
|
||||
vpc_id = dependency.vpc.outputs.vpc_id
|
||||
subnet_ids = dependency.vpc.outputs.private_subnet_ids
|
||||
}
|
||||
```
|
||||
|
||||
### When Terragrunt Adds Value
|
||||
|
||||
- **Yes**: 3+ environments with identical module structure, shared backend config, or cross-module dependencies.
|
||||
- **No**: Single environment, small team, or simple directory-based isolation already works. Terragrunt adds a learning curve and another binary to manage.
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
### One-liner (any tool)
|
||||
|
||||
Reference in New Issue
Block a user