diff --git a/engineering-team/security-pen-testing/SKILL.md b/engineering-team/security-pen-testing/SKILL.md index 50accab..458e1c0 100644 --- a/engineering-team/security-pen-testing/SKILL.md +++ b/engineering-team/security-pen-testing/SKILL.md @@ -60,353 +60,71 @@ python scripts/vulnerability_scanner.py --target web --scope full python scripts/vulnerability_scanner.py --target api --scope quick --json ``` -### A01:2021 — Broken Access Control +### Quick Reference -**Test Procedures:** -1. Attempt horizontal privilege escalation: access another user's resources by changing IDs -2. Test vertical escalation: access admin endpoints with regular user tokens -3. Verify CORS configuration — check `Access-Control-Allow-Origin` for wildcards -4. Test forced browsing to admin pages (`/admin`, `/api/admin`, `/debug`) -5. Modify JWT claims (`role`, `is_admin`) and replay tokens - -**What to Look For:** -- Missing authorization checks on API endpoints -- Predictable resource IDs (sequential integers vs. UUIDs) -- Client-side only access controls (hidden UI elements without server checks) -- CORS misconfigurations allowing arbitrary origins - -### A02:2021 — Cryptographic Failures - -**Test Procedures:** -1. Check TLS version — reject anything below TLS 1.2 -2. Verify password hashing: bcrypt/scrypt/argon2 with adequate cost factor -3. Look for sensitive data in URLs (tokens in query params get logged) -4. Check for hardcoded encryption keys in source code -5. Test for weak random number generation (Math.random() for tokens) - -**What to Look For:** -- MD5/SHA1 used for password hashing -- Secrets in environment variables without encryption at rest -- Missing `Strict-Transport-Security` header -- Self-signed certificates in production - -### A03:2021 — Injection - -**Test Procedures:** -1. SQL injection: test all input fields with `' OR 1=1--` and time-based payloads -2. NoSQL injection: test with `{"$gt": ""}` and `{"$ne": null}` in JSON bodies -3. Command injection: test inputs with `; whoami` and backtick substitution -4. LDAP injection: test with `*)(uid=*))(|(uid=*` -5. Template injection: test with `{{7*7}}` and `${7*7}` - -**What to Look For:** -- String concatenation in SQL queries -- User input passed to `eval()`, `exec()`, `os.system()` -- Unparameterized ORM queries -- Template engines rendering user input without sandboxing - -### A04:2021 — Insecure Design - -**Test Procedures:** -1. Review business logic flows for abuse scenarios (e.g., negative quantities in carts) -2. Check rate limiting on sensitive operations (login, password reset, OTP) -3. Test multi-step flows for state manipulation (skip payment step) -4. Verify security questions aren't guessable - -**What to Look For:** -- Missing rate limits on authentication endpoints -- Business logic that trusts client-side calculations -- Lack of account lockout after failed attempts -- Missing CAPTCHA on public-facing forms - -### A05:2021 — Security Misconfiguration - -**Test Procedures:** -1. Check for default credentials on admin panels -2. Verify unnecessary HTTP methods are disabled (TRACE, DELETE on public endpoints) -3. Check error handling — stack traces should never leak to users -4. Review HTTP security headers (CSP, X-Frame-Options, X-Content-Type-Options) -5. Check directory listing is disabled - -**What to Look For:** -- Debug mode enabled in production -- Default admin:admin credentials -- Verbose error messages with stack traces -- Missing security headers - -### A06:2021 — Vulnerable and Outdated Components - -**Test Procedures:** -1. Run dependency audit against known CVE databases -2. Check for end-of-life frameworks and libraries -3. Verify transitive dependency versions -4. Check for known vulnerable versions (e.g., Log4j 2.0-2.14.1) +| # | Category | Key Tests | +|---|----------|-----------| +| A01 | Broken Access Control | IDOR, vertical escalation, CORS, JWT claim manipulation, forced browsing | +| A02 | Cryptographic Failures | TLS version, password hashing, hardcoded keys, weak PRNG | +| A03 | Injection | SQLi, NoSQLi, command injection, template injection, XSS | +| A04 | Insecure Design | Rate limiting, business logic abuse, multi-step flow bypass | +| A05 | Security Misconfiguration | Default credentials, debug mode, security headers, directory listing | +| A06 | Vulnerable Components | Dependency audit (npm/pip/go), EOL checks, known CVEs | +| A07 | Auth Failures | Brute force, session cookie flags, session invalidation, MFA bypass | +| A08 | Integrity Failures | Unsafe deserialization, SRI checks, CI/CD pipeline integrity | +| A09 | Logging Failures | Auth event logging, sensitive data in logs, alerting thresholds | +| A10 | SSRF | Internal IP access, cloud metadata endpoints, DNS rebinding | ```bash -# Audit a package manifest +# Audit dependencies python scripts/dependency_auditor.py --file package.json --severity high python scripts/dependency_auditor.py --file requirements.txt --json ``` -### A07:2021 — Identification and Authentication Failures - -**Test Procedures:** -1. Test brute force protection on login endpoints -2. Check password policy enforcement (minimum length, complexity) -3. Verify session invalidation on logout and password change -4. Test "remember me" token security (HttpOnly, Secure, SameSite flags) -5. Check multi-factor authentication bypass paths - -**What to Look For:** -- Sessions that persist after logout -- Missing `HttpOnly` and `Secure` flags on session cookies -- Password reset tokens that don't expire -- Username enumeration via different error messages - -### A08:2021 — Software and Data Integrity Failures - -**Test Procedures:** -1. Check for unsigned updates or deployment artifacts -2. Verify CI/CD pipeline integrity (signed commits, protected branches) -3. Test deserialization endpoints with crafted payloads -4. Check for SRI (Subresource Integrity) on CDN-loaded scripts - -**What to Look For:** -- Unsafe deserialization of user input (pickle, Java serialization) -- Missing integrity checks on downloaded artifacts -- CI/CD pipelines running untrusted code -- CDN scripts without SRI hashes - -### A09:2021 — Security Logging and Monitoring Failures - -**Test Procedures:** -1. Verify authentication events are logged (success and failure) -2. Check that logs don't contain sensitive data (passwords, tokens, PII) -3. Test alerting thresholds (do 50 failed logins trigger an alert?) -4. Verify log integrity — can an attacker tamper with logs? - -**What to Look For:** -- Missing audit trail for admin actions -- Passwords or tokens appearing in logs -- No alerting on suspicious patterns -- Logs stored without integrity protection - -### A10:2021 — Server-Side Request Forgery (SSRF) - -**Test Procedures:** -1. Test URL input fields with internal addresses (`http://169.254.169.254/` for cloud metadata) -2. Check for open redirect chains that reach internal services -3. Test with DNS rebinding payloads -4. Verify allowlist validation on outbound requests - -**What to Look For:** -- User-controlled URLs passed to `fetch()`, `requests.get()`, `curl` -- Missing allowlist on outbound HTTP requests -- Ability to reach cloud metadata endpoints (AWS, GCP, Azure) -- PDF generators or screenshot services that fetch arbitrary URLs +See [owasp_top_10_checklist.md](references/owasp_top_10_checklist.md) for detailed test procedures, code patterns to detect, remediation steps, and CVSS scoring guidance for each category. --- ## Static Analysis -### CodeQL Custom Rules +**Recommended tools:** CodeQL (custom queries for project-specific patterns), Semgrep (rule-based scanning with auto-fix), ESLint security plugins (`eslint-plugin-security`, `eslint-plugin-no-unsanitized`). -Write custom CodeQL queries for project-specific vulnerability patterns: +Key patterns to detect: SQL injection via string concatenation, hardcoded JWT secrets, unsafe YAML/pickle deserialization, missing security middleware (e.g., Express without Helmet). -```ql -/** - * Detect SQL injection via string concatenation - */ -import python -import semmle.python.dataflow.new.DataFlow - -from Call call, StringFormatting fmt -where - call.getFunc().getName() = "execute" and - fmt = call.getArg(0) and - exists(DataFlow::Node source | - source.asExpr() instanceof Name and - DataFlow::localFlow(source, DataFlow::exprNode(fmt.getAnOperand())) - ) -select call, "Potential SQL injection: user input flows into execute()" -``` - -### Semgrep Custom Rules - -Create project-specific Semgrep rules: - -```yaml -rules: - - id: hardcoded-jwt-secret - pattern: | - jwt.encode($PAYLOAD, "...", ...) - message: "JWT signed with hardcoded secret" - severity: ERROR - languages: [python] - - - id: unsafe-yaml-load - pattern: yaml.load($DATA) - fix: yaml.safe_load($DATA) - message: "Use yaml.safe_load() to prevent arbitrary code execution" - severity: WARNING - languages: [python] - - - id: express-no-helmet - pattern: | - const app = express(); - ... - app.listen(...) - pattern-not: | - const app = express(); - ... - app.use(helmet(...)); - ... - app.listen(...) - message: "Express app missing helmet middleware for security headers" - severity: WARNING - languages: [javascript, typescript] -``` - -### ESLint Security Plugins - -Recommended configuration: - -```json -{ - "plugins": ["security", "no-unsanitized"], - "extends": ["plugin:security/recommended"], - "rules": { - "security/detect-object-injection": "error", - "security/detect-non-literal-regexp": "warn", - "security/detect-unsafe-regex": "error", - "security/detect-buffer-noassert": "error", - "security/detect-eval-with-expression": "error", - "no-unsanitized/method": "error", - "no-unsanitized/property": "error" - } -} -``` +See [attack_patterns.md](references/attack_patterns.md) for code patterns and detection payloads across injection types. --- ## Dependency Vulnerability Scanning -### Ecosystem-Specific Commands +**Ecosystem commands:** `npm audit`, `pip audit`, `govulncheck ./...`, `bundle audit check` + +**CVE Triage Workflow:** +1. **Collect** — Run ecosystem audit tools, aggregate findings +2. **Deduplicate** — Group by CVE ID across direct and transitive deps +3. **Prioritize** — Critical + exploitable + reachable = fix immediately +4. **Remediate** — Upgrade, patch, or mitigate with compensating controls +5. **Verify** — Rerun audit to confirm fix, update lock files ```bash -# Node.js -npm audit --json | jq '.vulnerabilities | to_entries[] | select(.value.severity == "critical")' - -# Python -pip audit --format json --desc -safety check --json - -# Go -govulncheck ./... - -# Ruby -bundle audit check --update -``` - -### CVE Triage Workflow - -1. **Collect**: Run ecosystem audit tools, aggregate findings -2. **Deduplicate**: Group by CVE ID across direct and transitive deps -3. **Score**: Use CVSS base score + environmental adjustments -4. **Prioritize**: Critical + exploitable + reachable = fix immediately -5. **Remediate**: Upgrade, patch, or mitigate with compensating controls -6. **Verify**: Rerun audit to confirm fix, update lock files - -```bash -# Use the dependency auditor for automated triage python scripts/dependency_auditor.py --file package.json --severity critical --json ``` -### Known Vulnerable Patterns - -| Package | Vulnerable Versions | CVE | Impact | -|---------|-------------------|-----|--------| -| log4j-core | 2.0 - 2.14.1 | CVE-2021-44228 | RCE via JNDI injection | -| lodash | < 4.17.21 | CVE-2021-23337 | Prototype pollution | -| axios | < 1.6.0 | CVE-2023-45857 | CSRF token exposure | -| pillow | < 9.3.0 | CVE-2022-45198 | DoS via crafted image | -| express | < 4.19.2 | CVE-2024-29041 | Open redirect | - --- ## Secret Scanning -### TruffleHog Patterns +**Tools:** TruffleHog (git history + filesystem), Gitleaks (regex-based with custom rules). ```bash -# Scan git history for secrets +# Scan git history for verified secrets trufflehog git file://. --only-verified --json -# Scan filesystem (no git history) +# Scan filesystem trufflehog filesystem . --json ``` -### Gitleaks Configuration - -```toml -# .gitleaks.toml -title = "Custom Gitleaks Config" - -[[rules]] -id = "aws-access-key" -description = "AWS Access Key ID" -regex = '''AKIA[0-9A-Z]{16}''' -tags = ["aws", "credentials"] - -[[rules]] -id = "generic-api-key" -description = "Generic API Key" -regex = '''(?i)(api[_-]?key|apikey)\s*[:=]\s*['\"][a-zA-Z0-9]{20,}['\"]''' -tags = ["api", "key"] - -[[rules]] -id = "private-key" -description = "Private Key Header" -regex = '''-----BEGIN (RSA|EC|DSA|OPENSSH) PRIVATE KEY-----''' -tags = ["private-key"] - -[allowlist] -paths = ['''\.test\.''', '''_test\.go''', '''mock''', '''fixture'''] -``` - -### Pre-commit Hook Integration - -```yaml -# .pre-commit-config.yaml -repos: - - repo: https://github.com/gitleaks/gitleaks - rev: v8.18.0 - hooks: - - id: gitleaks - - - repo: https://github.com/trufflesecurity/trufflehog - rev: v3.63.0 - hooks: - - id: trufflehog - args: ["git", "file://.", "--since-commit", "HEAD", "--only-verified"] -``` - -### CI Integration (GitHub Actions) - -```yaml -name: Secret Scan -on: [push, pull_request] -jobs: - scan: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - uses: trufflesecurity/trufflehog@main - with: - extra_args: --only-verified -``` +**Integration points:** Pre-commit hooks (gitleaks, trufflehog), CI/CD gates (GitHub Actions with `trufflesecurity/trufflehog@main`). Configure `.gitleaks.toml` for custom rules (AWS keys, API keys, private key headers) and allowlists for test fixtures. --- @@ -414,252 +132,45 @@ jobs: ### Authentication Bypass -**JWT Manipulation:** -1. Decode token at jwt.io — inspect claims without verification -2. Change `alg` to `none` and remove signature: `eyJ...payload.` -3. Change `alg` from RS256 to HS256 and sign with the public key -4. Modify claims (`role: "admin"`, `exp: 9999999999`) and re-sign with weak secrets -5. Test key confusion: HMAC signed with RSA public key bytes - -**Session Fixation:** -1. Obtain a session token before authentication -2. Authenticate — check if the session ID changes -3. If the same session ID persists, the app is vulnerable to session fixation +- **JWT manipulation:** Change `alg` to `none`, RS256-to-HS256 confusion, claim modification (`role: "admin"`, `exp: 9999999999`) +- **Session fixation:** Check if session ID changes after authentication ### Authorization Flaws -**IDOR (Insecure Direct Object Reference):** -``` -GET /api/users/123/profile → 200 (your profile) -GET /api/users/124/profile → 200 (someone else's profile — IDOR!) -GET /api/users/124/profile → 403 (properly protected) -``` +- **IDOR/BOLA:** Change resource IDs in every endpoint — test read, update, delete across users +- **BFLA:** Regular user tries admin endpoints (expect 403) +- **Mass assignment:** Add privileged fields (`role`, `is_admin`) to update requests -Test pattern: Change numeric IDs, UUIDs, slugs in every endpoint. Use Burp Intruder or a simple script to iterate. +### Rate Limiting & GraphQL -**BOLA (Broken Object Level Authorization):** -Same as IDOR but specifically in REST APIs. Test every CRUD operation: -- Can user A read user B's resource? -- Can user A update user B's resource? -- Can user A delete user B's resource? +- **Rate limiting:** Rapid-fire requests to auth endpoints; expect 429 after threshold +- **GraphQL:** Test introspection (should be disabled in prod), query depth attacks, batch mutations bypassing rate limits -**BFLA (Broken Function Level Authorization):** -``` -# Regular user tries admin endpoints -POST /api/admin/users → Should be 403 -DELETE /api/admin/users/123 → Should be 403 -PUT /api/settings/global → Should be 403 -``` - -### Rate Limiting Validation - -Test rate limits on critical endpoints: -```bash -# Rapid-fire login attempts -for i in $(seq 1 100); do - curl -s -o /dev/null -w "%{http_code}" \ - -X POST https://target.com/api/login \ - -d '{"email":"test@test.com","password":"wrong"}'; -done -# Expect: 429 after threshold (typically 5-10 attempts) -``` - -### Mass Assignment Detection - -```bash -# Try adding admin fields to a regular update request -PUT /api/users/profile -{ - "name": "Normal User", - "email": "user@test.com", - "role": "admin", # mass assignment attempt - "is_verified": true, # mass assignment attempt - "subscription": "enterprise" # mass assignment attempt -} -``` - -### GraphQL-Specific Testing - -**Introspection Query:** -```graphql -{ - __schema { - types { name fields { name type { name } } } - } -} -``` -Introspection should be **disabled in production**. - -**Query Depth Attack:** -```graphql -{ - user(id: 1) { - friends { - friends { - friends { - friends { # Keep nesting until server crashes - name - } - } - } - } - } -} -``` - -**Batching Attack:** -```json -[ - {"query": "mutation { login(user:\"admin\", pass:\"password1\") { token } }"}, - {"query": "mutation { login(user:\"admin\", pass:\"password2\") { token } }"}, - {"query": "mutation { login(user:\"admin\", pass:\"password3\") { token } }"} -] -``` -Batch mutations can bypass rate limiting if counted as a single request. +See [attack_patterns.md](references/attack_patterns.md) for complete JWT manipulation payloads, IDOR testing methodology, BFLA endpoint lists, GraphQL introspection/depth/batch attack patterns, and rate limiting bypass techniques. --- ## Web Vulnerability Testing -### XSS (Cross-Site Scripting) +| Vulnerability | Key Tests | +|--------------|-----------| +| **XSS** | Reflected (script/img/svg payloads), Stored (persistent fields), DOM-based (innerHTML + location.hash) | +| **CSRF** | Replay without token (expect 403), cross-session token replay, check SameSite cookie attribute | +| **SQL Injection** | Error-based (`' OR 1=1--`), union-based enumeration, time-based blind (`SLEEP(5)`), boolean-based blind | +| **SSRF** | Internal IPs, cloud metadata endpoints (AWS/GCP/Azure), IPv6/hex/decimal encoding bypasses | +| **Path Traversal** | `../../../etc/passwd`, URL encoding, double encoding bypasses | -**Reflected XSS Test Payloads** (non-destructive): -``` - -"> -javascript:alert(document.domain) - -'-alert(document.domain)-' - -``` - -**Stored XSS**: Submit payloads in persistent fields (comments, profiles, messages), then check if they render for other users. - -**DOM-Based XSS**: Look for `innerHTML`, `document.write()`, `eval()` operating on `location.hash`, `location.search`, or `document.referrer`. - -### CSRF Token Validation - -1. Capture a legitimate request with CSRF token -2. Replay the request without the token — should fail (403) -3. Replay with a token from a different session — should fail -4. Check if token changes per request or is static per session -5. Verify `SameSite` cookie attribute is set to `Strict` or `Lax` - -### SQL Injection - -**Detection Payloads** (safe, non-destructive): -``` -' OR '1'='1 -' OR '1'='1' -- -" OR "1"="1 -1 OR 1=1 -' UNION SELECT NULL-- -' AND SLEEP(5)-- (time-based blind) -' AND 1=1-- (boolean-based blind) -``` - -**Union-Based Enumeration** (authorized testing only): -```sql -' UNION SELECT 1,2,3-- -- Find column count -' UNION SELECT table_name,2,3 FROM information_schema.tables-- -' UNION SELECT column_name,2,3 FROM information_schema.columns WHERE table_name='users'-- -``` - -**Time-Based Blind:** -```sql -' AND IF(1=1, SLEEP(5), 0)-- -- MySQL -' AND pg_sleep(5)-- -- PostgreSQL -' WAITFOR DELAY '0:0:5'-- -- MSSQL -``` - -### SSRF Detection - -**Payloads for SSRF testing:** -``` -http://127.0.0.1 -http://localhost -http://169.254.169.254/latest/meta-data/ (AWS metadata) -http://metadata.google.internal/ (GCP metadata) -http://169.254.169.254/metadata/instance (Azure metadata) -http://[::1] (IPv6 localhost) -http://0x7f000001 (hex encoding) -http://2130706433 (decimal encoding) -``` - -### Path Traversal - -``` -GET /api/files?name=../../../etc/passwd -GET /api/files?name=....//....//....//etc/passwd -GET /api/files?name=%2e%2e%2f%2e%2e%2f%2e%2e%2fetc%2fpasswd -GET /api/files?name=..%252f..%252f..%252fetc%252fpasswd (double encoding) -``` +See [attack_patterns.md](references/attack_patterns.md) for complete test payloads (XSS filter bypasses, context-specific XSS, SQL injection per database engine, SSRF bypass techniques, and DOM-based XSS source/sink pairs). --- ## Infrastructure Security -### Misconfigured Cloud Storage - -**S3 Bucket Checks:** -```bash -# Check for public read access -aws s3 ls s3://target-bucket --no-sign-request - -# Check bucket policy -aws s3api get-bucket-policy --bucket target-bucket - -# Check ACL -aws s3api get-bucket-acl --bucket target-bucket -``` - -**Common Bucket Name Patterns:** -``` -{company}-backup, {company}-dev, {company}-staging -{company}-assets, {company}-uploads, {company}-logs -``` - -### HTTP Security Headers - -Required headers and expected values: - -| Header | Expected Value | -|--------|---------------| -| `Strict-Transport-Security` | `max-age=31536000; includeSubDomains; preload` | -| `Content-Security-Policy` | Restrictive policy, no `unsafe-inline` or `unsafe-eval` | -| `X-Content-Type-Options` | `nosniff` | -| `X-Frame-Options` | `DENY` or `SAMEORIGIN` | -| `Referrer-Policy` | `strict-origin-when-cross-origin` | -| `Permissions-Policy` | Restrict camera, microphone, geolocation | -| `X-XSS-Protection` | `0` (deprecated, CSP is preferred) | - -### TLS Configuration - -```bash -# Check TLS version and cipher suites -nmap --script ssl-enum-ciphers -p 443 target.com - -# Quick check with testssl.sh -./testssl.sh target.com - -# Check certificate expiry -echo | openssl s_client -connect target.com:443 2>/dev/null | openssl x509 -noout -dates -``` - -**Reject:** TLS 1.0, TLS 1.1, RC4, DES, 3DES, MD5 in cipher suites, CBC mode ciphers (BEAST), export-grade ciphers. - -### Open Port Scanning - -```bash -# Quick top-1000 ports -nmap -sV target.com - -# Full port scan -nmap -p- -sV target.com - -# Common dangerous open ports -# 21 (FTP), 23 (Telnet), 445 (SMB), 3389 (RDP), 6379 (Redis), 27017 (MongoDB) -``` +**Key checks:** +- **Cloud storage:** S3 bucket public access (`aws s3 ls s3://bucket --no-sign-request`), bucket policies, ACLs +- **HTTP security headers:** HSTS, CSP (no `unsafe-inline`/`unsafe-eval`), X-Content-Type-Options, X-Frame-Options, Referrer-Policy +- **TLS configuration:** `nmap --script ssl-enum-ciphers -p 443 target.com` or `testssl.sh` — reject TLS 1.0/1.1, RC4, 3DES, export-grade ciphers +- **Port scanning:** `nmap -sV target.com` — flag dangerous open ports (FTP/21, Telnet/23, Redis/6379, MongoDB/27017) --- @@ -708,26 +219,11 @@ python scripts/pentest_report_generator.py --findings findings.json --format jso ## Responsible Disclosure Workflow -Responsible disclosure is **mandatory** for any vulnerability found during authorized testing or independent research. See `references/responsible_disclosure.md` for full templates. +Responsible disclosure is **mandatory** for any vulnerability found during authorized testing. Standard timeline: report on day 1, follow up at day 7, status update at day 30, public disclosure at day 90. -### Timeline +**Key principles:** Never exploit beyond proof of concept, encrypt all communications, do not access real user data, document everything with timestamps. -| Day | Action | -|-----|--------| -| 0 | Discovery — document finding with evidence | -| 1 | Report to vendor via security contact or bug bounty program | -| 7 | Follow up if no acknowledgment received | -| 30 | Request status update and remediation timeline | -| 60 | Second follow-up — offer technical assistance | -| 90 | Public disclosure (with or without fix, per industry standard) | - -### Key Principles - -1. **Never exploit beyond proof of concept** — demonstrate impact without causing damage -2. **Encrypt all communications** — PGP/GPG for email, secure channels for details -3. **Do not access, modify, or exfiltrate real user data** — use your own test accounts -4. **Document everything** — timestamps, screenshots, request/response pairs -5. **Respect the vendor's timeline** — extend deadline if they're actively working on a fix +See [responsible_disclosure.md](references/responsible_disclosure.md) for full disclosure timelines (standard 90-day, accelerated 30-day, extended 120-day), communication templates, legal considerations, bug bounty program integration, and CVE request process. --- @@ -781,47 +277,7 @@ python scripts/pentest_report_generator.py --findings findings.json --format md ### Workflow 3: CI/CD Security Gate -Automated security checks that run on every pull request: - -```yaml -# .github/workflows/security-gate.yml -name: Security Gate -on: [pull_request] -jobs: - security: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - # Secret scanning - - name: Scan for secrets - uses: trufflesecurity/trufflehog@main - with: - extra_args: --only-verified - - # Dependency audit - - name: Audit dependencies - run: | - npm audit --audit-level=high - pip audit --desc - - # SAST - - name: Static analysis - uses: returntocorp/semgrep-action@v1 - with: - config: >- - p/security-audit - p/secrets - p/owasp-top-ten - - # Security headers check (staging only) - - name: Check security headers - if: github.base_ref == 'staging' - run: | - curl -sI $STAGING_URL | python scripts/vulnerability_scanner.py --target web --scope quick -``` +Automated security checks on every PR: secret scanning (TruffleHog), dependency audit (`npm audit`, `pip audit`), SAST (Semgrep with `p/security-audit`, `p/owasp-top-ten`), and security headers check on staging. **Gate Policy**: Block merge on critical/high findings. Warn on medium. Log low/info. diff --git a/engineering-team/security-pen-testing/references/attack_patterns.md b/engineering-team/security-pen-testing/references/attack_patterns.md index 6f59453..11c5b8d 100644 --- a/engineering-team/security-pen-testing/references/attack_patterns.md +++ b/engineering-team/security-pen-testing/references/attack_patterns.md @@ -547,3 +547,83 @@ PUT /api/login ``` If any of these bypass rate limiting, the implementation needs hardening. + +--- + +## Static Analysis Tool Configurations + +### CodeQL Custom Rules + +Write custom CodeQL queries for project-specific vulnerability patterns: + +```ql +/** + * Detect SQL injection via string concatenation + */ +import python +import semmle.python.dataflow.new.DataFlow + +from Call call, StringFormatting fmt +where + call.getFunc().getName() = "execute" and + fmt = call.getArg(0) and + exists(DataFlow::Node source | + source.asExpr() instanceof Name and + DataFlow::localFlow(source, DataFlow::exprNode(fmt.getAnOperand())) + ) +select call, "Potential SQL injection: user input flows into execute()" +``` + +### Semgrep Custom Rules + +```yaml +rules: + - id: hardcoded-jwt-secret + pattern: | + jwt.encode($PAYLOAD, "...", ...) + message: "JWT signed with hardcoded secret" + severity: ERROR + languages: [python] + + - id: unsafe-yaml-load + pattern: yaml.load($DATA) + fix: yaml.safe_load($DATA) + message: "Use yaml.safe_load() to prevent arbitrary code execution" + severity: WARNING + languages: [python] + + - id: express-no-helmet + pattern: | + const app = express(); + ... + app.listen(...) + pattern-not: | + const app = express(); + ... + app.use(helmet(...)); + ... + app.listen(...) + message: "Express app missing helmet middleware for security headers" + severity: WARNING + languages: [javascript, typescript] +``` + +### ESLint Security Plugins + +Recommended configuration: + +```json +{ + "plugins": ["security", "no-unsanitized"], + "extends": ["plugin:security/recommended"], + "rules": { + "security/detect-object-injection": "error", + "security/detect-non-literal-regexp": "warn", + "security/detect-unsafe-regex": "error", + "security/detect-buffer-noassert": "error", + "security/detect-eval-with-expression": "error", + "no-unsanitized/method": "error", + "no-unsanitized/property": "error" + } +} +``` diff --git a/engineering/browser-automation/SKILL.md b/engineering/browser-automation/SKILL.md index 1d8368c..cf566f4 100644 --- a/engineering/browser-automation/SKILL.md +++ b/engineering/browser-automation/SKILL.md @@ -33,9 +33,6 @@ The Browser Automation skill provides comprehensive tools and knowledge for buil ### 1. Web Scraping Patterns -#### DOM Extraction with CSS Selectors -CSS selectors are the primary tool for element targeting. Prefer them over XPath for readability and performance. - **Selector priority (most to least reliable):** 1. `data-testid`, `data-id`, or custom data attributes — stable across redesigns 2. `#id` selectors — unique but may change between deploys @@ -43,365 +40,70 @@ CSS selectors are the primary tool for element targeting. Prefer them over XPath 4. Class-based: `.product-card`, `.price` — brittle if classes are generated (e.g., CSS modules) 5. Positional: `nth-child()`, `nth-of-type()` — last resort, breaks on layout changes -**Compound selectors for precision:** -```python -# Product cards within a specific container -page.query_selector_all("div.search-results > article.product-card") +Use XPath only when CSS cannot express the relationship (e.g., ancestor traversal, text-based selection). -# Price inside a product card (scoped) -card.query_selector("span[data-field='price']") - -# Links with specific text content -page.locator("a", has_text="Next Page") -``` - -#### XPath for Complex Traversal -Use XPath only when CSS cannot express the relationship: -```python -# Find element by text content (XPath strength) -page.locator("//td[contains(text(), 'Total')]/following-sibling::td[1]") - -# Navigate up the DOM tree -page.locator("//span[@class='price']/ancestor::div[@class='product']") -``` - -#### Pagination Patterns -- **Next-button pagination**: Click "Next" until disabled or absent -- **URL-based pagination**: Increment `?page=N` or `&offset=N` in URL -- **Infinite scroll**: Scroll to bottom, wait for new content, repeat until no change -- **Load-more button**: Click button, wait for DOM mutation, repeat - -#### Infinite Scroll Handling -```python -async def scroll_to_bottom(page, max_scrolls=50, pause_ms=1500): - previous_height = 0 - for i in range(max_scrolls): - current_height = await page.evaluate("document.body.scrollHeight") - if current_height == previous_height: - break - await page.evaluate("window.scrollTo(0, document.body.scrollHeight)") - await page.wait_for_timeout(pause_ms) - previous_height = current_height - return i + 1 # number of scrolls performed -``` +**Pagination strategies:** next-button, URL-based (`?page=N`), infinite scroll, load-more button. See [data_extraction_recipes.md](references/data_extraction_recipes.md) for complete pagination handlers and scroll patterns. ### 2. Form Filling & Multi-Step Workflows -#### Login Flows -```python -async def login(page, url, username, password): - await page.goto(url) - await page.fill("input[name='username']", username) - await page.fill("input[name='password']", password) - await page.click("button[type='submit']") - # Wait for navigation to complete (post-login redirect) - await page.wait_for_url("**/dashboard**") -``` +Break multi-step forms into discrete functions per step. Each function fills fields, clicks "Next"/"Continue", and waits for the next step to load (URL change or DOM element). -#### Multi-Page Forms -Break multi-step forms into discrete functions per step. Each function: -1. Fills the fields for that step -2. Clicks the "Next" or "Continue" button -3. Waits for the next step to load (URL change or DOM element) - -```python -async def fill_step_1(page, data): - await page.fill("#first-name", data["first_name"]) - await page.fill("#last-name", data["last_name"]) - await page.select_option("#country", data["country"]) - await page.click("button:has-text('Continue')") - await page.wait_for_selector("#step-2-form") - -async def fill_step_2(page, data): - await page.fill("#address", data["address"]) - await page.fill("#city", data["city"]) - await page.click("button:has-text('Continue')") - await page.wait_for_selector("#step-3-form") -``` - -#### File Uploads -```python -# Single file -await page.set_input_files("input[type='file']", "/path/to/file.pdf") - -# Multiple files -await page.set_input_files("input[type='file']", [ - "/path/to/file1.pdf", - "/path/to/file2.pdf" -]) - -# Drag-and-drop upload zones (no visible input element) -async with page.expect_file_chooser() as fc_info: - await page.click("div.upload-zone") -file_chooser = await fc_info.value -await file_chooser.set_files("/path/to/file.pdf") -``` - -#### Dropdown and Select Handling -```python -# Native