Merge branch 'feature/sprint-phase-3-gaps' into dev
# Conflicts: # docs/skills/engineering-team/index.md # docs/skills/engineering/index.md # mkdocs.yml
This commit is contained in:
Reza Rezvani
429
docs/skills/engineering-team/gcp-cloud-architect.md
Normal file
429
docs/skills/engineering-team/gcp-cloud-architect.md
Normal file
@@ -0,0 +1,429 @@
|
||||
---
|
||||
title: "GCP Cloud Architect — Agent Skill & Codex Plugin"
|
||||
description: "Design GCP architectures for startups and enterprises. Use when asked to design Google Cloud infrastructure, deploy to GKE or Cloud Run, configure. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
---
|
||||
|
||||
# GCP Cloud Architect
|
||||
|
||||
<div class="page-meta" markdown>
|
||||
<span class="meta-badge">:material-code-braces: Engineering - Core</span>
|
||||
<span class="meta-badge">:material-identifier: `gcp-cloud-architect`</span>
|
||||
<span class="meta-badge">:material-github: <a href="https://github.com/alirezarezvani/claude-skills/tree/main/engineering-team/gcp-cloud-architect/SKILL.md">Source</a></span>
|
||||
</div>
|
||||
|
||||
<div class="install-banner" markdown>
|
||||
<span class="install-label">Install:</span> <code>claude /plugin install engineering-skills</code>
|
||||
</div>
|
||||
|
||||
|
||||
Design scalable, cost-effective Google Cloud architectures for startups and enterprises with infrastructure-as-code templates.
|
||||
|
||||
---
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: Gather Requirements
|
||||
|
||||
Collect application specifications:
|
||||
|
||||
```
|
||||
- Application type (web app, mobile backend, data pipeline, SaaS)
|
||||
- Expected users and requests per second
|
||||
- Budget constraints (monthly spend limit)
|
||||
- Team size and GCP experience level
|
||||
- Compliance requirements (GDPR, HIPAA, SOC 2)
|
||||
- Availability requirements (SLA, RPO/RTO)
|
||||
```
|
||||
|
||||
### Step 2: Design Architecture
|
||||
|
||||
Run the architecture designer to get pattern recommendations:
|
||||
|
||||
```bash
|
||||
python scripts/architecture_designer.py --input requirements.json
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
|
||||
```json
|
||||
{
|
||||
"recommended_pattern": "serverless_web",
|
||||
"service_stack": ["Cloud Storage", "Cloud CDN", "Cloud Run", "Firestore", "Identity Platform"],
|
||||
"estimated_monthly_cost_usd": 30,
|
||||
"pros": ["Low ops overhead", "Pay-per-use", "Auto-scaling", "No cold starts on Cloud Run min instances"],
|
||||
"cons": ["Vendor lock-in", "Regional limitations", "Eventual consistency with Firestore"]
|
||||
}
|
||||
```
|
||||
|
||||
Select from recommended patterns:
|
||||
- **Serverless Web**: Cloud Storage + Cloud CDN + Cloud Run + Firestore
|
||||
- **Microservices on GKE**: GKE Autopilot + Cloud SQL + Memorystore + Cloud Pub/Sub
|
||||
- **Serverless Data Pipeline**: Pub/Sub + Dataflow + BigQuery + Looker
|
||||
- **ML Platform**: Vertex AI + Cloud Storage + BigQuery + Cloud Functions
|
||||
|
||||
See `references/architecture_patterns.md` for detailed pattern specifications.
|
||||
|
||||
**Validation checkpoint:** Confirm the recommended pattern matches the team's operational maturity and compliance requirements before proceeding to Step 3.
|
||||
|
||||
### Step 3: Estimate Cost
|
||||
|
||||
Analyze estimated costs and optimization opportunities:
|
||||
|
||||
```bash
|
||||
python scripts/cost_optimizer.py --resources current_setup.json --monthly-spend 2000
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
|
||||
```json
|
||||
{
|
||||
"current_monthly_usd": 2000,
|
||||
"recommendations": [
|
||||
{ "action": "Right-size Cloud SQL db-custom-4-16384 to db-custom-2-8192", "savings_usd": 380, "priority": "high" },
|
||||
{ "action": "Purchase 1-yr committed use discount for GKE nodes", "savings_usd": 290, "priority": "high" },
|
||||
{ "action": "Move Cloud Storage objects >90 days to Nearline", "savings_usd": 75, "priority": "medium" }
|
||||
],
|
||||
"total_potential_savings_usd": 745
|
||||
}
|
||||
```
|
||||
|
||||
Output includes:
|
||||
- Monthly cost breakdown by service
|
||||
- Right-sizing recommendations
|
||||
- Committed use discount opportunities
|
||||
- Sustained use discount analysis
|
||||
- Potential monthly savings
|
||||
|
||||
Use the [GCP Pricing Calculator](https://cloud.google.com/products/calculator) for detailed estimates.
|
||||
|
||||
### Step 4: Generate IaC
|
||||
|
||||
Create infrastructure-as-code for the selected pattern:
|
||||
|
||||
```bash
|
||||
python scripts/deployment_manager.py --app-name my-app --pattern serverless_web --region us-central1
|
||||
```
|
||||
|
||||
**Example Terraform HCL output (Cloud Run + Firestore):**
|
||||
|
||||
```hcl
|
||||
terraform {
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "~> 5.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
}
|
||||
|
||||
variable "project_id" {
|
||||
description = "GCP project ID"
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
description = "GCP region"
|
||||
type = string
|
||||
default = "us-central1"
|
||||
}
|
||||
|
||||
resource "google_cloud_run_v2_service" "api" {
|
||||
name = "${var.environment}-${var.app_name}-api"
|
||||
location = var.region
|
||||
|
||||
template {
|
||||
containers {
|
||||
image = "gcr.io/${var.project_id}/${var.app_name}:latest"
|
||||
resources {
|
||||
limits = {
|
||||
cpu = "1000m"
|
||||
memory = "512Mi"
|
||||
}
|
||||
}
|
||||
env {
|
||||
name = "FIRESTORE_PROJECT"
|
||||
value = var.project_id
|
||||
}
|
||||
}
|
||||
scaling {
|
||||
min_instance_count = 0
|
||||
max_instance_count = 10
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_firestore_database" "default" {
|
||||
project = var.project_id
|
||||
name = "(default)"
|
||||
location_id = var.region
|
||||
type = "FIRESTORE_NATIVE"
|
||||
}
|
||||
```
|
||||
|
||||
**Example gcloud CLI deployment:**
|
||||
|
||||
```bash
|
||||
# Deploy Cloud Run service
|
||||
gcloud run deploy my-app-api \
|
||||
--image gcr.io/$PROJECT_ID/my-app:latest \
|
||||
--region us-central1 \
|
||||
--platform managed \
|
||||
--allow-unauthenticated \
|
||||
--memory 512Mi \
|
||||
--cpu 1 \
|
||||
--min-instances 0 \
|
||||
--max-instances 10
|
||||
|
||||
# Create Firestore database
|
||||
gcloud firestore databases create --location=us-central1
|
||||
```
|
||||
|
||||
> Full templates including Cloud CDN, Identity Platform, IAM, and Cloud Monitoring are generated by `deployment_manager.py` and also available in `references/architecture_patterns.md`.
|
||||
|
||||
### Step 5: Configure CI/CD
|
||||
|
||||
Set up automated deployment with Cloud Build or GitHub Actions:
|
||||
|
||||
```yaml
|
||||
# cloudbuild.yaml
|
||||
steps:
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args: ['build', '-t', 'gcr.io/$PROJECT_ID/my-app:$COMMIT_SHA', '.']
|
||||
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args: ['push', 'gcr.io/$PROJECT_ID/my-app:$COMMIT_SHA']
|
||||
|
||||
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
|
||||
entrypoint: gcloud
|
||||
args:
|
||||
- 'run'
|
||||
- 'deploy'
|
||||
- 'my-app-api'
|
||||
- '--image=gcr.io/$PROJECT_ID/my-app:$COMMIT_SHA'
|
||||
- '--region=us-central1'
|
||||
- '--platform=managed'
|
||||
|
||||
images:
|
||||
- 'gcr.io/$PROJECT_ID/my-app:$COMMIT_SHA'
|
||||
```
|
||||
|
||||
```bash
|
||||
# Connect repo and create trigger
|
||||
gcloud builds triggers create github \
|
||||
--repo-name=my-app \
|
||||
--repo-owner=my-org \
|
||||
--branch-pattern="^main$" \
|
||||
--build-config=cloudbuild.yaml
|
||||
```
|
||||
|
||||
### Step 6: Security Review
|
||||
|
||||
Verify security configuration:
|
||||
|
||||
```bash
|
||||
# Review IAM bindings
|
||||
gcloud projects get-iam-policy $PROJECT_ID --format=json
|
||||
|
||||
# Check service account permissions
|
||||
gcloud iam service-accounts list --project=$PROJECT_ID
|
||||
|
||||
# Verify VPC Service Controls (if applicable)
|
||||
gcloud access-context-manager perimeters list --policy=$POLICY_ID
|
||||
```
|
||||
|
||||
**Security checklist:**
|
||||
- IAM roles follow least privilege (prefer predefined roles over basic roles)
|
||||
- Service accounts use Workload Identity for GKE
|
||||
- VPC Service Controls configured for sensitive APIs
|
||||
- Cloud KMS encryption keys for customer-managed encryption
|
||||
- Cloud Audit Logs enabled for all admin activity
|
||||
- Organization policies restrict public access
|
||||
- Secret Manager used for all credentials
|
||||
|
||||
**If deployment fails:**
|
||||
|
||||
1. Check the failure reason:
|
||||
```bash
|
||||
gcloud run services describe my-app-api --region us-central1
|
||||
gcloud logging read "resource.type=cloud_run_revision" --limit=20
|
||||
```
|
||||
2. Review Cloud Logging for application errors.
|
||||
3. Fix the configuration or container image.
|
||||
4. Redeploy:
|
||||
```bash
|
||||
gcloud run deploy my-app-api --image gcr.io/$PROJECT_ID/my-app:latest --region us-central1
|
||||
```
|
||||
|
||||
**Common failure causes:**
|
||||
- IAM permission errors -- verify service account roles and `--allow-unauthenticated` flag
|
||||
- Quota exceeded -- request quota increase via IAM & Admin > Quotas
|
||||
- Container startup failure -- check container logs and health check configuration
|
||||
- Region not enabled -- enable the required APIs with `gcloud services enable`
|
||||
|
||||
---
|
||||
|
||||
## Tools
|
||||
|
||||
### architecture_designer.py
|
||||
|
||||
Recommends GCP services based on workload requirements.
|
||||
|
||||
```bash
|
||||
python scripts/architecture_designer.py --input requirements.json --output design.json
|
||||
```
|
||||
|
||||
**Input:** JSON with app type, scale, budget, compliance needs
|
||||
**Output:** Recommended pattern, service stack, cost estimate, pros/cons
|
||||
|
||||
### cost_optimizer.py
|
||||
|
||||
Analyzes GCP resources for cost savings.
|
||||
|
||||
```bash
|
||||
python scripts/cost_optimizer.py --resources inventory.json --monthly-spend 5000
|
||||
```
|
||||
|
||||
**Output:** Recommendations for:
|
||||
- Idle resource removal
|
||||
- Machine type right-sizing
|
||||
- Committed use discounts
|
||||
- Storage class transitions
|
||||
- Network egress optimization
|
||||
|
||||
### deployment_manager.py
|
||||
|
||||
Generates gcloud CLI deployment scripts and Terraform configurations.
|
||||
|
||||
```bash
|
||||
python scripts/deployment_manager.py --app-name my-app --pattern serverless_web --region us-central1
|
||||
```
|
||||
|
||||
**Output:** Production-ready deployment scripts with:
|
||||
- Cloud Run or GKE deployment
|
||||
- Firestore or Cloud SQL setup
|
||||
- Identity Platform configuration
|
||||
- IAM roles with least privilege
|
||||
- Cloud Monitoring and Logging
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Web App on Cloud Run (< $100/month)
|
||||
|
||||
```
|
||||
Ask: "Design a serverless web backend for a mobile app with 1000 users"
|
||||
|
||||
Result:
|
||||
- Cloud Run for API (auto-scaling, no cold start with min instances)
|
||||
- Firestore for data (pay-per-operation)
|
||||
- Identity Platform for authentication
|
||||
- Cloud Storage + Cloud CDN for static assets
|
||||
- Estimated: $15-40/month
|
||||
```
|
||||
|
||||
### Microservices on GKE ($500-2000/month)
|
||||
|
||||
```
|
||||
Ask: "Design a scalable architecture for a SaaS platform with 50k users"
|
||||
|
||||
Result:
|
||||
- GKE Autopilot for containerized workloads
|
||||
- Cloud SQL (PostgreSQL) with read replicas
|
||||
- Memorystore (Redis) for session caching
|
||||
- Cloud CDN for global delivery
|
||||
- Cloud Build for CI/CD
|
||||
- Multi-zone deployment
|
||||
```
|
||||
|
||||
### Serverless Data Pipeline
|
||||
|
||||
```
|
||||
Ask: "Design a real-time analytics pipeline for event data"
|
||||
|
||||
Result:
|
||||
- Pub/Sub for event ingestion
|
||||
- Dataflow (Apache Beam) for stream processing
|
||||
- BigQuery for analytics and warehousing
|
||||
- Looker for dashboards
|
||||
- Cloud Functions for lightweight transforms
|
||||
```
|
||||
|
||||
### ML Platform
|
||||
|
||||
```
|
||||
Ask: "Design a machine learning platform for model training and serving"
|
||||
|
||||
Result:
|
||||
- Vertex AI for training and prediction
|
||||
- Cloud Storage for datasets and model artifacts
|
||||
- BigQuery for feature store
|
||||
- Cloud Functions for preprocessing triggers
|
||||
- Cloud Monitoring for model drift detection
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Input Requirements
|
||||
|
||||
Provide these details for architecture design:
|
||||
|
||||
| Requirement | Description | Example |
|
||||
|-------------|-------------|---------|
|
||||
| Application type | What you're building | SaaS platform, mobile backend |
|
||||
| Expected scale | Users, requests/sec | 10k users, 100 RPS |
|
||||
| Budget | Monthly GCP limit | $500/month max |
|
||||
| Team context | Size, GCP experience | 3 devs, intermediate |
|
||||
| Compliance | Regulatory needs | HIPAA, GDPR, SOC 2 |
|
||||
| Availability | Uptime requirements | 99.9% SLA, 1hr RPO |
|
||||
|
||||
**JSON Format:**
|
||||
|
||||
```json
|
||||
{
|
||||
"application_type": "saas_platform",
|
||||
"expected_users": 10000,
|
||||
"requests_per_second": 100,
|
||||
"budget_monthly_usd": 500,
|
||||
"team_size": 3,
|
||||
"gcp_experience": "intermediate",
|
||||
"compliance": ["SOC2"],
|
||||
"availability_sla": "99.9%"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output Formats
|
||||
|
||||
### Architecture Design
|
||||
|
||||
- Pattern recommendation with rationale
|
||||
- Service stack diagram (ASCII)
|
||||
- Monthly cost estimate and trade-offs
|
||||
|
||||
### IaC Templates
|
||||
|
||||
- **Terraform HCL**: Production-ready Google provider configs
|
||||
- **gcloud CLI**: Scripted deployment commands
|
||||
- **Cloud Build YAML**: CI/CD pipeline definitions
|
||||
|
||||
### Cost Analysis
|
||||
|
||||
- Current spend breakdown with optimization recommendations
|
||||
- Priority action list (high/medium/low) and implementation checklist
|
||||
|
||||
---
|
||||
|
||||
## Reference Documentation
|
||||
|
||||
| Document | Contents |
|
||||
|----------|----------|
|
||||
| `references/architecture_patterns.md` | 6 patterns: serverless, GKE microservices, three-tier, data pipeline, ML platform, multi-region |
|
||||
| `references/service_selection.md` | Decision matrices for compute, database, storage, messaging |
|
||||
| `references/best_practices.md` | Naming, labels, IAM, networking, monitoring, disaster recovery |
|
||||
@@ -1,13 +1,13 @@
|
||||
---
|
||||
title: "Engineering - Core Skills — Agent Skills & Codex Plugins"
|
||||
description: "43 engineering - core skills — engineering agent skill and Claude Code plugin for code generation, DevOps, architecture, and testing. Works with Claude Code, Codex CLI, Gemini CLI, and OpenClaw."
|
||||
description: "44 engineering - core skills — engineering agent skill and Claude Code plugin for code generation, DevOps, architecture, and testing. Works with Claude Code, Codex CLI, Gemini CLI, and OpenClaw."
|
||||
---
|
||||
|
||||
<div class="domain-header" markdown>
|
||||
|
||||
# :material-code-braces: Engineering - Core
|
||||
|
||||
<p class="domain-count">43 skills in this domain</p>
|
||||
<p class="domain-count">44 skills in this domain</p>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -59,6 +59,12 @@ description: "43 engineering - core skills — engineering agent skill and Claud
|
||||
|
||||
You are now a world-class epic design expert. You build cinematic, immersive websites that feel premium and alive — u...
|
||||
|
||||
- **[GCP Cloud Architect](gcp-cloud-architect.md)**
|
||||
|
||||
---
|
||||
|
||||
Design scalable, cost-effective Google Cloud architectures for startups and enterprises with infrastructure-as-code t...
|
||||
|
||||
- **[Google Workspace CLI](google-workspace-cli.md)**
|
||||
|
||||
---
|
||||
|
||||
@@ -191,6 +191,12 @@ description: "46 engineering - powerful skills — advanced agent-native skill a
|
||||
|
||||
Tier: POWERFUL
|
||||
|
||||
- **[Secrets Vault Manager](secrets-vault-manager.md)**
|
||||
|
||||
---
|
||||
|
||||
Tier: POWERFUL
|
||||
|
||||
- **[Skill Security Auditor](skill-security-auditor.md)**
|
||||
|
||||
---
|
||||
@@ -209,6 +215,12 @@ description: "46 engineering - powerful skills — advanced agent-native skill a
|
||||
|
||||
Spec-driven workflow enforces a single, non-negotiable rule: write the specification BEFORE you write any code. Not a...
|
||||
|
||||
- **[SQL Database Assistant - POWERFUL Tier Skill](sql-database-assistant.md)**
|
||||
|
||||
---
|
||||
|
||||
The operational companion to database design. While database-designer focuses on schema architecture and database-sch...
|
||||
|
||||
- **[Tech Debt Tracker](tech-debt-tracker.md)**
|
||||
|
||||
---
|
||||
|
||||
414
docs/skills/engineering/secrets-vault-manager.md
Normal file
414
docs/skills/engineering/secrets-vault-manager.md
Normal file
@@ -0,0 +1,414 @@
|
||||
---
|
||||
title: "Secrets Vault Manager — Agent Skill for Codex & OpenClaw"
|
||||
description: "Use when the user asks to set up secret management infrastructure, integrate HashiCorp Vault, configure cloud secret stores (AWS Secrets Manager. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
---
|
||||
|
||||
# Secrets Vault Manager
|
||||
|
||||
<div class="page-meta" markdown>
|
||||
<span class="meta-badge">:material-rocket-launch: Engineering - POWERFUL</span>
|
||||
<span class="meta-badge">:material-identifier: `secrets-vault-manager`</span>
|
||||
<span class="meta-badge">:material-github: <a href="https://github.com/alirezarezvani/claude-skills/tree/main/engineering/secrets-vault-manager/SKILL.md">Source</a></span>
|
||||
</div>
|
||||
|
||||
<div class="install-banner" markdown>
|
||||
<span class="install-label">Install:</span> <code>claude /plugin install engineering-advanced-skills</code>
|
||||
</div>
|
||||
|
||||
|
||||
**Tier:** POWERFUL
|
||||
**Category:** Engineering
|
||||
**Domain:** Security / Infrastructure / DevOps
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Production secret infrastructure management for teams running HashiCorp Vault, cloud-native secret stores, or hybrid architectures. This skill covers policy authoring, auth method configuration, automated rotation, dynamic secrets, audit logging, and incident response.
|
||||
|
||||
**Distinct from env-secrets-manager** which handles local `.env` file hygiene and leak detection. This skill operates at the infrastructure layer — Vault clusters, cloud KMS, certificate authorities, and CI/CD secret injection.
|
||||
|
||||
### When to Use
|
||||
|
||||
- Standing up a new Vault cluster or migrating to a managed secret store
|
||||
- Designing auth methods for services, CI runners, and human operators
|
||||
- Implementing automated credential rotation (database, API keys, certificates)
|
||||
- Auditing secret access patterns for compliance (SOC 2, ISO 27001, HIPAA)
|
||||
- Responding to a secret leak that requires mass revocation
|
||||
- Integrating secrets into Kubernetes workloads or CI/CD pipelines
|
||||
|
||||
---
|
||||
|
||||
## HashiCorp Vault Patterns
|
||||
|
||||
### Architecture Decisions
|
||||
|
||||
| Decision | Recommendation | Rationale |
|
||||
|----------|---------------|-----------|
|
||||
| Deployment mode | HA with Raft storage | No external dependency, built-in leader election |
|
||||
| Auto-unseal | Cloud KMS (AWS KMS / Azure Key Vault / GCP KMS) | Eliminates manual unseal, enables automated restarts |
|
||||
| Namespaces | One per environment (dev/staging/prod) | Blast-radius isolation, independent policies |
|
||||
| Audit devices | File + syslog (dual) | Vault refuses requests if all audit devices fail — dual prevents outages |
|
||||
|
||||
### Auth Methods
|
||||
|
||||
**AppRole** — Machine-to-machine authentication for services and batch jobs.
|
||||
|
||||
```hcl
|
||||
# Enable AppRole
|
||||
path "auth/approle/*" {
|
||||
capabilities = ["create", "read", "update", "delete", "list"]
|
||||
}
|
||||
|
||||
# Application-specific role
|
||||
vault write auth/approle/role/payment-service \
|
||||
token_ttl=1h \
|
||||
token_max_ttl=4h \
|
||||
secret_id_num_uses=1 \
|
||||
secret_id_ttl=10m \
|
||||
token_policies="payment-service-read"
|
||||
```
|
||||
|
||||
**Kubernetes** — Pod-native authentication via service account tokens.
|
||||
|
||||
```hcl
|
||||
vault write auth/kubernetes/role/api-server \
|
||||
bound_service_account_names=api-server \
|
||||
bound_service_account_namespaces=production \
|
||||
policies=api-server-secrets \
|
||||
ttl=1h
|
||||
```
|
||||
|
||||
**OIDC** — Human operator access via SSO provider (Okta, Azure AD, Google Workspace).
|
||||
|
||||
```hcl
|
||||
vault write auth/oidc/role/engineering \
|
||||
bound_audiences="vault" \
|
||||
allowed_redirect_uris="https://vault.example.com/ui/vault/auth/oidc/oidc/callback" \
|
||||
user_claim="email" \
|
||||
oidc_scopes="openid,profile,email" \
|
||||
policies="engineering-read" \
|
||||
ttl=8h
|
||||
```
|
||||
|
||||
### Secret Engines
|
||||
|
||||
| Engine | Use Case | TTL Strategy |
|
||||
|--------|----------|-------------|
|
||||
| KV v2 | Static secrets (API keys, config) | Versioned, manual rotation |
|
||||
| Database | Dynamic DB credentials | 1h default, 24h max |
|
||||
| PKI | TLS certificates | 90d leaf certs, 5y intermediate CA |
|
||||
| Transit | Encryption-as-a-service | Key rotation every 90d |
|
||||
| SSH | Signed SSH certificates | 30m for interactive, 8h for automation |
|
||||
|
||||
### Policy Design
|
||||
|
||||
Follow least-privilege with path-based granularity:
|
||||
|
||||
```hcl
|
||||
# payment-service-read policy
|
||||
path "secret/data/production/payment/*" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
|
||||
path "database/creds/payment-readonly" {
|
||||
capabilities = ["read"]
|
||||
}
|
||||
|
||||
# Deny access to admin paths explicitly
|
||||
path "sys/*" {
|
||||
capabilities = ["deny"]
|
||||
}
|
||||
```
|
||||
|
||||
**Policy naming convention:** `{service}-{access-level}` (e.g., `payment-service-read`, `api-gateway-admin`).
|
||||
|
||||
---
|
||||
|
||||
## Cloud Secret Store Integration
|
||||
|
||||
### Comparison Matrix
|
||||
|
||||
| Feature | AWS Secrets Manager | Azure Key Vault | GCP Secret Manager |
|
||||
|---------|--------------------|-----------------|--------------------|
|
||||
| Rotation | Built-in Lambda | Custom logic via Functions | Cloud Functions |
|
||||
| Versioning | Automatic | Manual or automatic | Automatic |
|
||||
| Encryption | AWS KMS (default or CMK) | HSM-backed | Google-managed or CMEK |
|
||||
| Access control | IAM policies + resource policy | RBAC + Access Policies | IAM bindings |
|
||||
| Cross-region | Replication supported | Geo-redundant by default | Replication supported |
|
||||
| Audit | CloudTrail | Azure Monitor + Diagnostic Logs | Cloud Audit Logs |
|
||||
| Pricing model | Per-secret + per-API call | Per-operation + per-key | Per-secret version + per-access |
|
||||
|
||||
### When to Use Which
|
||||
|
||||
- **AWS Secrets Manager**: RDS/Aurora credential rotation out of the box. Best when fully on AWS.
|
||||
- **Azure Key Vault**: Certificate management strength. Required for Azure AD integrated workloads.
|
||||
- **GCP Secret Manager**: Simplest API surface. Best for GKE-native workloads with Workload Identity.
|
||||
- **HashiCorp Vault**: Multi-cloud, dynamic secrets, PKI, transit encryption. Best for complex or hybrid environments.
|
||||
|
||||
### SDK Access Patterns
|
||||
|
||||
**Principle:** Always fetch secrets at startup or via sidecar — never bake into images or config files.
|
||||
|
||||
```python
|
||||
# AWS Secrets Manager pattern
|
||||
import boto3, json
|
||||
|
||||
def get_secret(secret_name, region="us-east-1"):
|
||||
client = boto3.client("secretsmanager", region_name=region)
|
||||
response = client.get_secret_value(SecretId=secret_name)
|
||||
return json.loads(response["SecretString"])
|
||||
```
|
||||
|
||||
```python
|
||||
# GCP Secret Manager pattern
|
||||
from google.cloud import secretmanager
|
||||
|
||||
def get_secret(project_id, secret_id, version="latest"):
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
name = f"projects/{project_id}/secrets/{secret_id}/versions/{version}"
|
||||
response = client.access_secret_version(request={"name": name})
|
||||
return response.payload.data.decode("UTF-8")
|
||||
```
|
||||
|
||||
```python
|
||||
# Azure Key Vault pattern
|
||||
from azure.identity import DefaultAzureCredential
|
||||
from azure.keyvault.secrets import SecretClient
|
||||
|
||||
def get_secret(vault_url, secret_name):
|
||||
credential = DefaultAzureCredential()
|
||||
client = SecretClient(vault_url=vault_url, credential=credential)
|
||||
return client.get_secret(secret_name).value
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Secret Rotation Workflows
|
||||
|
||||
### Rotation Strategy by Secret Type
|
||||
|
||||
| Secret Type | Rotation Frequency | Method | Downtime Risk |
|
||||
|-------------|-------------------|--------|---------------|
|
||||
| Database passwords | 30 days | Dual-account swap | Zero (A/B rotation) |
|
||||
| API keys | 90 days | Generate new, deprecate old | Zero (overlap window) |
|
||||
| TLS certificates | 60 days before expiry | ACME or Vault PKI | Zero (graceful reload) |
|
||||
| SSH keys | 90 days | Vault-signed certificates | Zero (CA-based) |
|
||||
| Service tokens | 24 hours | Dynamic generation | Zero (short-lived) |
|
||||
| Encryption keys | 90 days | Key versioning (rewrap) | Zero (version coexistence) |
|
||||
|
||||
### Database Credential Rotation (Dual-Account)
|
||||
|
||||
1. Two database accounts exist: `app_user_a` and `app_user_b`
|
||||
2. Application currently uses `app_user_a`
|
||||
3. Rotation rotates `app_user_b` password, updates secret store
|
||||
4. Application switches to `app_user_b` on next credential fetch
|
||||
5. After grace period, `app_user_a` password is rotated
|
||||
6. Cycle repeats
|
||||
|
||||
### API Key Rotation (Overlap Window)
|
||||
|
||||
1. Generate new API key with provider
|
||||
2. Store new key in secret store as `current`, move old to `previous`
|
||||
3. Deploy applications — they read `current`
|
||||
4. After all instances restarted (or TTL expired), revoke `previous`
|
||||
5. Monitoring confirms zero usage of old key before revocation
|
||||
|
||||
---
|
||||
|
||||
## Dynamic Secrets
|
||||
|
||||
Dynamic secrets are generated on-demand with automatic expiration. Prefer dynamic secrets over static credentials wherever possible.
|
||||
|
||||
### Database Dynamic Credentials (Vault)
|
||||
|
||||
```hcl
|
||||
# Configure database engine
|
||||
vault write database/config/postgres \
|
||||
plugin_name=postgresql-database-plugin \
|
||||
connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/app" \
|
||||
allowed_roles="app-readonly,app-readwrite" \
|
||||
username="vault_admin" \
|
||||
password="<admin-password>"
|
||||
|
||||
# Create role with TTL
|
||||
vault write database/roles/app-readonly \
|
||||
db_name=postgres \
|
||||
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
|
||||
default_ttl=1h \
|
||||
max_ttl=24h
|
||||
```
|
||||
|
||||
### Cloud IAM Dynamic Credentials
|
||||
|
||||
Vault can generate short-lived AWS IAM credentials, Azure service principal passwords, or GCP service account keys — eliminating long-lived cloud credentials entirely.
|
||||
|
||||
### SSH Certificate Authority
|
||||
|
||||
Replace SSH key distribution with a Vault-signed certificate model:
|
||||
|
||||
1. Vault acts as SSH CA
|
||||
2. Users/machines request signed certificates with short TTL (30 min)
|
||||
3. SSH servers trust the CA public key — no `authorized_keys` management
|
||||
4. Certificates expire automatically — no revocation needed for normal operations
|
||||
|
||||
---
|
||||
|
||||
## Audit Logging
|
||||
|
||||
### What to Log
|
||||
|
||||
| Event | Priority | Retention |
|
||||
|-------|----------|-----------|
|
||||
| Secret read access | HIGH | 1 year minimum |
|
||||
| Secret creation/update | HIGH | 1 year minimum |
|
||||
| Auth method login | MEDIUM | 90 days |
|
||||
| Policy changes | CRITICAL | 2 years (compliance) |
|
||||
| Failed access attempts | CRITICAL | 1 year |
|
||||
| Token creation/revocation | MEDIUM | 90 days |
|
||||
| Seal/unseal operations | CRITICAL | Indefinite |
|
||||
|
||||
### Anomaly Detection Signals
|
||||
|
||||
- Secret accessed from new IP/CIDR range
|
||||
- Access volume spike (>3x baseline for a path)
|
||||
- Off-hours access for human auth methods
|
||||
- Service accessing secrets outside its policy scope (denied requests)
|
||||
- Multiple failed auth attempts from single source
|
||||
- Token created with unusually long TTL
|
||||
|
||||
### Compliance Reporting
|
||||
|
||||
Generate periodic reports covering:
|
||||
|
||||
1. **Access inventory** — Which identities accessed which secrets, when
|
||||
2. **Rotation compliance** — Secrets overdue for rotation
|
||||
3. **Policy drift** — Policies modified since last review
|
||||
4. **Orphaned secrets** — Secrets with no recent access (>90 days)
|
||||
|
||||
Use `audit_log_analyzer.py` to parse Vault or cloud audit logs for these signals.
|
||||
|
||||
---
|
||||
|
||||
## Emergency Procedures
|
||||
|
||||
### Secret Leak Response (Immediate)
|
||||
|
||||
**Time target: Contain within 15 minutes of detection.**
|
||||
|
||||
1. **Identify scope** — Which secret(s) leaked, where (repo, log, error message, third party)
|
||||
2. **Revoke immediately** — Rotate the compromised credential at the source (provider API, Vault, cloud SM)
|
||||
3. **Invalidate tokens** — Revoke all Vault tokens that accessed the leaked secret
|
||||
4. **Audit blast radius** — Query audit logs for usage of the compromised secret in the exposure window
|
||||
5. **Notify stakeholders** — Security team, affected service owners, compliance (if PII/regulated data)
|
||||
6. **Post-mortem** — Document root cause, update controls to prevent recurrence
|
||||
|
||||
### Vault Seal Operations
|
||||
|
||||
**When to seal:** Active security incident affecting Vault infrastructure, suspected key compromise.
|
||||
|
||||
**Sealing** stops all Vault operations. Use only as last resort.
|
||||
|
||||
**Unseal procedure:**
|
||||
1. Gather quorum of unseal key holders (Shamir threshold)
|
||||
2. Or confirm auto-unseal KMS key is accessible
|
||||
3. Unseal via `vault operator unseal` or restart with auto-unseal
|
||||
4. Verify audit devices reconnected
|
||||
5. Check active leases and token validity
|
||||
|
||||
See `references/emergency_procedures.md` for complete playbooks.
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### Vault Agent Sidecar (Kubernetes)
|
||||
|
||||
Vault Agent runs alongside application pods, handles authentication and secret rendering:
|
||||
|
||||
```yaml
|
||||
# Pod annotation for Vault Agent Injector
|
||||
annotations:
|
||||
vault.hashicorp.com/agent-inject: "true"
|
||||
vault.hashicorp.com/role: "api-server"
|
||||
vault.hashicorp.com/agent-inject-secret-db: "database/creds/app-readonly"
|
||||
vault.hashicorp.com/agent-inject-template-db: |
|
||||
{{- with secret "database/creds/app-readonly" -}}
|
||||
postgresql://{{ .Data.username }}:{{ .Data.password }}@db:5432/app
|
||||
{{- end }}
|
||||
```
|
||||
|
||||
### External Secrets Operator (Kubernetes)
|
||||
|
||||
For teams preferring declarative GitOps over agent sidecars:
|
||||
|
||||
```yaml
|
||||
apiVersion: external-secrets.io/v1beta1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: api-credentials
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
name: vault-backend
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: api-credentials
|
||||
data:
|
||||
- secretKey: api-key
|
||||
remoteRef:
|
||||
key: secret/data/production/api
|
||||
property: key
|
||||
```
|
||||
|
||||
### GitHub Actions OIDC
|
||||
|
||||
Eliminate long-lived secrets in CI by using OIDC federation:
|
||||
|
||||
```yaml
|
||||
- name: Authenticate to Vault
|
||||
uses: hashicorp/vault-action@v2
|
||||
with:
|
||||
url: https://vault.example.com
|
||||
method: jwt
|
||||
role: github-ci
|
||||
jwtGithubAudience: https://vault.example.com
|
||||
secrets: |
|
||||
secret/data/ci/deploy api_key | DEPLOY_API_KEY ;
|
||||
secret/data/ci/deploy db_password | DB_PASSWORD
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
| Anti-Pattern | Risk | Correct Approach |
|
||||
|-------------|------|-----------------|
|
||||
| Hardcoded secrets in source code | Leak via repo, logs, error output | Fetch from secret store at runtime |
|
||||
| Long-lived static tokens (>30 days) | Stale credentials, no accountability | Dynamic secrets or short TTL + rotation |
|
||||
| Shared service accounts | No audit trail per consumer | Per-service identity with unique credentials |
|
||||
| No rotation policy | Compromised creds persist indefinitely | Automated rotation on schedule |
|
||||
| Secrets in environment variables on CI | Visible in build logs, process table | Vault Agent or OIDC-based injection |
|
||||
| Single unseal key holder | Bus factor of 1, recovery blocked | Shamir split (3-of-5) or auto-unseal |
|
||||
| No audit device configured | Zero visibility into access | Dual audit devices (file + syslog) |
|
||||
| Wildcard policies (`path "*"`) | Over-permissioned, violates least privilege | Explicit path-based policies per service |
|
||||
|
||||
---
|
||||
|
||||
## Tools
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `vault_config_generator.py` | Generate Vault policy and auth config from application requirements |
|
||||
| `rotation_planner.py` | Create rotation schedule from a secret inventory file |
|
||||
| `audit_log_analyzer.py` | Analyze audit logs for anomalies and compliance gaps |
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
- **env-secrets-manager** — Local `.env` file hygiene, leak detection, drift awareness
|
||||
- **senior-secops** — Security operations, incident response, threat modeling
|
||||
- **ci-cd-pipeline-builder** — Pipeline design where secrets are consumed
|
||||
- **docker-development** — Container secret injection patterns
|
||||
- **helm-chart-builder** — Kubernetes secret management in Helm charts
|
||||
468
docs/skills/engineering/sql-database-assistant.md
Normal file
468
docs/skills/engineering/sql-database-assistant.md
Normal file
@@ -0,0 +1,468 @@
|
||||
---
|
||||
title: "SQL Database Assistant - POWERFUL Tier Skill — Agent Skill for Codex & OpenClaw"
|
||||
description: "Use when the user asks to write SQL queries, optimize database performance, generate migrations, explore database schemas, or work with ORMs like. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
---
|
||||
|
||||
# SQL Database Assistant - POWERFUL Tier Skill
|
||||
|
||||
<div class="page-meta" markdown>
|
||||
<span class="meta-badge">:material-rocket-launch: Engineering - POWERFUL</span>
|
||||
<span class="meta-badge">:material-identifier: `sql-database-assistant`</span>
|
||||
<span class="meta-badge">:material-github: <a href="https://github.com/alirezarezvani/claude-skills/tree/main/engineering/sql-database-assistant/SKILL.md">Source</a></span>
|
||||
</div>
|
||||
|
||||
<div class="install-banner" markdown>
|
||||
<span class="install-label">Install:</span> <code>claude /plugin install engineering-advanced-skills</code>
|
||||
</div>
|
||||
|
||||
|
||||
## Overview
|
||||
|
||||
The operational companion to database design. While **database-designer** focuses on schema architecture and **database-schema-designer** handles ERD modeling, this skill covers the day-to-day: writing queries, optimizing performance, generating migrations, and bridging the gap between application code and database engines.
|
||||
|
||||
### Core Capabilities
|
||||
|
||||
- **Natural Language to SQL** — translate requirements into correct, performant queries
|
||||
- **Schema Exploration** — introspect live databases across PostgreSQL, MySQL, SQLite, SQL Server
|
||||
- **Query Optimization** — EXPLAIN analysis, index recommendations, N+1 detection, rewrite patterns
|
||||
- **Migration Generation** — up/down scripts, zero-downtime strategies, rollback plans
|
||||
- **ORM Integration** — Prisma, Drizzle, TypeORM, SQLAlchemy patterns and escape hatches
|
||||
- **Multi-Database Support** — dialect-aware SQL with compatibility guidance
|
||||
|
||||
### Tools
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `scripts/query_optimizer.py` | Static analysis of SQL queries for performance issues |
|
||||
| `scripts/migration_generator.py` | Generate migration file templates from change descriptions |
|
||||
| `scripts/schema_explorer.py` | Generate schema documentation from introspection queries |
|
||||
|
||||
---
|
||||
|
||||
## Natural Language to SQL
|
||||
|
||||
### Translation Patterns
|
||||
|
||||
When converting requirements to SQL, follow this sequence:
|
||||
|
||||
1. **Identify entities** — map nouns to tables
|
||||
2. **Identify relationships** — map verbs to JOINs or subqueries
|
||||
3. **Identify filters** — map adjectives/conditions to WHERE clauses
|
||||
4. **Identify aggregations** — map "total", "average", "count" to GROUP BY
|
||||
5. **Identify ordering** — map "top", "latest", "highest" to ORDER BY + LIMIT
|
||||
|
||||
### Common Query Templates
|
||||
|
||||
**Top-N per group (window function)**
|
||||
```sql
|
||||
SELECT * FROM (
|
||||
SELECT *, ROW_NUMBER() OVER (PARTITION BY department_id ORDER BY salary DESC) AS rn
|
||||
FROM employees
|
||||
) ranked WHERE rn <= 3;
|
||||
```
|
||||
|
||||
**Running totals**
|
||||
```sql
|
||||
SELECT date, amount,
|
||||
SUM(amount) OVER (ORDER BY date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total
|
||||
FROM transactions;
|
||||
```
|
||||
|
||||
**Gap detection**
|
||||
```sql
|
||||
SELECT curr.id, curr.seq_num, prev.seq_num AS prev_seq
|
||||
FROM records curr
|
||||
LEFT JOIN records prev ON prev.seq_num = curr.seq_num - 1
|
||||
WHERE prev.id IS NULL AND curr.seq_num > 1;
|
||||
```
|
||||
|
||||
**UPSERT (PostgreSQL)**
|
||||
```sql
|
||||
INSERT INTO settings (key, value, updated_at)
|
||||
VALUES ('theme', 'dark', NOW())
|
||||
ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value, updated_at = EXCLUDED.updated_at;
|
||||
```
|
||||
|
||||
**UPSERT (MySQL)**
|
||||
```sql
|
||||
INSERT INTO settings (key_name, value, updated_at)
|
||||
VALUES ('theme', 'dark', NOW())
|
||||
ON DUPLICATE KEY UPDATE value = VALUES(value), updated_at = VALUES(updated_at);
|
||||
```
|
||||
|
||||
> See references/query_patterns.md for JOINs, CTEs, window functions, JSON operations, and more.
|
||||
|
||||
---
|
||||
|
||||
## Schema Exploration
|
||||
|
||||
### Introspection Queries
|
||||
|
||||
**PostgreSQL — list tables and columns**
|
||||
```sql
|
||||
SELECT table_name, column_name, data_type, is_nullable, column_default
|
||||
FROM information_schema.columns
|
||||
WHERE table_schema = 'public'
|
||||
ORDER BY table_name, ordinal_position;
|
||||
```
|
||||
|
||||
**PostgreSQL — foreign keys**
|
||||
```sql
|
||||
SELECT tc.table_name, kcu.column_name,
|
||||
ccu.table_name AS foreign_table, ccu.column_name AS foreign_column
|
||||
FROM information_schema.table_constraints tc
|
||||
JOIN information_schema.key_column_usage kcu ON tc.constraint_name = kcu.constraint_name
|
||||
JOIN information_schema.constraint_column_usage ccu ON tc.constraint_name = ccu.constraint_name
|
||||
WHERE tc.constraint_type = 'FOREIGN KEY';
|
||||
```
|
||||
|
||||
**MySQL — table sizes**
|
||||
```sql
|
||||
SELECT table_name, table_rows,
|
||||
ROUND(data_length / 1024 / 1024, 2) AS data_mb,
|
||||
ROUND(index_length / 1024 / 1024, 2) AS index_mb
|
||||
FROM information_schema.tables
|
||||
WHERE table_schema = DATABASE()
|
||||
ORDER BY data_length DESC;
|
||||
```
|
||||
|
||||
**SQLite — schema dump**
|
||||
```sql
|
||||
SELECT name, sql FROM sqlite_master WHERE type = 'table' ORDER BY name;
|
||||
```
|
||||
|
||||
**SQL Server — columns with types**
|
||||
```sql
|
||||
SELECT t.name AS table_name, c.name AS column_name,
|
||||
ty.name AS data_type, c.max_length, c.is_nullable
|
||||
FROM sys.columns c
|
||||
JOIN sys.tables t ON c.object_id = t.object_id
|
||||
JOIN sys.types ty ON c.user_type_id = ty.user_type_id
|
||||
ORDER BY t.name, c.column_id;
|
||||
```
|
||||
|
||||
### Generating Documentation from Schema
|
||||
|
||||
Use `scripts/schema_explorer.py` to produce markdown or JSON documentation:
|
||||
|
||||
```bash
|
||||
python scripts/schema_explorer.py --dialect postgres --tables all --format md
|
||||
python scripts/schema_explorer.py --dialect mysql --tables users,orders --format json --json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Query Optimization
|
||||
|
||||
### EXPLAIN Analysis Workflow
|
||||
|
||||
1. **Run EXPLAIN ANALYZE** (PostgreSQL) or **EXPLAIN FORMAT=JSON** (MySQL)
|
||||
2. **Identify the costliest node** — Seq Scan on large tables, Nested Loop with high row estimates
|
||||
3. **Check for missing indexes** — sequential scans on filtered columns
|
||||
4. **Look for estimation errors** — planned vs actual rows divergence signals stale statistics
|
||||
5. **Evaluate JOIN order** — ensure the smallest result set drives the join
|
||||
|
||||
### Index Recommendation Checklist
|
||||
|
||||
- Columns in WHERE clauses with high selectivity
|
||||
- Columns in JOIN conditions (foreign keys)
|
||||
- Columns in ORDER BY when combined with LIMIT
|
||||
- Composite indexes matching multi-column WHERE predicates (most selective column first)
|
||||
- Partial indexes for queries with constant filters (e.g., `WHERE status = 'active'`)
|
||||
- Covering indexes to avoid table lookups for read-heavy queries
|
||||
|
||||
### Query Rewriting Patterns
|
||||
|
||||
| Anti-Pattern | Rewrite |
|
||||
|-------------|---------|
|
||||
| `SELECT * FROM orders` | `SELECT id, status, total FROM orders` (explicit columns) |
|
||||
| `WHERE YEAR(created_at) = 2025` | `WHERE created_at >= '2025-01-01' AND created_at < '2026-01-01'` (sargable) |
|
||||
| Correlated subquery in SELECT | LEFT JOIN with aggregation |
|
||||
| `NOT IN (SELECT ...)` with NULLs | `NOT EXISTS (SELECT 1 ...)` |
|
||||
| `UNION` (dedup) when not needed | `UNION ALL` |
|
||||
| `LIKE '%search%'` | Full-text search index (GIN/FULLTEXT) |
|
||||
| `ORDER BY RAND()` | Application-side random sampling or `TABLESAMPLE` |
|
||||
|
||||
### N+1 Detection
|
||||
|
||||
**Symptoms:**
|
||||
- Application loop that executes one query per parent row
|
||||
- ORM lazy-loading related entities inside a loop
|
||||
- Query log shows hundreds of identical SELECT patterns with different IDs
|
||||
|
||||
**Fixes:**
|
||||
- Use eager loading (`include` in Prisma, `joinedload` in SQLAlchemy)
|
||||
- Batch queries with `WHERE id IN (...)`
|
||||
- Use DataLoader pattern for GraphQL resolvers
|
||||
|
||||
### Static Analysis Tool
|
||||
|
||||
```bash
|
||||
python scripts/query_optimizer.py --query "SELECT * FROM orders WHERE status = 'pending'" --dialect postgres
|
||||
python scripts/query_optimizer.py --query queries.sql --dialect mysql --json
|
||||
```
|
||||
|
||||
> See references/optimization_guide.md for EXPLAIN plan reading, index types, and connection pooling.
|
||||
|
||||
---
|
||||
|
||||
## Migration Generation
|
||||
|
||||
### Zero-Downtime Migration Patterns
|
||||
|
||||
**Adding a column (safe)**
|
||||
```sql
|
||||
-- Up
|
||||
ALTER TABLE users ADD COLUMN phone VARCHAR(20);
|
||||
|
||||
-- Down
|
||||
ALTER TABLE users DROP COLUMN phone;
|
||||
```
|
||||
|
||||
**Renaming a column (expand-contract)**
|
||||
```sql
|
||||
-- Step 1: Add new column
|
||||
ALTER TABLE users ADD COLUMN full_name VARCHAR(255);
|
||||
-- Step 2: Backfill
|
||||
UPDATE users SET full_name = name;
|
||||
-- Step 3: Deploy app reading both columns
|
||||
-- Step 4: Deploy app writing only new column
|
||||
-- Step 5: Drop old column
|
||||
ALTER TABLE users DROP COLUMN name;
|
||||
```
|
||||
|
||||
**Adding a NOT NULL column (safe sequence)**
|
||||
```sql
|
||||
-- Step 1: Add nullable
|
||||
ALTER TABLE orders ADD COLUMN region VARCHAR(50);
|
||||
-- Step 2: Backfill with default
|
||||
UPDATE orders SET region = 'unknown' WHERE region IS NULL;
|
||||
-- Step 3: Add constraint
|
||||
ALTER TABLE orders ALTER COLUMN region SET NOT NULL;
|
||||
ALTER TABLE orders ALTER COLUMN region SET DEFAULT 'unknown';
|
||||
```
|
||||
|
||||
**Index creation (non-blocking, PostgreSQL)**
|
||||
```sql
|
||||
CREATE INDEX CONCURRENTLY idx_orders_status ON orders (status);
|
||||
```
|
||||
|
||||
### Data Backfill Strategies
|
||||
|
||||
- **Batch updates** — process in chunks of 1000-10000 rows to avoid lock contention
|
||||
- **Background jobs** — run backfills asynchronously with progress tracking
|
||||
- **Dual-write** — write to old and new columns during transition period
|
||||
- **Validation queries** — verify row counts and data integrity after each batch
|
||||
|
||||
### Rollback Strategies
|
||||
|
||||
Every migration must have a reversible down script. For irreversible changes:
|
||||
|
||||
1. **Backup before execution** — `pg_dump` the affected tables
|
||||
2. **Feature flags** — application can switch between old/new schema reads
|
||||
3. **Shadow tables** — keep a copy of the original table during migration window
|
||||
|
||||
### Migration Generator Tool
|
||||
|
||||
```bash
|
||||
python scripts/migration_generator.py --change "add email_verified boolean to users" --dialect postgres --format sql
|
||||
python scripts/migration_generator.py --change "rename column name to full_name in customers" --dialect mysql --format alembic --json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Multi-Database Support
|
||||
|
||||
### Dialect Differences
|
||||
|
||||
| Feature | PostgreSQL | MySQL | SQLite | SQL Server |
|
||||
|---------|-----------|-------|--------|------------|
|
||||
| UPSERT | `ON CONFLICT DO UPDATE` | `ON DUPLICATE KEY UPDATE` | `ON CONFLICT DO UPDATE` | `MERGE` |
|
||||
| Boolean | Native `BOOLEAN` | `TINYINT(1)` | `INTEGER` | `BIT` |
|
||||
| Auto-increment | `SERIAL` / `GENERATED` | `AUTO_INCREMENT` | `INTEGER PRIMARY KEY` | `IDENTITY` |
|
||||
| JSON | `JSONB` (indexed) | `JSON` | Text (ext) | `NVARCHAR(MAX)` |
|
||||
| Array | Native `ARRAY` | Not supported | Not supported | Not supported |
|
||||
| CTE (recursive) | Full support | 8.0+ | 3.8.3+ | Full support |
|
||||
| Window functions | Full support | 8.0+ | 3.25.0+ | Full support |
|
||||
| Full-text search | `tsvector` + GIN | `FULLTEXT` index | FTS5 extension | Full-text catalog |
|
||||
| LIMIT/OFFSET | `LIMIT n OFFSET m` | `LIMIT n OFFSET m` | `LIMIT n OFFSET m` | `OFFSET m ROWS FETCH NEXT n ROWS ONLY` |
|
||||
|
||||
### Compatibility Tips
|
||||
|
||||
- **Always use parameterized queries** — prevents SQL injection across all dialects
|
||||
- **Avoid dialect-specific functions in shared code** — wrap in adapter layer
|
||||
- **Test migrations on target engine** — `information_schema` varies between engines
|
||||
- **Use ISO date format** — `'YYYY-MM-DD'` works everywhere
|
||||
- **Quote identifiers** — use double quotes (SQL standard) or backticks (MySQL)
|
||||
|
||||
---
|
||||
|
||||
## ORM Patterns
|
||||
|
||||
### Prisma
|
||||
|
||||
**Schema definition**
|
||||
```prisma
|
||||
model User {
|
||||
id Int @id @default(autoincrement())
|
||||
email String @unique
|
||||
name String?
|
||||
posts Post[]
|
||||
createdAt DateTime @default(now())
|
||||
}
|
||||
|
||||
model Post {
|
||||
id Int @id @default(autoincrement())
|
||||
title String
|
||||
author User @relation(fields: [authorId], references: [id])
|
||||
authorId Int
|
||||
}
|
||||
```
|
||||
|
||||
**Migrations**: `npx prisma migrate dev --name add_user_email`
|
||||
**Query API**: `prisma.user.findMany({ where: { email: { contains: '@' } }, include: { posts: true } })`
|
||||
**Raw SQL escape hatch**: `prisma.$queryRaw\`SELECT * FROM users WHERE id = ${userId}\``
|
||||
|
||||
### Drizzle
|
||||
|
||||
**Schema-first definition**
|
||||
```typescript
|
||||
export const users = pgTable('users', {
|
||||
id: serial('id').primaryKey(),
|
||||
email: varchar('email', { length: 255 }).notNull().unique(),
|
||||
name: text('name'),
|
||||
createdAt: timestamp('created_at').defaultNow(),
|
||||
});
|
||||
```
|
||||
|
||||
**Query builder**: `db.select().from(users).where(eq(users.email, email))`
|
||||
**Migrations**: `npx drizzle-kit generate:pg` then `npx drizzle-kit push:pg`
|
||||
|
||||
### TypeORM
|
||||
|
||||
**Entity decorators**
|
||||
```typescript
|
||||
@Entity()
|
||||
export class User {
|
||||
@PrimaryGeneratedColumn()
|
||||
id: number;
|
||||
|
||||
@Column({ unique: true })
|
||||
email: string;
|
||||
|
||||
@OneToMany(() => Post, post => post.author)
|
||||
posts: Post[];
|
||||
}
|
||||
```
|
||||
|
||||
**Repository pattern**: `userRepo.find({ where: { email }, relations: ['posts'] })`
|
||||
**Migrations**: `npx typeorm migration:generate -n AddUserEmail`
|
||||
|
||||
### SQLAlchemy
|
||||
|
||||
**Declarative models**
|
||||
```python
|
||||
class User(Base):
|
||||
__tablename__ = 'users'
|
||||
id = Column(Integer, primary_key=True)
|
||||
email = Column(String(255), unique=True, nullable=False)
|
||||
name = Column(String(255))
|
||||
posts = relationship('Post', back_populates='author')
|
||||
```
|
||||
|
||||
**Session management**: Always use `with Session() as session:` context manager
|
||||
**Alembic migrations**: `alembic revision --autogenerate -m "add user email"`
|
||||
|
||||
> See references/orm_patterns.md for side-by-side comparisons and migration workflows per ORM.
|
||||
|
||||
---
|
||||
|
||||
## Data Integrity
|
||||
|
||||
### Constraint Strategy
|
||||
|
||||
- **Primary keys** — every table must have one; prefer surrogate keys (serial/UUID)
|
||||
- **Foreign keys** — enforce referential integrity; define ON DELETE behavior explicitly
|
||||
- **UNIQUE constraints** — for business-level uniqueness (email, slug, API key)
|
||||
- **CHECK constraints** — validate ranges, enums, and business rules at the DB level
|
||||
- **NOT NULL** — default to NOT NULL; make nullable only when genuinely optional
|
||||
|
||||
### Transaction Isolation Levels
|
||||
|
||||
| Level | Dirty Read | Non-Repeatable Read | Phantom Read | Use Case |
|
||||
|-------|-----------|-------------------|-------------|----------|
|
||||
| READ UNCOMMITTED | Yes | Yes | Yes | Never recommended |
|
||||
| READ COMMITTED | No | Yes | Yes | Default for PostgreSQL, general OLTP |
|
||||
| REPEATABLE READ | No | No | Yes (InnoDB: No) | Financial calculations |
|
||||
| SERIALIZABLE | No | No | No | Critical consistency (billing, inventory) |
|
||||
|
||||
### Deadlock Prevention
|
||||
|
||||
1. **Consistent lock ordering** — always acquire locks in the same table/row order
|
||||
2. **Short transactions** — minimize time between first lock and commit
|
||||
3. **Advisory locks** — use `pg_advisory_lock()` for application-level coordination
|
||||
4. **Retry logic** — catch deadlock errors and retry with exponential backoff
|
||||
|
||||
---
|
||||
|
||||
## Backup & Restore
|
||||
|
||||
### PostgreSQL
|
||||
```bash
|
||||
# Full backup
|
||||
pg_dump -Fc --no-owner dbname > backup.dump
|
||||
# Restore
|
||||
pg_restore -d dbname --clean --no-owner backup.dump
|
||||
# Point-in-time recovery: configure WAL archiving + restore_command
|
||||
```
|
||||
|
||||
### MySQL
|
||||
```bash
|
||||
# Full backup
|
||||
mysqldump --single-transaction --routines --triggers dbname > backup.sql
|
||||
# Restore
|
||||
mysql dbname < backup.sql
|
||||
# Binary log for PITR: mysqlbinlog --start-datetime="2025-01-01 00:00:00" binlog.000001
|
||||
```
|
||||
|
||||
### SQLite
|
||||
```bash
|
||||
# Backup (safe with concurrent reads)
|
||||
sqlite3 dbname ".backup backup.db"
|
||||
```
|
||||
|
||||
### Backup Best Practices
|
||||
- **Automate** — cron or systemd timer, never manual-only
|
||||
- **Test restores** — untested backups are not backups
|
||||
- **Offsite copies** — S3, GCS, or separate region
|
||||
- **Retention policy** — daily for 7 days, weekly for 4 weeks, monthly for 12 months
|
||||
- **Monitor backup size and duration** — sudden changes signal issues
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
| Anti-Pattern | Problem | Fix |
|
||||
|-------------|---------|-----|
|
||||
| `SELECT *` | Transfers unnecessary data, breaks on schema changes | Explicit column list |
|
||||
| Missing indexes on FK columns | Slow JOINs and cascading deletes | Add indexes on all foreign keys |
|
||||
| N+1 queries | 1 + N round trips to database | Eager loading or batch queries |
|
||||
| Implicit type coercion | `WHERE id = '123'` prevents index use | Match types in predicates |
|
||||
| No connection pooling | Exhausts connections under load | PgBouncer, ProxySQL, or ORM pool |
|
||||
| Unbounded queries | No LIMIT risks returning millions of rows | Always paginate |
|
||||
| Storing money as FLOAT | Rounding errors | Use `DECIMAL(19,4)` or integer cents |
|
||||
| God tables | One table with 50+ columns | Normalize or use vertical partitioning |
|
||||
| Soft deletes everywhere | Complicates every query with `WHERE deleted_at IS NULL` | Archive tables or event sourcing |
|
||||
| Raw string concatenation | SQL injection | Parameterized queries always |
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
| Skill | Relationship |
|
||||
|-------|-------------|
|
||||
| **database-designer** | Schema architecture, normalization analysis, ERD generation |
|
||||
| **database-schema-designer** | Visual ERD modeling, relationship mapping |
|
||||
| **migration-architect** | Complex multi-step migration orchestration |
|
||||
| **api-design-reviewer** | Ensuring API endpoints align with query patterns |
|
||||
| **observability-platform** | Query performance monitoring, slow query alerts |
|
||||
@@ -1,13 +1,13 @@
|
||||
---
|
||||
title: "Regulatory & Quality Skills — Agent Skills & Codex Plugins"
|
||||
description: "13 regulatory & quality skills — regulatory and quality management agent skill for ISO 13485, MDR, FDA, and GDPR compliance. Works with Claude Code, Codex CLI, Gemini CLI, and OpenClaw."
|
||||
description: "14 regulatory & quality skills — regulatory and quality management agent skill for ISO 13485, MDR, FDA, and GDPR compliance. Works with Claude Code, Codex CLI, Gemini CLI, and OpenClaw."
|
||||
---
|
||||
|
||||
<div class="domain-header" markdown>
|
||||
|
||||
# :material-shield-check-outline: Regulatory & Quality
|
||||
|
||||
<p class="domain-count">13 skills in this domain</p>
|
||||
<p class="domain-count">14 skills in this domain</p>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -95,4 +95,10 @@ description: "13 regulatory & quality skills — regulatory and quality manageme
|
||||
|
||||
ISO 14971:2019 risk management implementation throughout the medical device lifecycle.
|
||||
|
||||
- **[SOC 2 Compliance](soc2-compliance.md)**
|
||||
|
||||
---
|
||||
|
||||
SOC 2 Type I and Type II compliance preparation for SaaS companies. Covers Trust Service Criteria mapping, control ma...
|
||||
|
||||
</div>
|
||||
|
||||
428
docs/skills/ra-qm-team/soc2-compliance.md
Normal file
428
docs/skills/ra-qm-team/soc2-compliance.md
Normal file
@@ -0,0 +1,428 @@
|
||||
---
|
||||
title: "SOC 2 Compliance — Agent Skill for Compliance"
|
||||
description: "Use when the user asks to prepare for SOC 2 audits, map Trust Service Criteria, build control matrices, collect audit evidence, perform gap analysis. Agent skill for Claude Code, Codex CLI, Gemini CLI, OpenClaw."
|
||||
---
|
||||
|
||||
# SOC 2 Compliance
|
||||
|
||||
<div class="page-meta" markdown>
|
||||
<span class="meta-badge">:material-shield-check-outline: Regulatory & Quality</span>
|
||||
<span class="meta-badge">:material-identifier: `soc2-compliance`</span>
|
||||
<span class="meta-badge">:material-github: <a href="https://github.com/alirezarezvani/claude-skills/tree/main/ra-qm-team/soc2-compliance/SKILL.md">Source</a></span>
|
||||
</div>
|
||||
|
||||
<div class="install-banner" markdown>
|
||||
<span class="install-label">Install:</span> <code>claude /plugin install ra-qm-skills</code>
|
||||
</div>
|
||||
|
||||
|
||||
SOC 2 Type I and Type II compliance preparation for SaaS companies. Covers Trust Service Criteria mapping, control matrix generation, evidence collection, gap analysis, and audit readiness assessment.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Trust Service Criteria](#trust-service-criteria)
|
||||
- [Control Matrix Generation](#control-matrix-generation)
|
||||
- [Gap Analysis Workflow](#gap-analysis-workflow)
|
||||
- [Evidence Collection](#evidence-collection)
|
||||
- [Audit Readiness Checklist](#audit-readiness-checklist)
|
||||
- [Vendor Management](#vendor-management)
|
||||
- [Continuous Compliance](#continuous-compliance)
|
||||
- [Anti-Patterns](#anti-patterns)
|
||||
- [Tools](#tools)
|
||||
- [References](#references)
|
||||
- [Cross-References](#cross-references)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
### What Is SOC 2?
|
||||
|
||||
SOC 2 (System and Organization Controls 2) is an auditing framework developed by the AICPA that evaluates how a service organization manages customer data. It applies to any technology company that stores, processes, or transmits customer information — primarily SaaS, cloud infrastructure, and managed service providers.
|
||||
|
||||
### Type I vs Type II
|
||||
|
||||
| Aspect | Type I | Type II |
|
||||
|--------|--------|---------|
|
||||
| **Scope** | Design of controls at a point in time | Design AND operating effectiveness over a period |
|
||||
| **Duration** | Snapshot (single date) | Observation window (3-12 months, typically 6) |
|
||||
| **Evidence** | Control descriptions, policies | Control descriptions + operating evidence (logs, tickets, screenshots) |
|
||||
| **Cost** | $20K-$50K (audit fees) | $30K-$100K+ (audit fees) |
|
||||
| **Timeline** | 1-2 months (audit phase) | 6-12 months (observation + audit) |
|
||||
| **Best For** | First-time compliance, rapid market need | Mature organizations, enterprise customers |
|
||||
|
||||
### Who Needs SOC 2?
|
||||
|
||||
- **SaaS companies** selling to enterprise customers
|
||||
- **Cloud infrastructure providers** handling customer workloads
|
||||
- **Data processors** managing PII, PHI, or financial data
|
||||
- **Managed service providers** with access to client systems
|
||||
- **Any vendor** whose customers require third-party assurance
|
||||
|
||||
### Typical Journey
|
||||
|
||||
```
|
||||
Gap Assessment → Remediation → Type I Audit → Observation Period → Type II Audit → Annual Renewal
|
||||
(4-8 wk) (8-16 wk) (4-6 wk) (6-12 mo) (4-6 wk) (ongoing)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Trust Service Criteria
|
||||
|
||||
SOC 2 is organized around five Trust Service Criteria (TSC) categories. **Security** is required for every SOC 2 report; the remaining four are optional and selected based on business need.
|
||||
|
||||
### Security (Common Criteria CC1-CC9) — Required
|
||||
|
||||
The foundation of every SOC 2 report. Maps to COSO 2013 principles.
|
||||
|
||||
| Criteria | Domain | Key Controls |
|
||||
|----------|--------|-------------|
|
||||
| **CC1** | Control Environment | Integrity/ethics, board oversight, org structure, competence, accountability |
|
||||
| **CC2** | Communication & Information | Internal/external communication, information quality |
|
||||
| **CC3** | Risk Assessment | Risk identification, fraud risk, change impact analysis |
|
||||
| **CC4** | Monitoring Activities | Ongoing monitoring, deficiency evaluation, corrective actions |
|
||||
| **CC5** | Control Activities | Policies/procedures, technology controls, deployment through policies |
|
||||
| **CC6** | Logical & Physical Access | Access provisioning, authentication, encryption, physical restrictions |
|
||||
| **CC7** | System Operations | Vulnerability management, anomaly detection, incident response |
|
||||
| **CC8** | Change Management | Change authorization, testing, approval, emergency changes |
|
||||
| **CC9** | Risk Mitigation | Vendor/business partner risk management |
|
||||
|
||||
### Availability (A1) — Optional
|
||||
|
||||
| Criteria | Focus | Key Controls |
|
||||
|----------|-------|-------------|
|
||||
| **A1.1** | Capacity management | Infrastructure scaling, resource monitoring, capacity planning |
|
||||
| **A1.2** | Recovery operations | Backup procedures, disaster recovery, BCP testing |
|
||||
| **A1.3** | Recovery testing | DR drills, failover testing, RTO/RPO validation |
|
||||
|
||||
**Select when:** Customers depend on your uptime; you have SLAs; downtime causes direct business impact.
|
||||
|
||||
### Confidentiality (C1) — Optional
|
||||
|
||||
| Criteria | Focus | Key Controls |
|
||||
|----------|-------|-------------|
|
||||
| **C1.1** | Identification | Data classification policy, confidential data inventory |
|
||||
| **C1.2** | Protection | Encryption at rest and in transit, DLP, access restrictions |
|
||||
| **C1.3** | Disposal | Secure deletion procedures, media sanitization, retention enforcement |
|
||||
|
||||
**Select when:** You handle trade secrets, proprietary data, or contractually confidential information.
|
||||
|
||||
### Processing Integrity (PI1) — Optional
|
||||
|
||||
| Criteria | Focus | Key Controls |
|
||||
|----------|-------|-------------|
|
||||
| **PI1.1** | Accuracy | Input validation, processing checks, output verification |
|
||||
| **PI1.2** | Completeness | Transaction monitoring, reconciliation, error handling |
|
||||
| **PI1.3** | Timeliness | SLA monitoring, processing delay alerts, batch job monitoring |
|
||||
| **PI1.4** | Authorization | Processing authorization controls, segregation of duties |
|
||||
|
||||
**Select when:** Data accuracy is critical (financial processing, healthcare records, analytics platforms).
|
||||
|
||||
### Privacy (P1-P8) — Optional
|
||||
|
||||
| Criteria | Focus | Key Controls |
|
||||
|----------|-------|-------------|
|
||||
| **P1** | Notice | Privacy policy, data collection notice, purpose limitation |
|
||||
| **P2** | Choice & Consent | Opt-in/opt-out, consent management, preference tracking |
|
||||
| **P3** | Collection | Minimal collection, lawful basis, purpose specification |
|
||||
| **P4** | Use, Retention, Disposal | Purpose limitation, retention schedules, secure disposal |
|
||||
| **P5** | Access | Data subject access requests, correction rights |
|
||||
| **P6** | Disclosure & Notification | Third-party sharing, breach notification |
|
||||
| **P7** | Quality | Data accuracy verification, correction mechanisms |
|
||||
| **P8** | Monitoring & Enforcement | Privacy program monitoring, complaint handling |
|
||||
|
||||
**Select when:** You process PII and customers expect privacy assurance (complements GDPR compliance).
|
||||
|
||||
---
|
||||
|
||||
## Control Matrix Generation
|
||||
|
||||
A control matrix maps each TSC criterion to specific controls, owners, evidence, and testing procedures.
|
||||
|
||||
### Matrix Structure
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| **Control ID** | Unique identifier (e.g., SEC-001, AVL-003) |
|
||||
| **TSC Mapping** | Which criteria the control addresses (e.g., CC6.1, A1.2) |
|
||||
| **Control Description** | What the control does |
|
||||
| **Control Type** | Preventive, Detective, or Corrective |
|
||||
| **Owner** | Responsible person/team |
|
||||
| **Frequency** | Continuous, Daily, Weekly, Monthly, Quarterly, Annual |
|
||||
| **Evidence Type** | Screenshot, Log, Policy, Config, Ticket |
|
||||
| **Testing Procedure** | How the auditor verifies the control |
|
||||
|
||||
### Control Naming Convention
|
||||
|
||||
```
|
||||
{CATEGORY}-{NUMBER}
|
||||
SEC-001 through SEC-NNN → Security
|
||||
AVL-001 through AVL-NNN → Availability
|
||||
CON-001 through CON-NNN → Confidentiality
|
||||
PRI-001 through PRI-NNN → Processing Integrity
|
||||
PRV-001 through PRV-NNN → Privacy
|
||||
```
|
||||
|
||||
### Workflow
|
||||
|
||||
1. Select applicable TSC categories based on business needs
|
||||
2. Run `control_matrix_builder.py` to generate the baseline matrix
|
||||
3. Customize controls to match your actual environment
|
||||
4. Assign owners and evidence requirements
|
||||
5. Validate coverage — every selected TSC criterion must have at least one control
|
||||
|
||||
---
|
||||
|
||||
## Gap Analysis Workflow
|
||||
|
||||
### Phase 1: Current State Assessment
|
||||
|
||||
1. **Document existing controls** — inventory all security policies, procedures, and technical controls
|
||||
2. **Map to TSC** — align existing controls to Trust Service Criteria
|
||||
3. **Collect evidence samples** — gather proof that controls exist and operate
|
||||
4. **Interview control owners** — verify understanding and execution
|
||||
|
||||
### Phase 2: Gap Identification
|
||||
|
||||
Run `gap_analyzer.py` against your current controls to identify:
|
||||
|
||||
- **Missing controls** — TSC criteria with no corresponding control
|
||||
- **Partially implemented** — Control exists but lacks evidence or consistency
|
||||
- **Design gaps** — Control designed but does not adequately address the criteria
|
||||
- **Operating gaps** (Type II only) — Control designed correctly but not operating effectively
|
||||
|
||||
### Phase 3: Remediation Planning
|
||||
|
||||
For each gap, define:
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| Gap ID | Reference identifier |
|
||||
| TSC Criteria | Affected criteria |
|
||||
| Gap Description | What is missing or insufficient |
|
||||
| Remediation Action | Specific steps to close the gap |
|
||||
| Owner | Person responsible for remediation |
|
||||
| Priority | Critical / High / Medium / Low |
|
||||
| Target Date | Completion deadline |
|
||||
| Dependencies | Other gaps or projects that must complete first |
|
||||
|
||||
### Phase 4: Timeline Planning
|
||||
|
||||
| Priority | Target Remediation |
|
||||
|----------|--------------------|
|
||||
| Critical | 2-4 weeks |
|
||||
| High | 4-8 weeks |
|
||||
| Medium | 8-12 weeks |
|
||||
| Low | 12-16 weeks |
|
||||
|
||||
---
|
||||
|
||||
## Evidence Collection
|
||||
|
||||
### Evidence Types by Control Category
|
||||
|
||||
| Control Area | Primary Evidence | Secondary Evidence |
|
||||
|--------------|-----------------|-------------------|
|
||||
| Access Management | User access reviews, provisioning tickets | Role matrix, access logs |
|
||||
| Change Management | Change tickets, approval records | Deployment logs, test results |
|
||||
| Incident Response | Incident tickets, postmortems | Runbooks, escalation records |
|
||||
| Vulnerability Management | Scan reports, patch records | Remediation timelines |
|
||||
| Encryption | Configuration screenshots, certificate inventory | Key rotation logs |
|
||||
| Backup & Recovery | Backup logs, DR test results | Recovery time measurements |
|
||||
| Monitoring | Alert configurations, dashboard screenshots | On-call schedules, escalation records |
|
||||
| Policy Management | Signed policies, version history | Training completion records |
|
||||
| Vendor Management | Vendor assessments, SOC 2 reports | Contract reviews, risk registers |
|
||||
|
||||
### Automation Opportunities
|
||||
|
||||
| Area | Automation Approach |
|
||||
|------|-------------------|
|
||||
| Access reviews | Integrate IAM with ticketing (automatic quarterly review triggers) |
|
||||
| Configuration evidence | Infrastructure-as-code snapshots, compliance-as-code tools |
|
||||
| Vulnerability scans | Scheduled scanning with auto-generated reports |
|
||||
| Change management | Git-based audit trail (commits, PRs, approvals) |
|
||||
| Uptime monitoring | Automated SLA dashboards with historical data |
|
||||
| Backup verification | Automated restore tests with success/failure logging |
|
||||
|
||||
### Continuous Monitoring
|
||||
|
||||
Move from point-in-time evidence collection to continuous compliance:
|
||||
|
||||
1. **Automated evidence gathering** — scripts that pull evidence on schedule
|
||||
2. **Control dashboards** — real-time visibility into control status
|
||||
3. **Alert-based monitoring** — notify when a control drifts out of compliance
|
||||
4. **Evidence repository** — centralized, timestamped evidence storage
|
||||
|
||||
---
|
||||
|
||||
## Audit Readiness Checklist
|
||||
|
||||
### Pre-Audit Preparation (4-6 Weeks Before)
|
||||
|
||||
- [ ] All controls documented with descriptions, owners, and frequencies
|
||||
- [ ] Evidence collected for the entire observation period (Type II)
|
||||
- [ ] Control matrix reviewed and gaps remediated
|
||||
- [ ] Policies signed and distributed within the last 12 months
|
||||
- [ ] Access reviews completed within the required frequency
|
||||
- [ ] Vulnerability scans current (no critical/high unpatched > SLA)
|
||||
- [ ] Incident response plan tested within the last 12 months
|
||||
- [ ] Vendor risk assessments current for all subservice organizations
|
||||
- [ ] DR/BCP tested and documented within the last 12 months
|
||||
- [ ] Employee security training completed for all staff
|
||||
|
||||
### Readiness Scoring
|
||||
|
||||
| Score | Rating | Meaning |
|
||||
|-------|--------|---------|
|
||||
| 90-100% | Audit Ready | Proceed with confidence |
|
||||
| 75-89% | Minor Gaps | Address before scheduling audit |
|
||||
| 50-74% | Significant Gaps | Remediation required |
|
||||
| < 50% | Not Ready | Major program build-out needed |
|
||||
|
||||
### Common Audit Findings
|
||||
|
||||
| Finding | Root Cause | Prevention |
|
||||
|---------|-----------|-----------|
|
||||
| Incomplete access reviews | Manual process, no reminders | Automate quarterly review triggers |
|
||||
| Missing change approvals | Emergency changes bypass process | Define emergency change procedure with post-hoc approval |
|
||||
| Stale vulnerability scans | Scanner misconfigured | Automated weekly scans with alerting |
|
||||
| Policy not acknowledged | No tracking mechanism | Annual e-signature workflow |
|
||||
| Missing vendor assessments | No vendor inventory | Maintain vendor register with review schedule |
|
||||
|
||||
---
|
||||
|
||||
## Vendor Management
|
||||
|
||||
### Third-Party Risk Assessment
|
||||
|
||||
Every vendor that accesses, stores, or processes customer data must be assessed:
|
||||
|
||||
1. **Vendor inventory** — maintain a register of all service providers
|
||||
2. **Risk classification** — categorize vendors by data access level
|
||||
3. **Due diligence** — collect SOC 2 reports, security questionnaires, certifications
|
||||
4. **Contractual protections** — ensure DPAs, security requirements, breach notification clauses
|
||||
5. **Ongoing monitoring** — annual reassessment, continuous news monitoring
|
||||
|
||||
### Vendor Risk Tiers
|
||||
|
||||
| Tier | Data Access | Assessment Frequency | Requirements |
|
||||
|------|-------------|---------------------|-------------|
|
||||
| Critical | Processes/stores customer data | Annual + continuous monitoring | SOC 2 Type II, penetration test, security review |
|
||||
| High | Accesses customer environment | Annual | SOC 2 Type II or equivalent, questionnaire |
|
||||
| Medium | Indirect access, support tools | Annual questionnaire | Security certifications, questionnaire |
|
||||
| Low | No data access | Biennial questionnaire | Basic security questionnaire |
|
||||
|
||||
### Subservice Organizations
|
||||
|
||||
When your SOC 2 report relies on controls at a subservice organization (e.g., AWS, GCP, Azure):
|
||||
|
||||
- **Inclusive method** — your report covers the subservice org's controls (requires their cooperation)
|
||||
- **Carve-out method** — your report excludes their controls but references their SOC 2 report
|
||||
- Most companies use **carve-out** and include complementary user entity controls (CUECs)
|
||||
|
||||
---
|
||||
|
||||
## Continuous Compliance
|
||||
|
||||
### From Point-in-Time to Continuous
|
||||
|
||||
| Aspect | Point-in-Time | Continuous |
|
||||
|--------|---------------|-----------|
|
||||
| Evidence collection | Manual, before audit | Automated, ongoing |
|
||||
| Control monitoring | Periodic review | Real-time dashboards |
|
||||
| Drift detection | Found during audit | Alert-based, immediate |
|
||||
| Remediation | Reactive | Proactive |
|
||||
| Audit preparation | 4-8 week scramble | Always ready |
|
||||
|
||||
### Implementation Steps
|
||||
|
||||
1. **Automate evidence gathering** — cron jobs, API integrations, IaC snapshots
|
||||
2. **Build control dashboards** — aggregate control status into a single view
|
||||
3. **Configure drift alerts** — notify when controls fall out of compliance
|
||||
4. **Establish review cadence** — weekly control owner check-ins, monthly steering
|
||||
5. **Maintain evidence repository** — centralized, timestamped, auditor-accessible
|
||||
|
||||
### Annual Re-Assessment Cycle
|
||||
|
||||
| Quarter | Activities |
|
||||
|---------|-----------|
|
||||
| Q1 | Annual risk assessment, policy refresh, vendor reassessment launch |
|
||||
| Q2 | Internal control testing, remediation of findings |
|
||||
| Q3 | Pre-audit readiness review, evidence completeness check |
|
||||
| Q4 | External audit, management assertion, report distribution |
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
| Anti-Pattern | Why It Fails | Better Approach |
|
||||
|--------------|-------------|----------------|
|
||||
| Point-in-time compliance | Controls degrade between audits; gaps found during audit | Implement continuous monitoring and automated evidence |
|
||||
| Manual evidence collection | Time-consuming, inconsistent, error-prone | Automate with scripts, IaC, and compliance platforms |
|
||||
| Missing vendor assessments | Auditors flag incomplete vendor due diligence | Maintain vendor register with risk-tiered assessment schedule |
|
||||
| Copy-paste policies | Generic policies don't match actual operations | Tailor policies to your actual environment and technology stack |
|
||||
| Security theater | Controls exist on paper but aren't followed | Verify operating effectiveness; build controls into workflows |
|
||||
| Skipping Type I | Jumping to Type II without foundational readiness | Start with Type I to validate control design before observation |
|
||||
| Over-scoping TSC | Including all 5 categories when only Security is needed | Select categories based on actual customer/business requirements |
|
||||
| Treating audit as a project | Compliance degrades after the report is issued | Build compliance into daily operations and engineering culture |
|
||||
|
||||
---
|
||||
|
||||
## Tools
|
||||
|
||||
### Control Matrix Builder
|
||||
|
||||
Generates a SOC 2 control matrix from selected TSC categories.
|
||||
|
||||
```bash
|
||||
# Generate full security matrix in markdown
|
||||
python scripts/control_matrix_builder.py --categories security --format md
|
||||
|
||||
# Generate matrix for multiple categories as JSON
|
||||
python scripts/control_matrix_builder.py --categories security,availability,confidentiality --format json
|
||||
|
||||
# All categories, CSV output
|
||||
python scripts/control_matrix_builder.py --categories security,availability,confidentiality,processing-integrity,privacy --format csv
|
||||
```
|
||||
|
||||
### Evidence Tracker
|
||||
|
||||
Tracks evidence collection status per control.
|
||||
|
||||
```bash
|
||||
# Check evidence status from a control matrix
|
||||
python scripts/evidence_tracker.py --matrix controls.json --status
|
||||
|
||||
# JSON output for integration
|
||||
python scripts/evidence_tracker.py --matrix controls.json --status --json
|
||||
```
|
||||
|
||||
### Gap Analyzer
|
||||
|
||||
Analyzes current controls against SOC 2 requirements and identifies gaps.
|
||||
|
||||
```bash
|
||||
# Type I gap analysis
|
||||
python scripts/gap_analyzer.py --controls current_controls.json --type type1
|
||||
|
||||
# Type II gap analysis (includes operating effectiveness)
|
||||
python scripts/gap_analyzer.py --controls current_controls.json --type type2 --json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Trust Service Criteria Reference](https://github.com/alirezarezvani/claude-skills/tree/main/ra-qm-team/soc2-compliance/references/trust_service_criteria.md) — All 5 TSC categories with sub-criteria, control objectives, and evidence examples
|
||||
- [Evidence Collection Guide](https://github.com/alirezarezvani/claude-skills/tree/main/ra-qm-team/soc2-compliance/references/evidence_collection_guide.md) — Evidence types per control, automation tools, documentation requirements
|
||||
- [Type I vs Type II Comparison](https://github.com/alirezarezvani/claude-skills/tree/main/ra-qm-team/soc2-compliance/references/type1_vs_type2.md) — Detailed comparison, timeline, cost analysis, and upgrade path
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
- **[gdpr-dsgvo-expert](https://github.com/alirezarezvani/claude-skills/tree/main/ra-qm-team/gdpr-dsgvo-expert/SKILL.md)** — SOC 2 Privacy criteria overlaps significantly with GDPR requirements; use together when processing EU personal data
|
||||
- **[information-security-manager-iso27001](https://github.com/alirezarezvani/claude-skills/tree/main/ra-qm-team/information-security-manager-iso27001/SKILL.md)** — ISO 27001 Annex A controls map closely to SOC 2 Security criteria; organizations pursuing both can share evidence
|
||||
- **[isms-audit-expert](https://github.com/alirezarezvani/claude-skills/tree/main/ra-qm-team/isms-audit-expert/SKILL.md)** — Audit methodology and finding management patterns transfer directly to SOC 2 audit preparation
|
||||
Reference in New Issue
Block a user