feat: Add comprehensive incident-commander skill
- Add SKILL.md with 300+ lines of incident response playbook - Implement incident_classifier.py: severity classification and response recommendations - Implement timeline_reconstructor.py: event timeline reconstruction with phase analysis - Implement pir_generator.py: comprehensive PIR generation with multiple RCA frameworks - Add reference documentation: severity matrix, RCA frameworks, communication templates - Add sample data files and expected outputs for testing - All scripts are standalone with zero external dependencies - Dual output formats: JSON + human-readable text - Professional, opinionated defaults based on SRE best practices This POWERFUL-tier skill provides end-to-end incident response capabilities from detection through post-incident review.
This commit is contained in:
Leo
252
engineering-team/incident-commander/README.md
Normal file
252
engineering-team/incident-commander/README.md
Normal file
@@ -0,0 +1,252 @@
|
||||
# Incident Commander Skill
|
||||
|
||||
A comprehensive incident response framework providing structured tools for managing technology incidents from detection through resolution and post-incident review.
|
||||
|
||||
## Overview
|
||||
|
||||
This skill implements battle-tested practices from SRE and DevOps teams at scale, providing:
|
||||
|
||||
- **Automated Severity Classification** - Intelligent incident triage
|
||||
- **Timeline Reconstruction** - Transform scattered events into coherent narratives
|
||||
- **Post-Incident Review Generation** - Structured PIRs with RCA frameworks
|
||||
- **Communication Templates** - Pre-built stakeholder communication
|
||||
- **Comprehensive Documentation** - Reference guides for incident response
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Classify an Incident
|
||||
|
||||
```bash
|
||||
# From JSON file
|
||||
python scripts/incident_classifier.py --input incident.json --format text
|
||||
|
||||
# From stdin text
|
||||
echo "Database is down affecting all users" | python scripts/incident_classifier.py --format text
|
||||
|
||||
# Interactive mode
|
||||
python scripts/incident_classifier.py --interactive
|
||||
```
|
||||
|
||||
### Reconstruct Timeline
|
||||
|
||||
```bash
|
||||
# Analyze event timeline
|
||||
python scripts/timeline_reconstructor.py --input events.json --format text
|
||||
|
||||
# With gap analysis
|
||||
python scripts/timeline_reconstructor.py --input events.json --gap-analysis --format markdown
|
||||
```
|
||||
|
||||
### Generate PIR Document
|
||||
|
||||
```bash
|
||||
# Basic PIR
|
||||
python scripts/pir_generator.py --incident incident.json --format markdown
|
||||
|
||||
# Comprehensive PIR with timeline
|
||||
python scripts/pir_generator.py --incident incident.json --timeline timeline.json --rca-method fishbone
|
||||
```
|
||||
|
||||
## Scripts
|
||||
|
||||
### incident_classifier.py
|
||||
|
||||
**Purpose:** Analyzes incident descriptions and provides severity classification, team recommendations, and response templates.
|
||||
|
||||
**Input:** JSON object with incident details or plain text description
|
||||
**Output:** JSON + human-readable classification report
|
||||
|
||||
**Example Input:**
|
||||
```json
|
||||
{
|
||||
"description": "Database connection timeouts causing 500 errors",
|
||||
"service": "payment-api",
|
||||
"affected_users": "80%",
|
||||
"business_impact": "high"
|
||||
}
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- SEV1-4 severity classification
|
||||
- Recommended response teams
|
||||
- Initial action prioritization
|
||||
- Communication templates
|
||||
- Response timelines
|
||||
|
||||
### timeline_reconstructor.py
|
||||
|
||||
**Purpose:** Reconstructs incident timelines from timestamped events, identifies phases, and performs gap analysis.
|
||||
|
||||
**Input:** JSON array of timestamped events
|
||||
**Output:** Formatted timeline with phase analysis and metrics
|
||||
|
||||
**Example Input:**
|
||||
```json
|
||||
[
|
||||
{
|
||||
"timestamp": "2024-01-01T12:00:00Z",
|
||||
"source": "monitoring",
|
||||
"message": "High error rate detected",
|
||||
"severity": "critical",
|
||||
"actor": "system"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- Phase detection (detection → triage → mitigation → resolution)
|
||||
- Duration analysis
|
||||
- Gap identification
|
||||
- Communication effectiveness analysis
|
||||
- Response metrics
|
||||
|
||||
### pir_generator.py
|
||||
|
||||
**Purpose:** Generates comprehensive Post-Incident Review documents with multiple RCA frameworks.
|
||||
|
||||
**Input:** Incident data JSON, optional timeline data
|
||||
**Output:** Structured PIR document with RCA analysis
|
||||
|
||||
**Key Features:**
|
||||
- Multiple RCA methods (5 Whys, Fishbone, Timeline, Bow Tie)
|
||||
- Automated action item generation
|
||||
- Lessons learned categorization
|
||||
- Follow-up planning
|
||||
- Completeness assessment
|
||||
|
||||
## Sample Data
|
||||
|
||||
The `assets/` directory contains sample data files for testing:
|
||||
|
||||
- `sample_incident_classification.json` - Database connection pool exhaustion incident
|
||||
- `sample_timeline_events.json` - Complete timeline with 21 events across phases
|
||||
- `sample_incident_pir_data.json` - Comprehensive incident data for PIR generation
|
||||
- `simple_incident.json` - Minimal incident for basic testing
|
||||
- `simple_timeline_events.json` - Simple 4-event timeline
|
||||
|
||||
## Expected Outputs
|
||||
|
||||
The `expected_outputs/` directory contains reference outputs showing what each script produces:
|
||||
|
||||
- `incident_classification_text_output.txt` - Detailed classification report
|
||||
- `timeline_reconstruction_text_output.txt` - Complete timeline analysis
|
||||
- `pir_markdown_output.md` - Full PIR document
|
||||
- `simple_incident_classification.txt` - Basic classification example
|
||||
|
||||
## Reference Documentation
|
||||
|
||||
### references/incident_severity_matrix.md
|
||||
Complete severity classification system with:
|
||||
- SEV1-4 definitions and criteria
|
||||
- Response requirements and timelines
|
||||
- Escalation paths
|
||||
- Communication requirements
|
||||
- Decision trees and examples
|
||||
|
||||
### references/rca_frameworks_guide.md
|
||||
Detailed guide for root cause analysis:
|
||||
- 5 Whys methodology
|
||||
- Fishbone (Ishikawa) diagram analysis
|
||||
- Timeline analysis techniques
|
||||
- Bow Tie analysis for high-risk incidents
|
||||
- Framework selection guidelines
|
||||
|
||||
### references/communication_templates.md
|
||||
Standardized communication templates:
|
||||
- Severity-specific notification templates
|
||||
- Stakeholder-specific messaging
|
||||
- Escalation communications
|
||||
- Resolution notifications
|
||||
- Customer communication guidelines
|
||||
|
||||
## Usage Patterns
|
||||
|
||||
### End-to-End Incident Workflow
|
||||
|
||||
1. **Initial Classification**
|
||||
```bash
|
||||
echo "Payment API returning 500 errors for 70% of requests" | \
|
||||
python scripts/incident_classifier.py --format text
|
||||
```
|
||||
|
||||
2. **Timeline Reconstruction** (after collecting events)
|
||||
```bash
|
||||
python scripts/timeline_reconstructor.py \
|
||||
--input events.json \
|
||||
--gap-analysis \
|
||||
--format markdown \
|
||||
--output timeline.md
|
||||
```
|
||||
|
||||
3. **PIR Generation** (after incident resolution)
|
||||
```bash
|
||||
python scripts/pir_generator.py \
|
||||
--incident incident.json \
|
||||
--timeline timeline.md \
|
||||
--rca-method fishbone \
|
||||
--output pir.md
|
||||
```
|
||||
|
||||
### Integration Examples
|
||||
|
||||
**CI/CD Pipeline Integration:**
|
||||
```bash
|
||||
# Classify deployment issues
|
||||
cat deployment_error.log | python scripts/incident_classifier.py --format json
|
||||
```
|
||||
|
||||
**Monitoring Integration:**
|
||||
```bash
|
||||
# Process alert events
|
||||
curl -s "monitoring-api/events" | python scripts/timeline_reconstructor.py --format text
|
||||
```
|
||||
|
||||
**Runbook Generation:**
|
||||
Use classification output to automatically select appropriate runbooks and escalation procedures.
|
||||
|
||||
## Quality Standards
|
||||
|
||||
- **Zero External Dependencies** - All scripts use only Python standard library
|
||||
- **Dual Output Format** - Both JSON (machine-readable) and text (human-readable)
|
||||
- **Robust Input Handling** - Graceful handling of missing or malformed data
|
||||
- **Professional Defaults** - Opinionated, battle-tested configurations
|
||||
- **Comprehensive Testing** - Sample data and expected outputs included
|
||||
|
||||
## Technical Requirements
|
||||
|
||||
- Python 3.6+
|
||||
- No external dependencies required
|
||||
- Works with standard Unix tools (pipes, redirection)
|
||||
- Cross-platform compatible
|
||||
|
||||
## Severity Classification Reference
|
||||
|
||||
| Severity | Description | Response Time | Update Frequency |
|
||||
|----------|-------------|---------------|------------------|
|
||||
| **SEV1** | Complete outage | 5 minutes | Every 15 minutes |
|
||||
| **SEV2** | Major degradation | 15 minutes | Every 30 minutes |
|
||||
| **SEV3** | Minor impact | 2 hours | At milestones |
|
||||
| **SEV4** | Low impact | 1-2 days | Weekly |
|
||||
|
||||
## Getting Help
|
||||
|
||||
Each script includes comprehensive help:
|
||||
```bash
|
||||
python scripts/incident_classifier.py --help
|
||||
python scripts/timeline_reconstructor.py --help
|
||||
python scripts/pir_generator.py --help
|
||||
```
|
||||
|
||||
For methodology questions, refer to the reference documentation in the `references/` directory.
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new features:
|
||||
1. Maintain zero external dependencies
|
||||
2. Add comprehensive examples to `assets/`
|
||||
3. Update expected outputs in `expected_outputs/`
|
||||
4. Follow the established patterns for argument parsing and output formatting
|
||||
|
||||
## License
|
||||
|
||||
This skill is part of the claude-skills repository. See the main repository LICENSE for details.
|
||||
668
engineering-team/incident-commander/SKILL.md
Normal file
668
engineering-team/incident-commander/SKILL.md
Normal file
@@ -0,0 +1,668 @@
|
||||
# Incident Commander Skill
|
||||
|
||||
**Category:** Engineering Team
|
||||
**Tier:** POWERFUL
|
||||
**Author:** Claude Skills Team
|
||||
**Version:** 1.0.0
|
||||
**Last Updated:** February 2026
|
||||
|
||||
## Overview
|
||||
|
||||
The Incident Commander skill provides a comprehensive incident response framework for managing technology incidents from detection through resolution and post-incident review. This skill implements battle-tested practices from SRE and DevOps teams at scale, providing structured tools for severity classification, timeline reconstruction, and thorough post-incident analysis.
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Automated Severity Classification** - Intelligent incident triage based on impact and urgency metrics
|
||||
- **Timeline Reconstruction** - Transform scattered logs and events into coherent incident narratives
|
||||
- **Post-Incident Review Generation** - Structured PIRs with multiple RCA frameworks
|
||||
- **Communication Templates** - Pre-built templates for stakeholder updates and escalations
|
||||
- **Runbook Integration** - Generate actionable runbooks from incident patterns
|
||||
|
||||
## Skills Included
|
||||
|
||||
### Core Tools
|
||||
|
||||
1. **Incident Classifier** (`incident_classifier.py`)
|
||||
- Analyzes incident descriptions and outputs severity levels
|
||||
- Recommends response teams and initial actions
|
||||
- Generates communication templates based on severity
|
||||
|
||||
2. **Timeline Reconstructor** (`timeline_reconstructor.py`)
|
||||
- Processes timestamped events from multiple sources
|
||||
- Reconstructs chronological incident timeline
|
||||
- Identifies gaps and provides duration analysis
|
||||
|
||||
3. **PIR Generator** (`pir_generator.py`)
|
||||
- Creates comprehensive Post-Incident Review documents
|
||||
- Applies multiple RCA frameworks (5 Whys, Fishbone, Timeline)
|
||||
- Generates actionable follow-up items
|
||||
|
||||
## Incident Response Framework
|
||||
|
||||
### Severity Classification System
|
||||
|
||||
#### SEV1 - Critical Outage
|
||||
**Definition:** Complete service failure affecting all users or critical business functions
|
||||
|
||||
**Characteristics:**
|
||||
- Customer-facing services completely unavailable
|
||||
- Data loss or corruption affecting users
|
||||
- Security breaches with customer data exposure
|
||||
- Revenue-generating systems down
|
||||
- SLA violations with financial penalties
|
||||
|
||||
**Response Requirements:**
|
||||
- Immediate escalation to on-call engineer
|
||||
- Incident Commander assigned within 5 minutes
|
||||
- Executive notification within 15 minutes
|
||||
- Public status page update within 15 minutes
|
||||
- War room established
|
||||
- All hands on deck if needed
|
||||
|
||||
**Communication Frequency:** Every 15 minutes until resolution
|
||||
|
||||
#### SEV2 - Major Impact
|
||||
**Definition:** Significant degradation affecting subset of users or non-critical functions
|
||||
|
||||
**Characteristics:**
|
||||
- Partial service degradation (>25% of users affected)
|
||||
- Performance issues causing user frustration
|
||||
- Non-critical features unavailable
|
||||
- Internal tools impacting productivity
|
||||
- Data inconsistencies not affecting user experience
|
||||
|
||||
**Response Requirements:**
|
||||
- On-call engineer response within 15 minutes
|
||||
- Incident Commander assigned within 30 minutes
|
||||
- Status page update within 30 minutes
|
||||
- Stakeholder notification within 1 hour
|
||||
- Regular team updates
|
||||
|
||||
**Communication Frequency:** Every 30 minutes during active response
|
||||
|
||||
#### SEV3 - Minor Impact
|
||||
**Definition:** Limited impact with workarounds available
|
||||
|
||||
**Characteristics:**
|
||||
- Single feature or component affected
|
||||
- <25% of users impacted
|
||||
- Workarounds available
|
||||
- Performance degradation not significantly impacting UX
|
||||
- Non-urgent monitoring alerts
|
||||
|
||||
**Response Requirements:**
|
||||
- Response within 2 hours during business hours
|
||||
- Next business day response acceptable outside hours
|
||||
- Internal team notification
|
||||
- Optional status page update
|
||||
|
||||
**Communication Frequency:** At key milestones only
|
||||
|
||||
#### SEV4 - Low Impact
|
||||
**Definition:** Minimal impact, cosmetic issues, or planned maintenance
|
||||
|
||||
**Characteristics:**
|
||||
- Cosmetic bugs
|
||||
- Documentation issues
|
||||
- Logging or monitoring gaps
|
||||
- Performance issues with no user impact
|
||||
- Development/test environment issues
|
||||
|
||||
**Response Requirements:**
|
||||
- Response within 1-2 business days
|
||||
- Standard ticket/issue tracking
|
||||
- No special escalation required
|
||||
|
||||
**Communication Frequency:** Standard development cycle updates
|
||||
|
||||
### Incident Commander Role
|
||||
|
||||
#### Primary Responsibilities
|
||||
|
||||
1. **Command and Control**
|
||||
- Own the incident response process
|
||||
- Make critical decisions about resource allocation
|
||||
- Coordinate between technical teams and stakeholders
|
||||
- Maintain situational awareness across all response streams
|
||||
|
||||
2. **Communication Hub**
|
||||
- Provide regular updates to stakeholders
|
||||
- Manage external communications (status pages, customer notifications)
|
||||
- Facilitate effective communication between response teams
|
||||
- Shield responders from external distractions
|
||||
|
||||
3. **Process Management**
|
||||
- Ensure proper incident tracking and documentation
|
||||
- Drive toward resolution while maintaining quality
|
||||
- Coordinate handoffs between team members
|
||||
- Plan and execute rollback strategies if needed
|
||||
|
||||
4. **Post-Incident Leadership**
|
||||
- Ensure thorough post-incident reviews are conducted
|
||||
- Drive implementation of preventive measures
|
||||
- Share learnings with broader organization
|
||||
|
||||
#### Decision-Making Framework
|
||||
|
||||
**Emergency Decisions (SEV1/2):**
|
||||
- Incident Commander has full authority
|
||||
- Bias toward action over analysis
|
||||
- Document decisions for later review
|
||||
- Consult subject matter experts but don't get blocked
|
||||
|
||||
**Resource Allocation:**
|
||||
- Can pull in any necessary team members
|
||||
- Authority to escalate to senior leadership
|
||||
- Can approve emergency spend for external resources
|
||||
- Make call on communication channels and timing
|
||||
|
||||
**Technical Decisions:**
|
||||
- Lean on technical leads for implementation details
|
||||
- Make final calls on trade-offs between speed and risk
|
||||
- Approve rollback vs. fix-forward strategies
|
||||
- Coordinate testing and validation approaches
|
||||
|
||||
### Communication Templates
|
||||
|
||||
#### Initial Incident Notification (SEV1/2)
|
||||
|
||||
```
|
||||
Subject: [SEV{severity}] {Service Name} - {Brief Description}
|
||||
|
||||
Incident Details:
|
||||
- Start Time: {timestamp}
|
||||
- Severity: SEV{level}
|
||||
- Impact: {user impact description}
|
||||
- Current Status: {investigating/mitigating/resolved}
|
||||
|
||||
Technical Details:
|
||||
- Affected Services: {service list}
|
||||
- Symptoms: {what users are experiencing}
|
||||
- Initial Assessment: {suspected root cause if known}
|
||||
|
||||
Response Team:
|
||||
- Incident Commander: {name}
|
||||
- Technical Lead: {name}
|
||||
- SMEs Engaged: {list}
|
||||
|
||||
Next Update: {timestamp}
|
||||
Status Page: {link}
|
||||
War Room: {bridge/chat link}
|
||||
|
||||
---
|
||||
{Incident Commander Name}
|
||||
{Contact Information}
|
||||
```
|
||||
|
||||
#### Executive Summary (SEV1)
|
||||
|
||||
```
|
||||
Subject: URGENT - Customer-Impacting Outage - {Service Name}
|
||||
|
||||
Executive Summary:
|
||||
{2-3 sentence description of customer impact and business implications}
|
||||
|
||||
Key Metrics:
|
||||
- Time to Detection: {X minutes}
|
||||
- Time to Engagement: {X minutes}
|
||||
- Estimated Customer Impact: {number/percentage}
|
||||
- Current Status: {status}
|
||||
- ETA to Resolution: {time or "investigating"}
|
||||
|
||||
Leadership Actions Required:
|
||||
- [ ] Customer communication approval
|
||||
- [ ] PR/Communications coordination
|
||||
- [ ] Resource allocation decisions
|
||||
- [ ] External vendor engagement
|
||||
|
||||
Incident Commander: {name} ({contact})
|
||||
Next Update: {time}
|
||||
|
||||
---
|
||||
This is an automated alert from our incident response system.
|
||||
```
|
||||
|
||||
#### Customer Communication Template
|
||||
|
||||
```
|
||||
We are currently experiencing {brief description of issue} affecting {scope of impact}.
|
||||
|
||||
Our engineering team was alerted at {time} and is actively working to resolve the issue. We will provide updates every {frequency} until resolved.
|
||||
|
||||
What we know:
|
||||
- {factual statement of impact}
|
||||
- {factual statement of scope}
|
||||
- {brief status of response}
|
||||
|
||||
What we're doing:
|
||||
- {primary response action}
|
||||
- {secondary response action}
|
||||
|
||||
Workaround (if available):
|
||||
{workaround steps or "No workaround currently available"}
|
||||
|
||||
We apologize for the inconvenience and will share more information as it becomes available.
|
||||
|
||||
Next update: {time}
|
||||
Status page: {link}
|
||||
```
|
||||
|
||||
### Stakeholder Management
|
||||
|
||||
#### Stakeholder Classification
|
||||
|
||||
**Internal Stakeholders:**
|
||||
- **Engineering Leadership** - Technical decisions and resource allocation
|
||||
- **Product Management** - Customer impact assessment and feature implications
|
||||
- **Customer Support** - User communication and support ticket management
|
||||
- **Sales/Account Management** - Customer relationship management for enterprise clients
|
||||
- **Executive Team** - Business impact decisions and external communication approval
|
||||
- **Legal/Compliance** - Regulatory reporting and liability assessment
|
||||
|
||||
**External Stakeholders:**
|
||||
- **Customers** - Service availability and impact communication
|
||||
- **Partners** - API availability and integration impacts
|
||||
- **Vendors** - Third-party service dependencies and support escalation
|
||||
- **Regulators** - Compliance reporting for regulated industries
|
||||
- **Public/Media** - Transparency for public-facing outages
|
||||
|
||||
#### Communication Cadence by Stakeholder
|
||||
|
||||
| Stakeholder | SEV1 | SEV2 | SEV3 | SEV4 |
|
||||
|-------------|------|------|------|------|
|
||||
| Engineering Leadership | Real-time | 30min | 4hrs | Daily |
|
||||
| Executive Team | 15min | 1hr | EOD | Weekly |
|
||||
| Customer Support | Real-time | 30min | 2hrs | As needed |
|
||||
| Customers | 15min | 1hr | Optional | None |
|
||||
| Partners | 30min | 2hrs | Optional | None |
|
||||
|
||||
### Runbook Generation Framework
|
||||
|
||||
#### Dynamic Runbook Components
|
||||
|
||||
1. **Detection Playbooks**
|
||||
- Monitoring alert definitions
|
||||
- Triage decision trees
|
||||
- Escalation trigger points
|
||||
- Initial response actions
|
||||
|
||||
2. **Response Playbooks**
|
||||
- Step-by-step mitigation procedures
|
||||
- Rollback instructions
|
||||
- Validation checkpoints
|
||||
- Communication checkpoints
|
||||
|
||||
3. **Recovery Playbooks**
|
||||
- Service restoration procedures
|
||||
- Data consistency checks
|
||||
- Performance validation
|
||||
- User notification processes
|
||||
|
||||
#### Runbook Template Structure
|
||||
|
||||
```markdown
|
||||
# {Service/Component} Incident Response Runbook
|
||||
|
||||
## Quick Reference
|
||||
- **Severity Indicators:** {list of conditions for each severity level}
|
||||
- **Key Contacts:** {on-call rotations and escalation paths}
|
||||
- **Critical Commands:** {list of emergency commands with descriptions}
|
||||
|
||||
## Detection
|
||||
### Monitoring Alerts
|
||||
- {Alert name}: {description and thresholds}
|
||||
- {Alert name}: {description and thresholds}
|
||||
|
||||
### Manual Detection Signs
|
||||
- {Symptom}: {what to look for and where}
|
||||
- {Symptom}: {what to look for and where}
|
||||
|
||||
## Initial Response (0-15 minutes)
|
||||
1. **Assess Severity**
|
||||
- [ ] Check {primary metric}
|
||||
- [ ] Verify {secondary indicator}
|
||||
- [ ] Classify as SEV{level} based on {criteria}
|
||||
|
||||
2. **Establish Command**
|
||||
- [ ] Page Incident Commander if SEV1/2
|
||||
- [ ] Create incident tracking ticket
|
||||
- [ ] Join war room: {link/bridge info}
|
||||
|
||||
3. **Initial Investigation**
|
||||
- [ ] Check recent deployments: {deployment log location}
|
||||
- [ ] Review error logs: {log location and queries}
|
||||
- [ ] Verify dependencies: {dependency check commands}
|
||||
|
||||
## Mitigation Strategies
|
||||
### Strategy 1: {Name}
|
||||
**Use when:** {conditions}
|
||||
**Steps:**
|
||||
1. {detailed step with commands}
|
||||
2. {detailed step with expected outcomes}
|
||||
3. {validation step}
|
||||
|
||||
**Rollback Plan:**
|
||||
1. {rollback step}
|
||||
2. {verification step}
|
||||
|
||||
### Strategy 2: {Name}
|
||||
{similar structure}
|
||||
|
||||
## Recovery and Validation
|
||||
1. **Service Restoration**
|
||||
- [ ] {restoration step}
|
||||
- [ ] Wait for {metric} to return to normal
|
||||
- [ ] Validate end-to-end functionality
|
||||
|
||||
2. **Communication**
|
||||
- [ ] Update status page
|
||||
- [ ] Notify stakeholders
|
||||
- [ ] Schedule PIR
|
||||
|
||||
## Common Pitfalls
|
||||
- **{Pitfall}:** {description and how to avoid}
|
||||
- **{Pitfall}:** {description and how to avoid}
|
||||
|
||||
## Reference Information
|
||||
- **Architecture Diagram:** {link}
|
||||
- **Monitoring Dashboard:** {link}
|
||||
- **Related Runbooks:** {links to dependent service runbooks}
|
||||
```
|
||||
|
||||
### Post-Incident Review (PIR) Framework
|
||||
|
||||
#### PIR Timeline and Ownership
|
||||
|
||||
**Timeline:**
|
||||
- **24 hours:** Initial PIR draft completed by Incident Commander
|
||||
- **3 business days:** Final PIR published with all stakeholder input
|
||||
- **1 week:** Action items assigned with owners and due dates
|
||||
- **4 weeks:** Follow-up review on action item progress
|
||||
|
||||
**Roles:**
|
||||
- **PIR Owner:** Incident Commander (can delegate writing but owns completion)
|
||||
- **Technical Contributors:** All engineers involved in response
|
||||
- **Review Committee:** Engineering leadership, affected product teams
|
||||
- **Action Item Owners:** Assigned based on expertise and capacity
|
||||
|
||||
#### Root Cause Analysis Frameworks
|
||||
|
||||
#### 1. Five Whys Method
|
||||
|
||||
The Five Whys technique involves asking "why" repeatedly to drill down to root causes:
|
||||
|
||||
**Example Application:**
|
||||
- **Problem:** Database became unresponsive during peak traffic
|
||||
- **Why 1:** Why did the database become unresponsive? → Connection pool was exhausted
|
||||
- **Why 2:** Why was the connection pool exhausted? → Application was creating more connections than usual
|
||||
- **Why 3:** Why was the application creating more connections? → New feature wasn't properly connection pooling
|
||||
- **Why 4:** Why wasn't the feature properly connection pooling? → Code review missed this pattern
|
||||
- **Why 5:** Why did code review miss this? → No automated checks for connection pooling patterns
|
||||
|
||||
**Best Practices:**
|
||||
- Ask "why" at least 3 times, often need 5+ iterations
|
||||
- Focus on process failures, not individual blame
|
||||
- Each "why" should point to a actionable system improvement
|
||||
- Consider multiple root cause paths, not just one linear chain
|
||||
|
||||
#### 2. Fishbone (Ishikawa) Diagram
|
||||
|
||||
Systematic analysis across multiple categories of potential causes:
|
||||
|
||||
**Categories:**
|
||||
- **People:** Training, experience, communication, handoffs
|
||||
- **Process:** Procedures, change management, review processes
|
||||
- **Technology:** Architecture, tooling, monitoring, automation
|
||||
- **Environment:** Infrastructure, dependencies, external factors
|
||||
|
||||
**Application Method:**
|
||||
1. State the problem clearly at the "head" of the fishbone
|
||||
2. For each category, brainstorm potential contributing factors
|
||||
3. For each factor, ask what caused that factor (sub-causes)
|
||||
4. Identify the factors most likely to be root causes
|
||||
5. Validate root causes with evidence from the incident
|
||||
|
||||
#### 3. Timeline Analysis
|
||||
|
||||
Reconstruct the incident chronologically to identify decision points and missed opportunities:
|
||||
|
||||
**Timeline Elements:**
|
||||
- **Detection:** When was the issue first observable? When was it first detected?
|
||||
- **Notification:** How quickly were the right people informed?
|
||||
- **Response:** What actions were taken and how effective were they?
|
||||
- **Communication:** When were stakeholders updated?
|
||||
- **Resolution:** What finally resolved the issue?
|
||||
|
||||
**Analysis Questions:**
|
||||
- Where were there delays and what caused them?
|
||||
- What decisions would we make differently with perfect information?
|
||||
- Where did communication break down?
|
||||
- What automation could have detected/resolved faster?
|
||||
|
||||
### Escalation Paths
|
||||
|
||||
#### Technical Escalation
|
||||
|
||||
**Level 1:** On-call engineer
|
||||
- **Responsibility:** Initial response and common issue resolution
|
||||
- **Escalation Trigger:** Issue not resolved within SLA timeframe
|
||||
- **Timeframe:** 15 minutes (SEV1), 30 minutes (SEV2)
|
||||
|
||||
**Level 2:** Senior engineer/Team lead
|
||||
- **Responsibility:** Complex technical issues requiring deeper expertise
|
||||
- **Escalation Trigger:** Level 1 requests help or timeout occurs
|
||||
- **Timeframe:** 30 minutes (SEV1), 1 hour (SEV2)
|
||||
|
||||
**Level 3:** Engineering Manager/Staff Engineer
|
||||
- **Responsibility:** Cross-team coordination and architectural decisions
|
||||
- **Escalation Trigger:** Issue spans multiple systems or teams
|
||||
- **Timeframe:** 45 minutes (SEV1), 2 hours (SEV2)
|
||||
|
||||
**Level 4:** Director of Engineering/CTO
|
||||
- **Responsibility:** Resource allocation and business impact decisions
|
||||
- **Escalation Trigger:** Extended outage or significant business impact
|
||||
- **Timeframe:** 1 hour (SEV1), 4 hours (SEV2)
|
||||
|
||||
#### Business Escalation
|
||||
|
||||
**Customer Impact Assessment:**
|
||||
- **High:** Revenue loss, SLA breaches, customer churn risk
|
||||
- **Medium:** User experience degradation, support ticket volume
|
||||
- **Low:** Internal tools, development impact only
|
||||
|
||||
**Escalation Matrix:**
|
||||
|
||||
| Severity | Duration | Business Escalation |
|
||||
|----------|----------|-------------------|
|
||||
| SEV1 | Immediate | VP Engineering |
|
||||
| SEV1 | 30 minutes | CTO + Customer Success VP |
|
||||
| SEV1 | 1 hour | CEO + Full Executive Team |
|
||||
| SEV2 | 2 hours | VP Engineering |
|
||||
| SEV2 | 4 hours | CTO |
|
||||
| SEV3 | 1 business day | Engineering Manager |
|
||||
|
||||
### Status Page Management
|
||||
|
||||
#### Update Principles
|
||||
|
||||
1. **Transparency:** Provide factual information without speculation
|
||||
2. **Timeliness:** Update within committed timeframes
|
||||
3. **Clarity:** Use customer-friendly language, avoid technical jargon
|
||||
4. **Completeness:** Include impact scope, status, and next update time
|
||||
|
||||
#### Status Categories
|
||||
|
||||
- **Operational:** All systems functioning normally
|
||||
- **Degraded Performance:** Some users may experience slowness
|
||||
- **Partial Outage:** Subset of features unavailable
|
||||
- **Major Outage:** Service unavailable for most/all users
|
||||
- **Under Maintenance:** Planned maintenance window
|
||||
|
||||
#### Update Template
|
||||
|
||||
```
|
||||
{Timestamp} - {Status Category}
|
||||
|
||||
{Brief description of current state}
|
||||
|
||||
Impact: {who is affected and how}
|
||||
Cause: {root cause if known, "under investigation" if not}
|
||||
Resolution: {what's being done to fix it}
|
||||
|
||||
Next update: {specific time}
|
||||
|
||||
We apologize for any inconvenience this may cause.
|
||||
```
|
||||
|
||||
### Action Item Framework
|
||||
|
||||
#### Action Item Categories
|
||||
|
||||
1. **Immediate Fixes**
|
||||
- Critical bugs discovered during incident
|
||||
- Security vulnerabilities exposed
|
||||
- Data integrity issues
|
||||
|
||||
2. **Process Improvements**
|
||||
- Communication gaps
|
||||
- Escalation procedure updates
|
||||
- Runbook additions/updates
|
||||
|
||||
3. **Technical Debt**
|
||||
- Architecture improvements
|
||||
- Monitoring enhancements
|
||||
- Automation opportunities
|
||||
|
||||
4. **Organizational Changes**
|
||||
- Team structure adjustments
|
||||
- Training requirements
|
||||
- Tool/platform investments
|
||||
|
||||
#### Action Item Template
|
||||
|
||||
```
|
||||
**Title:** {Concise description of the action}
|
||||
**Priority:** {Critical/High/Medium/Low}
|
||||
**Category:** {Fix/Process/Technical/Organizational}
|
||||
**Owner:** {Assigned person}
|
||||
**Due Date:** {Specific date}
|
||||
**Success Criteria:** {How will we know this is complete}
|
||||
**Dependencies:** {What needs to happen first}
|
||||
**Related PIRs:** {Links to other incidents this addresses}
|
||||
|
||||
**Description:**
|
||||
{Detailed description of what needs to be done and why}
|
||||
|
||||
**Implementation Plan:**
|
||||
1. {Step 1}
|
||||
2. {Step 2}
|
||||
3. {Validation step}
|
||||
|
||||
**Progress Updates:**
|
||||
- {Date}: {Progress update}
|
||||
- {Date}: {Progress update}
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Example 1: Database Connection Pool Exhaustion
|
||||
|
||||
```bash
|
||||
# Classify the incident
|
||||
echo '{"description": "Users reporting 500 errors, database connections timing out", "affected_users": "80%", "business_impact": "high"}' | python scripts/incident_classifier.py
|
||||
|
||||
# Reconstruct timeline from logs
|
||||
python scripts/timeline_reconstructor.py --input assets/db_incident_events.json --output timeline.md
|
||||
|
||||
# Generate PIR after resolution
|
||||
python scripts/pir_generator.py --incident assets/db_incident_data.json --timeline timeline.md --output pir.md
|
||||
```
|
||||
|
||||
### Example 2: API Rate Limiting Incident
|
||||
|
||||
```bash
|
||||
# Quick classification from stdin
|
||||
echo "API rate limits causing customer API calls to fail" | python scripts/incident_classifier.py --format text
|
||||
|
||||
# Build timeline from multiple sources
|
||||
python scripts/timeline_reconstructor.py --input assets/api_incident_logs.json --detect-phases --gap-analysis
|
||||
|
||||
# Generate comprehensive PIR
|
||||
python scripts/pir_generator.py --incident assets/api_incident_summary.json --rca-method fishbone --action-items
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### During Incident Response
|
||||
|
||||
1. **Maintain Calm Leadership**
|
||||
- Stay composed under pressure
|
||||
- Make decisive calls with incomplete information
|
||||
- Communicate confidence while acknowledging uncertainty
|
||||
|
||||
2. **Document Everything**
|
||||
- All actions taken and their outcomes
|
||||
- Decision rationale, especially for controversial calls
|
||||
- Timeline of events as they happen
|
||||
|
||||
3. **Effective Communication**
|
||||
- Use clear, jargon-free language
|
||||
- Provide regular updates even when there's no new information
|
||||
- Manage stakeholder expectations proactively
|
||||
|
||||
4. **Technical Excellence**
|
||||
- Prefer rollbacks to risky fixes under pressure
|
||||
- Validate fixes before declaring resolution
|
||||
- Plan for secondary failures and cascading effects
|
||||
|
||||
### Post-Incident
|
||||
|
||||
1. **Blameless Culture**
|
||||
- Focus on system failures, not individual mistakes
|
||||
- Encourage honest reporting of what went wrong
|
||||
- Celebrate learning and improvement opportunities
|
||||
|
||||
2. **Action Item Discipline**
|
||||
- Assign specific owners and due dates
|
||||
- Track progress publicly
|
||||
- Prioritize based on risk and effort
|
||||
|
||||
3. **Knowledge Sharing**
|
||||
- Share PIRs broadly within the organization
|
||||
- Update runbooks based on lessons learned
|
||||
- Conduct training sessions for common failure modes
|
||||
|
||||
4. **Continuous Improvement**
|
||||
- Look for patterns across multiple incidents
|
||||
- Invest in tooling and automation
|
||||
- Regularly review and update processes
|
||||
|
||||
## Integration with Existing Tools
|
||||
|
||||
### Monitoring and Alerting
|
||||
- PagerDuty/Opsgenie integration for escalation
|
||||
- Datadog/Grafana for metrics and dashboards
|
||||
- ELK/Splunk for log analysis and correlation
|
||||
|
||||
### Communication Platforms
|
||||
- Slack/Teams for war room coordination
|
||||
- Zoom/Meet for video bridges
|
||||
- Status page providers (Statuspage.io, etc.)
|
||||
|
||||
### Documentation Systems
|
||||
- Confluence/Notion for PIR storage
|
||||
- GitHub/GitLab for runbook version control
|
||||
- JIRA/Linear for action item tracking
|
||||
|
||||
### Change Management
|
||||
- CI/CD pipeline integration
|
||||
- Deployment tracking systems
|
||||
- Feature flag platforms for quick rollbacks
|
||||
|
||||
## Conclusion
|
||||
|
||||
The Incident Commander skill provides a comprehensive framework for managing incidents from detection through post-incident review. By implementing structured processes, clear communication templates, and thorough analysis tools, teams can improve their incident response capabilities and build more resilient systems.
|
||||
|
||||
The key to successful incident management is preparation, practice, and continuous learning. Use this framework as a starting point, but adapt it to your organization's specific needs, culture, and technical environment.
|
||||
|
||||
Remember: The goal isn't to prevent all incidents (which is impossible), but to detect them quickly, respond effectively, communicate clearly, and learn continuously.
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"description": "Database connection timeouts causing 500 errors for payment processing API. Users unable to complete checkout. Error rate spiked from 0.1% to 45% starting at 14:30 UTC. Database monitoring shows connection pool exhaustion with 200/200 connections active.",
|
||||
"service": "payment-api",
|
||||
"affected_users": "80%",
|
||||
"business_impact": "high",
|
||||
"duration_minutes": 95,
|
||||
"metadata": {
|
||||
"error_rate": "45%",
|
||||
"connection_pool_utilization": "100%",
|
||||
"affected_regions": ["us-west", "us-east", "eu-west"],
|
||||
"detection_method": "monitoring_alert",
|
||||
"customer_escalations": 12
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
{
|
||||
"incident_id": "INC-2024-0315-001",
|
||||
"title": "Payment API Database Connection Pool Exhaustion",
|
||||
"description": "Database connection pool exhaustion caused widespread 500 errors in payment processing API, preventing users from completing purchases. Root cause was an inefficient database query introduced in deployment v2.3.1.",
|
||||
"severity": "sev2",
|
||||
"start_time": "2024-03-15T14:30:00Z",
|
||||
"end_time": "2024-03-15T15:35:00Z",
|
||||
"duration": "1h 5m",
|
||||
"affected_services": ["payment-api", "checkout-service", "subscription-billing"],
|
||||
"customer_impact": "80% of users unable to complete payments or checkout. Approximately 2,400 failed payment attempts during the incident. Users experienced immediate 500 errors when attempting to pay.",
|
||||
"business_impact": "Estimated revenue loss of $45,000 during outage period. No SLA breaches as resolution was within 2-hour window. 12 customer escalations through support channels.",
|
||||
"incident_commander": "Mike Rodriguez",
|
||||
"responders": [
|
||||
"Sarah Chen - On-call Engineer, Primary Responder",
|
||||
"Tom Wilson - Database Team Lead",
|
||||
"Lisa Park - Database Engineer",
|
||||
"Mike Rodriguez - Incident Commander",
|
||||
"David Kumar - DevOps Engineer"
|
||||
],
|
||||
"status": "resolved",
|
||||
"detection_details": {
|
||||
"detection_method": "automated_monitoring",
|
||||
"detection_time": "2024-03-15T14:30:00Z",
|
||||
"alert_source": "Datadog error rate threshold",
|
||||
"time_to_detection": "immediate"
|
||||
},
|
||||
"response_details": {
|
||||
"time_to_response": "5 minutes",
|
||||
"time_to_escalation": "10 minutes",
|
||||
"time_to_resolution": "65 minutes",
|
||||
"war_room_established": "2024-03-15T14:45:00Z",
|
||||
"executives_notified": false,
|
||||
"status_page_updated": true
|
||||
},
|
||||
"technical_details": {
|
||||
"root_cause": "Inefficient database query introduced in deployment v2.3.1 caused each payment validation to take 15 seconds instead of normal 0.1 seconds, exhausting the 200-connection database pool",
|
||||
"affected_regions": ["us-west", "us-east", "eu-west"],
|
||||
"error_metrics": {
|
||||
"peak_error_rate": "45%",
|
||||
"normal_error_rate": "0.1%",
|
||||
"connection_pool_max": 200,
|
||||
"connections_exhausted_at": "100%"
|
||||
},
|
||||
"resolution_method": "rollback",
|
||||
"rollback_target": "v2.2.9",
|
||||
"rollback_duration": "7 minutes"
|
||||
},
|
||||
"communication_log": [
|
||||
{
|
||||
"timestamp": "2024-03-15T14:50:00Z",
|
||||
"type": "status_page",
|
||||
"message": "Investigating payment processing issues",
|
||||
"audience": "customers"
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T15:35:00Z",
|
||||
"type": "status_page",
|
||||
"message": "Payment processing issues resolved",
|
||||
"audience": "customers"
|
||||
}
|
||||
],
|
||||
"lessons_learned_preview": [
|
||||
"Deployment v2.3.1 code review missed performance implications of query change",
|
||||
"Load testing didn't include realistic database query patterns",
|
||||
"Connection pool monitoring could have provided earlier warning",
|
||||
"Rollback procedure worked effectively - 7 minute rollback time"
|
||||
],
|
||||
"preliminary_action_items": [
|
||||
"Fix inefficient query for v2.3.2 deployment",
|
||||
"Add database query performance checks to CI pipeline",
|
||||
"Improve load testing to include database performance scenarios",
|
||||
"Add connection pool utilization alerts"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,263 @@
|
||||
[
|
||||
{
|
||||
"timestamp": "2024-03-15T14:30:00Z",
|
||||
"source": "datadog",
|
||||
"type": "alert",
|
||||
"message": "High error rate detected on payment-api: 45% error rate (threshold: 5%)",
|
||||
"severity": "critical",
|
||||
"actor": "monitoring-system",
|
||||
"metadata": {
|
||||
"alert_id": "ALT-001",
|
||||
"metric_value": "45%",
|
||||
"threshold": "5%"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T14:32:00Z",
|
||||
"source": "pagerduty",
|
||||
"type": "escalation",
|
||||
"message": "Paged on-call engineer Sarah Chen for payment-api alerts",
|
||||
"severity": "high",
|
||||
"actor": "pagerduty-system",
|
||||
"metadata": {
|
||||
"incident_id": "PD-12345",
|
||||
"responder": "sarah.chen@company.com"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T14:35:00Z",
|
||||
"source": "slack",
|
||||
"type": "communication",
|
||||
"message": "Sarah Chen acknowledged the alert and is investigating payment-api issues",
|
||||
"severity": "medium",
|
||||
"actor": "sarah.chen",
|
||||
"metadata": {
|
||||
"channel": "#incidents",
|
||||
"message_id": "1234567890.123456"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T14:38:00Z",
|
||||
"source": "application_logs",
|
||||
"type": "log",
|
||||
"message": "Database connection pool exhausted: 200/200 connections active, unable to acquire new connections",
|
||||
"severity": "critical",
|
||||
"actor": "payment-api",
|
||||
"metadata": {
|
||||
"log_level": "ERROR",
|
||||
"component": "database_pool",
|
||||
"connection_count": 200,
|
||||
"max_connections": 200
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T14:40:00Z",
|
||||
"source": "slack",
|
||||
"type": "escalation",
|
||||
"message": "Sarah Chen: Escalating to incident commander - database connection pool exhausted, need database team",
|
||||
"severity": "high",
|
||||
"actor": "sarah.chen",
|
||||
"metadata": {
|
||||
"channel": "#incidents",
|
||||
"escalation_reason": "database_expertise_needed"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T14:42:00Z",
|
||||
"source": "pagerduty",
|
||||
"type": "escalation",
|
||||
"message": "Incident commander Mike Rodriguez assigned to incident PD-12345",
|
||||
"severity": "high",
|
||||
"actor": "pagerduty-system",
|
||||
"metadata": {
|
||||
"incident_commander": "mike.rodriguez@company.com",
|
||||
"role": "incident_commander"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T14:45:00Z",
|
||||
"source": "slack",
|
||||
"type": "communication",
|
||||
"message": "Mike Rodriguez: War room established in #war-room-payment-api. Engaging database team.",
|
||||
"severity": "high",
|
||||
"actor": "mike.rodriguez",
|
||||
"metadata": {
|
||||
"channel": "#incidents",
|
||||
"war_room": "#war-room-payment-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T14:47:00Z",
|
||||
"source": "pagerduty",
|
||||
"type": "escalation",
|
||||
"message": "Database team engineers paged: Tom Wilson, Lisa Park",
|
||||
"severity": "medium",
|
||||
"actor": "pagerduty-system",
|
||||
"metadata": {
|
||||
"team": "database-team",
|
||||
"responders": ["tom.wilson@company.com", "lisa.park@company.com"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T14:50:00Z",
|
||||
"source": "statuspage",
|
||||
"type": "communication",
|
||||
"message": "Status page updated: Investigating payment processing issues",
|
||||
"severity": "medium",
|
||||
"actor": "mike.rodriguez",
|
||||
"metadata": {
|
||||
"status": "investigating",
|
||||
"affected_systems": ["payment-api"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T14:52:00Z",
|
||||
"source": "slack",
|
||||
"type": "communication",
|
||||
"message": "Tom Wilson: Joining war room. Looking at database metrics now. Seeing unusual query patterns from recent deployment.",
|
||||
"severity": "medium",
|
||||
"actor": "tom.wilson",
|
||||
"metadata": {
|
||||
"channel": "#war-room-payment-api",
|
||||
"investigation_focus": "database_metrics"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T14:55:00Z",
|
||||
"source": "database_monitoring",
|
||||
"type": "log",
|
||||
"message": "Identified slow query introduced in deployment v2.3.1: payment validation taking 15s per request",
|
||||
"severity": "critical",
|
||||
"actor": "database-monitor",
|
||||
"metadata": {
|
||||
"deployment_version": "v2.3.1",
|
||||
"query_time": "15s",
|
||||
"normal_query_time": "0.1s"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T15:00:00Z",
|
||||
"source": "slack",
|
||||
"type": "communication",
|
||||
"message": "Tom Wilson: Root cause identified - inefficient query in v2.3.1 deployment. Recommending immediate rollback.",
|
||||
"severity": "high",
|
||||
"actor": "tom.wilson",
|
||||
"metadata": {
|
||||
"channel": "#war-room-payment-api",
|
||||
"root_cause": "inefficient_query",
|
||||
"recommendation": "rollback"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T15:02:00Z",
|
||||
"source": "slack",
|
||||
"type": "communication",
|
||||
"message": "Mike Rodriguez: Approved rollback to v2.2.9. Sarah initiating rollback procedure.",
|
||||
"severity": "high",
|
||||
"actor": "mike.rodriguez",
|
||||
"metadata": {
|
||||
"channel": "#war-room-payment-api",
|
||||
"decision": "rollback_approved",
|
||||
"target_version": "v2.2.9"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T15:05:00Z",
|
||||
"source": "deployment_system",
|
||||
"type": "action",
|
||||
"message": "Rollback initiated: payment-api v2.3.1 → v2.2.9",
|
||||
"severity": "medium",
|
||||
"actor": "sarah.chen",
|
||||
"metadata": {
|
||||
"from_version": "v2.3.1",
|
||||
"to_version": "v2.2.9",
|
||||
"deployment_type": "rollback"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T15:12:00Z",
|
||||
"source": "deployment_system",
|
||||
"type": "action",
|
||||
"message": "Rollback completed successfully: payment-api now running v2.2.9 across all regions",
|
||||
"severity": "medium",
|
||||
"actor": "deployment-system",
|
||||
"metadata": {
|
||||
"deployment_status": "completed",
|
||||
"regions": ["us-west", "us-east", "eu-west"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T15:15:00Z",
|
||||
"source": "datadog",
|
||||
"type": "log",
|
||||
"message": "Error rate decreasing: payment-api error rate dropped to 8% and continuing to decline",
|
||||
"severity": "medium",
|
||||
"actor": "monitoring-system",
|
||||
"metadata": {
|
||||
"error_rate": "8%",
|
||||
"trend": "decreasing"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T15:18:00Z",
|
||||
"source": "database_monitoring",
|
||||
"type": "log",
|
||||
"message": "Connection pool utilization normalizing: 45/200 connections active",
|
||||
"severity": "low",
|
||||
"actor": "database-monitor",
|
||||
"metadata": {
|
||||
"connection_count": 45,
|
||||
"max_connections": 200,
|
||||
"utilization": "22.5%"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T15:25:00Z",
|
||||
"source": "datadog",
|
||||
"type": "log",
|
||||
"message": "Error rate returned to normal: payment-api error rate now 0.2% (within normal range)",
|
||||
"severity": "low",
|
||||
"actor": "monitoring-system",
|
||||
"metadata": {
|
||||
"error_rate": "0.2%",
|
||||
"status": "normal"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T15:30:00Z",
|
||||
"source": "slack",
|
||||
"type": "communication",
|
||||
"message": "Mike Rodriguez: All metrics returned to normal. Declaring incident resolved. Thanks to all responders.",
|
||||
"severity": "low",
|
||||
"actor": "mike.rodriguez",
|
||||
"metadata": {
|
||||
"channel": "#war-room-payment-api",
|
||||
"status": "resolved"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T15:35:00Z",
|
||||
"source": "statuspage",
|
||||
"type": "communication",
|
||||
"message": "Status page updated: Payment processing issues resolved. All systems operational.",
|
||||
"severity": "low",
|
||||
"actor": "mike.rodriguez",
|
||||
"metadata": {
|
||||
"status": "resolved",
|
||||
"duration": "65 minutes"
|
||||
}
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-15T15:40:00Z",
|
||||
"source": "slack",
|
||||
"type": "communication",
|
||||
"message": "Mike Rodriguez: PIR scheduled for tomorrow 10am. Action item: fix the inefficient query in v2.3.2",
|
||||
"severity": "low",
|
||||
"actor": "mike.rodriguez",
|
||||
"metadata": {
|
||||
"channel": "#incidents",
|
||||
"pir_time": "2024-03-16T10:00:00Z",
|
||||
"action_item": "fix_query_v2.3.2"
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"description": "Users reporting slow page loads on the main website",
|
||||
"service": "web-frontend",
|
||||
"affected_users": "25%",
|
||||
"business_impact": "medium"
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
[
|
||||
{
|
||||
"timestamp": "2024-03-10T09:00:00Z",
|
||||
"source": "monitoring",
|
||||
"message": "High CPU utilization detected on web servers",
|
||||
"severity": "medium",
|
||||
"actor": "system"
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-10T09:05:00Z",
|
||||
"source": "slack",
|
||||
"message": "Engineer investigating high CPU alerts",
|
||||
"severity": "medium",
|
||||
"actor": "john.doe"
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-10T09:15:00Z",
|
||||
"source": "deployment",
|
||||
"message": "Deployed hotfix to reduce CPU usage",
|
||||
"severity": "low",
|
||||
"actor": "john.doe"
|
||||
},
|
||||
{
|
||||
"timestamp": "2024-03-10T09:25:00Z",
|
||||
"source": "monitoring",
|
||||
"message": "CPU utilization returned to normal levels",
|
||||
"severity": "low",
|
||||
"actor": "system"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,44 @@
|
||||
============================================================
|
||||
INCIDENT CLASSIFICATION REPORT
|
||||
============================================================
|
||||
|
||||
CLASSIFICATION:
|
||||
Severity: SEV1
|
||||
Confidence: 100.0%
|
||||
Reasoning: Classified as SEV1 based on: keywords: timeout, 500 error; user impact: 80%
|
||||
Timestamp: 2026-02-16T12:41:46.644096+00:00
|
||||
|
||||
RECOMMENDED RESPONSE:
|
||||
Primary Team: Analytics Team
|
||||
Supporting Teams: SRE, API Team, Backend Engineering, Finance Engineering, Payments Team, DevOps, Compliance Team, Database Team, Platform Team, Data Engineering
|
||||
Response Time: 5 minutes
|
||||
|
||||
INITIAL ACTIONS:
|
||||
1. Establish incident command (Priority 1)
|
||||
Timeout: 5 minutes
|
||||
Page incident commander and establish war room
|
||||
|
||||
2. Create incident ticket (Priority 1)
|
||||
Timeout: 2 minutes
|
||||
Create tracking ticket with all known details
|
||||
|
||||
3. Update status page (Priority 2)
|
||||
Timeout: 15 minutes
|
||||
Post initial status page update acknowledging incident
|
||||
|
||||
4. Notify executives (Priority 2)
|
||||
Timeout: 15 minutes
|
||||
Alert executive team of customer-impacting outage
|
||||
|
||||
5. Engage subject matter experts (Priority 3)
|
||||
Timeout: 10 minutes
|
||||
Page relevant SMEs based on affected systems
|
||||
|
||||
COMMUNICATION:
|
||||
Subject: 🚨 [SEV1] payment-api - Database connection timeouts causing 500 errors fo...
|
||||
Urgency: SEV1
|
||||
Recipients: on-call, engineering-leadership, executives, customer-success
|
||||
Channels: pager, phone, slack, email, status-page
|
||||
Update Frequency: Every 15 minutes
|
||||
|
||||
============================================================
|
||||
@@ -0,0 +1,88 @@
|
||||
# Post-Incident Review: Payment API Database Connection Pool Exhaustion
|
||||
|
||||
## Executive Summary
|
||||
On March 15, 2024, we experienced a sev2 incident affecting ['payment-api', 'checkout-service', 'subscription-billing']. The incident lasted 1h 5m and had the following impact: 80% of users unable to complete payments or checkout. Approximately 2,400 failed payment attempts during the incident. Users experienced immediate 500 errors when attempting to pay. The incident has been resolved and we have identified specific actions to prevent recurrence.
|
||||
|
||||
## Incident Overview
|
||||
- **Incident ID:** INC-2024-0315-001
|
||||
- **Date & Time:** 2024-03-15 14:30:00 UTC
|
||||
- **Duration:** 1h 5m
|
||||
- **Severity:** SEV2
|
||||
- **Status:** Resolved
|
||||
- **Incident Commander:** Mike Rodriguez
|
||||
- **Responders:** Sarah Chen - On-call Engineer, Primary Responder, Tom Wilson - Database Team Lead, Lisa Park - Database Engineer, Mike Rodriguez - Incident Commander, David Kumar - DevOps Engineer
|
||||
|
||||
### Customer Impact
|
||||
80% of users unable to complete payments or checkout. Approximately 2,400 failed payment attempts during the incident. Users experienced immediate 500 errors when attempting to pay.
|
||||
|
||||
### Business Impact
|
||||
Estimated revenue loss of $45,000 during outage period. No SLA breaches as resolution was within 2-hour window. 12 customer escalations through support channels.
|
||||
|
||||
## Timeline
|
||||
No detailed timeline available.
|
||||
|
||||
## Root Cause Analysis
|
||||
### Analysis Method: 5 Whys Analysis
|
||||
|
||||
#### Why Analysis
|
||||
|
||||
**Why 1:** Why did Database connection pool exhaustion caused widespread 500 errors in payment processing API, preventing users from completing purchases. Root cause was an inefficient database query introduced in deployment v2.3.1.?
|
||||
**Answer:** New deployment introduced a regression
|
||||
|
||||
**Why 2:** Why wasn't this detected earlier?
|
||||
**Answer:** Code review process missed the issue
|
||||
|
||||
**Why 3:** Why didn't existing safeguards prevent this?
|
||||
**Answer:** Testing environment didn't match production
|
||||
|
||||
**Why 4:** Why wasn't there a backup mechanism?
|
||||
**Answer:** Further investigation needed
|
||||
|
||||
**Why 5:** Why wasn't this scenario anticipated?
|
||||
**Answer:** Further investigation needed
|
||||
|
||||
|
||||
## What Went Well
|
||||
- The incident was successfully resolved
|
||||
- Incident command was established
|
||||
- Multiple team members collaborated on resolution
|
||||
|
||||
## What Didn't Go Well
|
||||
- Analysis in progress
|
||||
|
||||
## Lessons Learned
|
||||
Lessons learned to be documented following detailed analysis.
|
||||
|
||||
## Action Items
|
||||
Action items to be defined.
|
||||
|
||||
## Follow-up and Prevention
|
||||
### Prevention Measures
|
||||
|
||||
Based on the root cause analysis, the following preventive measures have been identified:
|
||||
|
||||
- Implement comprehensive testing for similar scenarios
|
||||
- Improve monitoring and alerting coverage
|
||||
- Enhance error handling and resilience patterns
|
||||
|
||||
### Follow-up Schedule
|
||||
|
||||
- 1 week: Review action item progress
|
||||
- 1 month: Evaluate effectiveness of implemented changes
|
||||
- 3 months: Conduct follow-up assessment and update preventive measures
|
||||
|
||||
## Appendix
|
||||
### Additional Information
|
||||
|
||||
- Incident ID: INC-2024-0315-001
|
||||
- Severity Classification: sev2
|
||||
- Affected Services: payment-api, checkout-service, subscription-billing
|
||||
|
||||
### References
|
||||
|
||||
- Incident tracking ticket: [Link TBD]
|
||||
- Monitoring dashboards: [Link TBD]
|
||||
- Communication thread: [Link TBD]
|
||||
|
||||
---
|
||||
*Generated on 2026-02-16 by PIR Generator*
|
||||
@@ -0,0 +1,44 @@
|
||||
============================================================
|
||||
INCIDENT CLASSIFICATION REPORT
|
||||
============================================================
|
||||
|
||||
CLASSIFICATION:
|
||||
Severity: SEV2
|
||||
Confidence: 100.0%
|
||||
Reasoning: Classified as SEV2 based on: keywords: slow; user impact: 25%
|
||||
Timestamp: 2026-02-16T12:42:41.889774+00:00
|
||||
|
||||
RECOMMENDED RESPONSE:
|
||||
Primary Team: UX Engineering
|
||||
Supporting Teams: Product Engineering, Frontend Team
|
||||
Response Time: 15 minutes
|
||||
|
||||
INITIAL ACTIONS:
|
||||
1. Assign incident commander (Priority 1)
|
||||
Timeout: 30 minutes
|
||||
Assign IC and establish coordination channel
|
||||
|
||||
2. Create incident tracking (Priority 1)
|
||||
Timeout: 5 minutes
|
||||
Create incident ticket with details and timeline
|
||||
|
||||
3. Assess customer impact (Priority 2)
|
||||
Timeout: 15 minutes
|
||||
Determine scope and severity of user impact
|
||||
|
||||
4. Engage response team (Priority 2)
|
||||
Timeout: 30 minutes
|
||||
Page appropriate technical responders
|
||||
|
||||
5. Begin investigation (Priority 3)
|
||||
Timeout: 15 minutes
|
||||
Start technical analysis and debugging
|
||||
|
||||
COMMUNICATION:
|
||||
Subject: ⚠️ [SEV2] web-frontend - Users reporting slow page loads on the main websit...
|
||||
Urgency: SEV2
|
||||
Recipients: on-call, engineering-leadership, product-team
|
||||
Channels: pager, slack, email
|
||||
Update Frequency: Every 30 minutes
|
||||
|
||||
============================================================
|
||||
@@ -0,0 +1,110 @@
|
||||
================================================================================
|
||||
INCIDENT TIMELINE RECONSTRUCTION
|
||||
================================================================================
|
||||
|
||||
OVERVIEW:
|
||||
Time Range: 2024-03-15T14:30:00+00:00 to 2024-03-15T15:40:00+00:00
|
||||
Total Duration: 70 minutes
|
||||
Total Events: 21
|
||||
Phases Detected: 12
|
||||
|
||||
PHASES:
|
||||
DETECTION:
|
||||
Start: 2024-03-15T14:30:00+00:00
|
||||
Duration: 0.0 minutes
|
||||
Events: 1
|
||||
Description: Initial detection of the incident through monitoring or observation
|
||||
|
||||
ESCALATION:
|
||||
Start: 2024-03-15T14:32:00+00:00
|
||||
Duration: 0.0 minutes
|
||||
Events: 1
|
||||
Description: Escalation to additional resources or higher severity response
|
||||
|
||||
TRIAGE:
|
||||
Start: 2024-03-15T14:35:00+00:00
|
||||
Duration: 0.0 minutes
|
||||
Events: 1
|
||||
Description: Assessment and initial investigation of the incident
|
||||
|
||||
ESCALATION:
|
||||
Start: 2024-03-15T14:38:00+00:00
|
||||
Duration: 9.0 minutes
|
||||
Events: 5
|
||||
Description: Escalation to additional resources or higher severity response
|
||||
|
||||
TRIAGE:
|
||||
Start: 2024-03-15T14:50:00+00:00
|
||||
Duration: 0.0 minutes
|
||||
Events: 1
|
||||
Description: Assessment and initial investigation of the incident
|
||||
|
||||
ESCALATION:
|
||||
Start: 2024-03-15T14:52:00+00:00
|
||||
Duration: 10.0 minutes
|
||||
Events: 4
|
||||
Description: Escalation to additional resources or higher severity response
|
||||
|
||||
TRIAGE:
|
||||
Start: 2024-03-15T15:05:00+00:00
|
||||
Duration: 7.0 minutes
|
||||
Events: 2
|
||||
Description: Assessment and initial investigation of the incident
|
||||
|
||||
DETECTION:
|
||||
Start: 2024-03-15T15:15:00+00:00
|
||||
Duration: 0.0 minutes
|
||||
Events: 1
|
||||
Description: Initial detection of the incident through monitoring or observation
|
||||
|
||||
RESOLUTION:
|
||||
Start: 2024-03-15T15:18:00+00:00
|
||||
Duration: 0.0 minutes
|
||||
Events: 1
|
||||
Description: Confirmation that the incident has been resolved
|
||||
|
||||
DETECTION:
|
||||
Start: 2024-03-15T15:25:00+00:00
|
||||
Duration: 0.0 minutes
|
||||
Events: 1
|
||||
Description: Initial detection of the incident through monitoring or observation
|
||||
|
||||
RESOLUTION:
|
||||
Start: 2024-03-15T15:30:00+00:00
|
||||
Duration: 5.0 minutes
|
||||
Events: 2
|
||||
Description: Confirmation that the incident has been resolved
|
||||
|
||||
TRIAGE:
|
||||
Start: 2024-03-15T15:40:00+00:00
|
||||
Duration: 0.0 minutes
|
||||
Events: 1
|
||||
Description: Assessment and initial investigation of the incident
|
||||
|
||||
KEY METRICS:
|
||||
Time to Mitigation: 0 minutes
|
||||
Time to Resolution: 48.0 minutes
|
||||
Events per Hour: 18.0
|
||||
Unique Sources: 7
|
||||
|
||||
INCIDENT NARRATIVE:
|
||||
Incident Timeline Summary:
|
||||
The incident began at 2024-03-15 14:30:00 UTC and concluded at 2024-03-15 15:40:00 UTC, lasting approximately 70 minutes.
|
||||
|
||||
The incident progressed through 12 distinct phases: detection, escalation, triage, escalation, triage, escalation, triage, detection, resolution, detection, resolution, triage.
|
||||
|
||||
Key milestones:
|
||||
- Detection: 14:30 (0 min)
|
||||
- Escalation: 14:32 (0 min)
|
||||
- Triage: 14:35 (0 min)
|
||||
- Escalation: 14:38 (9 min)
|
||||
- Triage: 14:50 (0 min)
|
||||
- Escalation: 14:52 (10 min)
|
||||
- Triage: 15:05 (7 min)
|
||||
- Detection: 15:15 (0 min)
|
||||
- Resolution: 15:18 (0 min)
|
||||
- Detection: 15:25 (0 min)
|
||||
- Resolution: 15:30 (5 min)
|
||||
- Triage: 15:40 (0 min)
|
||||
|
||||
================================================================================
|
||||
@@ -0,0 +1,591 @@
|
||||
# Incident Communication Templates
|
||||
|
||||
## Overview
|
||||
|
||||
This document provides standardized communication templates for incident response. These templates ensure consistent, clear communication across different severity levels and stakeholder groups.
|
||||
|
||||
## Template Usage Guidelines
|
||||
|
||||
### General Principles
|
||||
1. **Be Clear and Concise** - Use simple language, avoid jargon
|
||||
2. **Be Factual** - Only state what is known, avoid speculation
|
||||
3. **Be Timely** - Send updates at committed intervals
|
||||
4. **Be Actionable** - Include next steps and expected timelines
|
||||
5. **Be Accountable** - Include contact information for follow-up
|
||||
|
||||
### Template Selection
|
||||
- Choose templates based on incident severity and audience
|
||||
- Customize templates with specific incident details
|
||||
- Always include next update time and contact information
|
||||
- Escalate template types as severity increases
|
||||
|
||||
---
|
||||
|
||||
## SEV1 Templates
|
||||
|
||||
### Initial Alert - Internal Teams
|
||||
|
||||
**Subject:** 🚨 [SEV1] CRITICAL: {Service} Complete Outage - Immediate Response Required
|
||||
|
||||
```
|
||||
CRITICAL INCIDENT ALERT - IMMEDIATE ATTENTION REQUIRED
|
||||
|
||||
Incident Summary:
|
||||
- Service: {Service Name}
|
||||
- Status: Complete Outage
|
||||
- Start Time: {Timestamp}
|
||||
- Customer Impact: {Impact Description}
|
||||
- Estimated Affected Users: {Number/Percentage}
|
||||
|
||||
Immediate Actions Needed:
|
||||
✓ Incident Commander: {Name} - ASSIGNED
|
||||
✓ War Room: {Bridge/Chat Link} - JOIN NOW
|
||||
✓ On-Call Response: {Team} - PAGED
|
||||
⏳ Executive Notification: In progress
|
||||
⏳ Status Page Update: Within 15 minutes
|
||||
|
||||
Current Situation:
|
||||
{Brief description of what we know}
|
||||
|
||||
What We're Doing:
|
||||
{Immediate response actions being taken}
|
||||
|
||||
Next Update: {Timestamp - 15 minutes from now}
|
||||
|
||||
Incident Commander: {Name}
|
||||
Contact: {Phone/Slack}
|
||||
|
||||
THIS IS A CUSTOMER-IMPACTING INCIDENT REQUIRING IMMEDIATE ATTENTION
|
||||
```
|
||||
|
||||
### Executive Notification - SEV1
|
||||
|
||||
**Subject:** 🚨 URGENT: Customer-Impacting Outage - {Service}
|
||||
|
||||
```
|
||||
EXECUTIVE ALERT: Critical customer-facing incident
|
||||
|
||||
Service: {Service Name}
|
||||
Impact: {Customer impact description}
|
||||
Duration: {Current duration} (started {start time})
|
||||
Business Impact: {Revenue/SLA/compliance implications}
|
||||
|
||||
Customer Impact Summary:
|
||||
- Affected Users: {Number/percentage}
|
||||
- Revenue Impact: {$ amount if known}
|
||||
- SLA Status: {Breach status}
|
||||
- Customer Escalations: {Number if any}
|
||||
|
||||
Response Status:
|
||||
- Incident Commander: {Name} ({contact})
|
||||
- Response Team Size: {Number of engineers}
|
||||
- Root Cause: {If known, otherwise "Under investigation"}
|
||||
- ETA to Resolution: {If known, otherwise "Investigating"}
|
||||
|
||||
Executive Actions Required:
|
||||
- [ ] Customer communication approval needed
|
||||
- [ ] Legal/compliance notification: {If applicable}
|
||||
- [ ] PR/Media response preparation: {If needed}
|
||||
- [ ] Resource allocation decisions: {If escalation needed}
|
||||
|
||||
War Room: {Link}
|
||||
Next Update: {15 minutes from now}
|
||||
|
||||
This incident meets SEV1 criteria and requires executive oversight.
|
||||
|
||||
{Incident Commander contact information}
|
||||
```
|
||||
|
||||
### Customer Communication - SEV1
|
||||
|
||||
**Subject:** Service Disruption - Immediate Action Being Taken
|
||||
|
||||
```
|
||||
We are currently experiencing a service disruption affecting {service description}.
|
||||
|
||||
What's Happening:
|
||||
{Clear, customer-friendly description of the issue}
|
||||
|
||||
Impact:
|
||||
{What customers are experiencing - be specific}
|
||||
|
||||
What We're Doing:
|
||||
We detected this issue at {time} and immediately mobilized our engineering team. We are actively working to resolve this issue and will provide updates every 15 minutes.
|
||||
|
||||
Current Actions:
|
||||
• {Action 1 - customer-friendly description}
|
||||
• {Action 2 - customer-friendly description}
|
||||
• {Action 3 - customer-friendly description}
|
||||
|
||||
Workaround:
|
||||
{If available, provide clear steps}
|
||||
{If not available: "We are working on alternative solutions and will share them as soon as available."}
|
||||
|
||||
Next Update: {Timestamp}
|
||||
Status Page: {Link}
|
||||
Support: {Contact information if different from usual}
|
||||
|
||||
We sincerely apologize for the inconvenience and are committed to resolving this as quickly as possible.
|
||||
|
||||
{Company Name} Team
|
||||
```
|
||||
|
||||
### Status Page Update - SEV1
|
||||
|
||||
**Status:** Major Outage
|
||||
|
||||
```
|
||||
{Timestamp} - Investigating
|
||||
|
||||
We are currently investigating reports of {service} being unavailable. Our team has been alerted and is actively investigating the cause.
|
||||
|
||||
Affected Services: {List of affected services}
|
||||
Impact: {Customer-facing impact description}
|
||||
|
||||
We will provide an update within 15 minutes.
|
||||
```
|
||||
|
||||
```
|
||||
{Timestamp} - Identified
|
||||
|
||||
We have identified the cause of the {service} outage. Our engineering team is implementing a fix.
|
||||
|
||||
Root Cause: {Brief, customer-friendly explanation}
|
||||
Expected Resolution: {Timeline if known}
|
||||
|
||||
Next update in 15 minutes.
|
||||
```
|
||||
|
||||
```
|
||||
{Timestamp} - Monitoring
|
||||
|
||||
The fix has been implemented and we are monitoring the service recovery.
|
||||
|
||||
Current Status: {Recovery progress}
|
||||
Next Steps: {What we're monitoring}
|
||||
|
||||
We expect full service restoration within {timeframe}.
|
||||
```
|
||||
|
||||
```
|
||||
{Timestamp} - Resolved
|
||||
|
||||
{Service} is now fully operational. We have confirmed that all functionality is working as expected.
|
||||
|
||||
Total Duration: {Duration}
|
||||
Root Cause: {Brief summary}
|
||||
|
||||
We apologize for the inconvenience. A full post-incident review will be conducted and shared within 24 hours.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SEV2 Templates
|
||||
|
||||
### Team Notification - SEV2
|
||||
|
||||
**Subject:** ⚠️ [SEV2] {Service} Performance Issues - Response Team Mobilizing
|
||||
|
||||
```
|
||||
SEV2 INCIDENT: Performance degradation requiring active response
|
||||
|
||||
Incident Details:
|
||||
- Service: {Service Name}
|
||||
- Issue: {Description of performance issue}
|
||||
- Start Time: {Timestamp}
|
||||
- Affected Users: {Percentage/description}
|
||||
- Business Impact: {Impact on business operations}
|
||||
|
||||
Current Status:
|
||||
{What we know about the issue}
|
||||
|
||||
Response Team:
|
||||
- Incident Commander: {Name} ({contact})
|
||||
- Primary Responder: {Name} ({team})
|
||||
- Supporting Teams: {List of engaged teams}
|
||||
|
||||
Immediate Actions:
|
||||
✓ {Action 1 - completed}
|
||||
⏳ {Action 2 - in progress}
|
||||
⏳ {Action 3 - next step}
|
||||
|
||||
Metrics:
|
||||
- Error Rate: {Current vs normal}
|
||||
- Response Time: {Current vs normal}
|
||||
- Throughput: {Current vs normal}
|
||||
|
||||
Communication Plan:
|
||||
- Internal Updates: Every 30 minutes
|
||||
- Stakeholder Notification: {If needed}
|
||||
- Status Page Update: {Planned/not needed}
|
||||
|
||||
Coordination Channel: {Slack channel}
|
||||
Next Update: {30 minutes from now}
|
||||
|
||||
Incident Commander: {Name} | {Contact}
|
||||
```
|
||||
|
||||
### Stakeholder Update - SEV2
|
||||
|
||||
**Subject:** [SEV2] Service Performance Update - {Service}
|
||||
|
||||
```
|
||||
Service Performance Incident Update
|
||||
|
||||
Service: {Service Name}
|
||||
Duration: {Current duration}
|
||||
Impact: {Description of user impact}
|
||||
|
||||
Current Status:
|
||||
{Brief status of the incident and response efforts}
|
||||
|
||||
What We Know:
|
||||
• {Key finding 1}
|
||||
• {Key finding 2}
|
||||
• {Key finding 3}
|
||||
|
||||
What We're Doing:
|
||||
• {Response action 1}
|
||||
• {Response action 2}
|
||||
• {Monitoring/verification steps}
|
||||
|
||||
Customer Impact:
|
||||
{Realistic assessment of what users are experiencing}
|
||||
|
||||
Workaround:
|
||||
{If available, provide steps}
|
||||
|
||||
Expected Resolution:
|
||||
{Timeline if known, otherwise "Continuing investigation"}
|
||||
|
||||
Next Update: {30 minutes}
|
||||
Contact: {Incident Commander information}
|
||||
|
||||
This incident is being actively managed and does not currently require escalation.
|
||||
```
|
||||
|
||||
### Customer Communication - SEV2 (Optional)
|
||||
|
||||
**Subject:** Temporary Service Performance Issues
|
||||
|
||||
```
|
||||
We are currently experiencing performance issues with {service name} that may affect your experience.
|
||||
|
||||
What You Might Notice:
|
||||
{Specific symptoms users might experience}
|
||||
|
||||
What We're Doing:
|
||||
Our team identified this issue at {time} and is actively working on a resolution. We expect to have this resolved within {timeframe}.
|
||||
|
||||
Workaround:
|
||||
{If applicable, provide simple workaround steps}
|
||||
|
||||
We will update our status page at {link} with progress information.
|
||||
|
||||
Thank you for your patience as we work to resolve this issue quickly.
|
||||
|
||||
{Company Name} Support Team
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SEV3 Templates
|
||||
|
||||
### Team Assignment - SEV3
|
||||
|
||||
**Subject:** [SEV3] Issue Assignment - {Component} Issue
|
||||
|
||||
```
|
||||
SEV3 Issue Assignment
|
||||
|
||||
Service/Component: {Affected component}
|
||||
Issue: {Description}
|
||||
Reported: {Timestamp}
|
||||
Reporter: {Person/system that reported}
|
||||
|
||||
Issue Details:
|
||||
{Detailed description of the problem}
|
||||
|
||||
Impact Assessment:
|
||||
- Affected Users: {Scope}
|
||||
- Business Impact: {Assessment}
|
||||
- Urgency: {Business hours response appropriate}
|
||||
|
||||
Assignment:
|
||||
- Primary: {Engineer name}
|
||||
- Team: {Responsible team}
|
||||
- Expected Response: {Within 2-4 hours}
|
||||
|
||||
Investigation Plan:
|
||||
1. {Investigation step 1}
|
||||
2. {Investigation step 2}
|
||||
3. {Communication checkpoint}
|
||||
|
||||
Workaround:
|
||||
{If known, otherwise "Investigating alternatives"}
|
||||
|
||||
This issue will be tracked in {ticket system} as {ticket number}.
|
||||
|
||||
Team Lead: {Name} | {Contact}
|
||||
```
|
||||
|
||||
### Status Update - SEV3
|
||||
|
||||
**Subject:** [SEV3] Progress Update - {Component}
|
||||
|
||||
```
|
||||
SEV3 Issue Progress Update
|
||||
|
||||
Issue: {Brief description}
|
||||
Assigned to: {Engineer/Team}
|
||||
Investigation Status: {Current progress}
|
||||
|
||||
Findings So Far:
|
||||
{What has been discovered during investigation}
|
||||
|
||||
Next Steps:
|
||||
{Planned actions and timeline}
|
||||
|
||||
Impact Update:
|
||||
{Any changes to scope or urgency}
|
||||
|
||||
Expected Resolution:
|
||||
{Timeline if known}
|
||||
|
||||
This issue continues to be tracked as SEV3 with no escalation required.
|
||||
|
||||
Contact: {Assigned engineer} | {Team lead}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SEV4 Templates
|
||||
|
||||
### Issue Documentation - SEV4
|
||||
|
||||
**Subject:** [SEV4] Issue Documented - {Description}
|
||||
|
||||
```
|
||||
SEV4 Issue Logged
|
||||
|
||||
Description: {Clear description of the issue}
|
||||
Reporter: {Name/system}
|
||||
Date: {Date reported}
|
||||
|
||||
Impact:
|
||||
{Minimal impact description}
|
||||
|
||||
Priority Assessment:
|
||||
This issue has been classified as SEV4 and will be addressed in the normal development cycle.
|
||||
|
||||
Assignment:
|
||||
- Team: {Responsible team}
|
||||
- Sprint: {Target sprint}
|
||||
- Estimated Effort: {Story points/hours}
|
||||
|
||||
This issue is tracked as {ticket number} in {system}.
|
||||
|
||||
Product Owner: {Name}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Escalation Templates
|
||||
|
||||
### Severity Escalation
|
||||
|
||||
**Subject:** ESCALATION: {Original Severity} → {New Severity} - {Service}
|
||||
|
||||
```
|
||||
SEVERITY ESCALATION NOTIFICATION
|
||||
|
||||
Original Classification: {Original severity}
|
||||
New Classification: {New severity}
|
||||
Escalation Time: {Timestamp}
|
||||
Escalated By: {Name and role}
|
||||
|
||||
Escalation Reasons:
|
||||
• {Reason 1 - scope expansion/duration/impact}
|
||||
• {Reason 2}
|
||||
• {Reason 3}
|
||||
|
||||
Updated Impact:
|
||||
{New assessment of customer/business impact}
|
||||
|
||||
Updated Response Requirements:
|
||||
{New response team, communication frequency, etc.}
|
||||
|
||||
Previous Response Actions:
|
||||
{Summary of actions taken under previous severity}
|
||||
|
||||
New Incident Commander: {If changed}
|
||||
Updated Communication Plan: {New frequency/recipients}
|
||||
|
||||
All stakeholders should adjust response according to {new severity} protocols.
|
||||
|
||||
Incident Commander: {Name} | {Contact}
|
||||
```
|
||||
|
||||
### Management Escalation
|
||||
|
||||
**Subject:** MANAGEMENT ESCALATION: Extended {Severity} Incident - {Service}
|
||||
|
||||
```
|
||||
Management Escalation Required
|
||||
|
||||
Incident: {Service} {brief description}
|
||||
Original Severity: {Severity}
|
||||
Duration: {Current duration}
|
||||
Escalation Trigger: {Duration threshold/scope change/customer escalation}
|
||||
|
||||
Current Status:
|
||||
{Brief status of incident response}
|
||||
|
||||
Challenges Encountered:
|
||||
• {Challenge 1}
|
||||
• {Challenge 2}
|
||||
• {Resource/expertise needs}
|
||||
|
||||
Business Impact:
|
||||
{Updated assessment of business implications}
|
||||
|
||||
Management Decision Required:
|
||||
• {Decision 1 - resource allocation/external expertise/communication}
|
||||
• {Decision 2}
|
||||
|
||||
Recommended Actions:
|
||||
{Incident Commander's recommendations}
|
||||
|
||||
This escalation follows standard procedures for {trigger type}.
|
||||
|
||||
Incident Commander: {Name}
|
||||
Contact: {Phone/Slack}
|
||||
War Room: {Link}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Resolution Templates
|
||||
|
||||
### Resolution Confirmation - All Severities
|
||||
|
||||
**Subject:** RESOLVED: [{Severity}] {Service} Incident - {Brief Description}
|
||||
|
||||
```
|
||||
INCIDENT RESOLVED
|
||||
|
||||
Service: {Service Name}
|
||||
Issue: {Brief description}
|
||||
Duration: {Total duration}
|
||||
Resolution Time: {Timestamp}
|
||||
|
||||
Resolution Summary:
|
||||
{Brief description of how the issue was resolved}
|
||||
|
||||
Root Cause:
|
||||
{Brief explanation - detailed PIR to follow}
|
||||
|
||||
Impact Summary:
|
||||
- Users Affected: {Final count/percentage}
|
||||
- Business Impact: {Final assessment}
|
||||
- Services Affected: {List}
|
||||
|
||||
Resolution Actions Taken:
|
||||
• {Action 1}
|
||||
• {Action 2}
|
||||
• {Verification steps}
|
||||
|
||||
Monitoring:
|
||||
We will continue monitoring {service} for {duration} to ensure stability.
|
||||
|
||||
Next Steps:
|
||||
• Post-incident review scheduled for {date}
|
||||
• Action items to be tracked in {system}
|
||||
• Follow-up communication: {If needed}
|
||||
|
||||
Thank you to everyone who participated in the incident response.
|
||||
|
||||
Incident Commander: {Name}
|
||||
```
|
||||
|
||||
### Customer Resolution Communication
|
||||
|
||||
**Subject:** Service Restored - Thank You for Your Patience
|
||||
|
||||
```
|
||||
Service Update: Issue Resolved
|
||||
|
||||
We're pleased to report that the {service} issues have been fully resolved as of {timestamp}.
|
||||
|
||||
What Was Fixed:
|
||||
{Customer-friendly explanation of the resolution}
|
||||
|
||||
Duration:
|
||||
The issue lasted {duration} from {start time} to {end time}.
|
||||
|
||||
What We Learned:
|
||||
{Brief, high-level takeaway}
|
||||
|
||||
Our Commitment:
|
||||
We are conducting a thorough review of this incident and will implement improvements to prevent similar issues in the future. A summary of our findings and improvements will be shared {timeframe}.
|
||||
|
||||
We sincerely apologize for any inconvenience this may have caused and appreciate your patience while we worked to resolve the issue.
|
||||
|
||||
If you continue to experience any problems, please contact our support team at {contact information}.
|
||||
|
||||
Thank you,
|
||||
{Company Name} Team
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Template Customization Guidelines
|
||||
|
||||
### Placeholders to Always Replace
|
||||
- `{Service}` / `{Service Name}` - Specific service or component
|
||||
- `{Timestamp}` - Specific date/time in consistent format
|
||||
- `{Name}` / `{Contact}` - Actual names and contact information
|
||||
- `{Duration}` - Actual time durations
|
||||
- `{Link}` - Real URLs to war rooms, status pages, etc.
|
||||
|
||||
### Language Guidelines
|
||||
- Use active voice ("We are investigating" not "The issue is being investigated")
|
||||
- Be specific about timelines ("within 30 minutes" not "soon")
|
||||
- Avoid technical jargon in customer communications
|
||||
- Include empathy in customer-facing messages
|
||||
- Use consistent terminology throughout incident lifecycle
|
||||
|
||||
### Timing Guidelines
|
||||
| Severity | Initial Notification | Update Frequency | Resolution Notification |
|
||||
|----------|---------------------|------------------|------------------------|
|
||||
| SEV1 | Immediate (< 5 min) | Every 15 minutes | Immediate |
|
||||
| SEV2 | Within 15 minutes | Every 30 minutes | Within 15 minutes |
|
||||
| SEV3 | Within 2 hours | At milestones | Within 1 hour |
|
||||
| SEV4 | Within 1 business day | Weekly | When resolved |
|
||||
|
||||
### Audience-Specific Considerations
|
||||
|
||||
#### Engineering Teams
|
||||
- Include technical details
|
||||
- Provide specific metrics and logs
|
||||
- Include coordination channels
|
||||
- List specific actions and owners
|
||||
|
||||
#### Executive/Business
|
||||
- Focus on business impact
|
||||
- Include customer and revenue implications
|
||||
- Provide clear timeline and resource needs
|
||||
- Highlight any external factors (PR, legal, compliance)
|
||||
|
||||
#### Customers
|
||||
- Use plain language
|
||||
- Focus on customer impact and workarounds
|
||||
- Provide realistic timelines
|
||||
- Include support contact information
|
||||
- Show empathy and accountability
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** February 2026
|
||||
**Next Review:** May 2026
|
||||
**Owner:** Incident Management Team
|
||||
@@ -0,0 +1,292 @@
|
||||
# Incident Severity Classification Matrix
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the severity classification system used for incident response. The classification determines response requirements, escalation paths, and communication frequency.
|
||||
|
||||
## Severity Levels
|
||||
|
||||
### SEV1 - Critical Outage
|
||||
|
||||
**Definition:** Complete service failure affecting all users or critical business functions
|
||||
|
||||
#### Impact Criteria
|
||||
- Customer-facing services completely unavailable
|
||||
- Data loss or corruption affecting users
|
||||
- Security breaches with customer data exposure
|
||||
- Revenue-generating systems down
|
||||
- SLA violations with financial penalties
|
||||
- > 75% of users affected
|
||||
|
||||
#### Response Requirements
|
||||
| Metric | Requirement |
|
||||
|--------|-------------|
|
||||
| **Response Time** | Immediate (0-5 minutes) |
|
||||
| **Incident Commander** | Assigned within 5 minutes |
|
||||
| **War Room** | Established within 10 minutes |
|
||||
| **Executive Notification** | Within 15 minutes |
|
||||
| **Public Status Page** | Updated within 15 minutes |
|
||||
| **Customer Communication** | Within 30 minutes |
|
||||
|
||||
#### Escalation Path
|
||||
1. **Immediate**: On-call Engineer → Incident Commander
|
||||
2. **15 minutes**: VP Engineering + Customer Success VP
|
||||
3. **30 minutes**: CTO
|
||||
4. **60 minutes**: CEO + Full Executive Team
|
||||
|
||||
#### Communication Requirements
|
||||
- **Frequency**: Every 15 minutes until resolution
|
||||
- **Channels**: PagerDuty, Phone, Slack, Email, Status Page
|
||||
- **Recipients**: All engineering, executives, customer success
|
||||
- **Template**: SEV1 Executive Alert Template
|
||||
|
||||
---
|
||||
|
||||
### SEV2 - Major Impact
|
||||
|
||||
**Definition:** Significant degradation affecting subset of users or non-critical functions
|
||||
|
||||
#### Impact Criteria
|
||||
- Partial service degradation (25-75% of users affected)
|
||||
- Performance issues causing user frustration
|
||||
- Non-critical features unavailable
|
||||
- Internal tools impacting productivity
|
||||
- Data inconsistencies not affecting user experience
|
||||
- API errors affecting integrations
|
||||
|
||||
#### Response Requirements
|
||||
| Metric | Requirement |
|
||||
|--------|-------------|
|
||||
| **Response Time** | 15 minutes |
|
||||
| **Incident Commander** | Assigned within 30 minutes |
|
||||
| **Status Page Update** | Within 30 minutes |
|
||||
| **Stakeholder Notification** | Within 1 hour |
|
||||
| **Team Assembly** | Within 30 minutes |
|
||||
|
||||
#### Escalation Path
|
||||
1. **Immediate**: On-call Engineer → Team Lead
|
||||
2. **30 minutes**: Engineering Manager
|
||||
3. **2 hours**: VP Engineering
|
||||
4. **4 hours**: CTO (if unresolved)
|
||||
|
||||
#### Communication Requirements
|
||||
- **Frequency**: Every 30 minutes during active response
|
||||
- **Channels**: PagerDuty, Slack, Email
|
||||
- **Recipients**: Engineering team, product team, relevant stakeholders
|
||||
- **Template**: SEV2 Major Impact Template
|
||||
|
||||
---
|
||||
|
||||
### SEV3 - Minor Impact
|
||||
|
||||
**Definition:** Limited impact with workarounds available
|
||||
|
||||
#### Impact Criteria
|
||||
- Single feature or component affected
|
||||
- < 25% of users impacted
|
||||
- Workarounds available
|
||||
- Performance degradation not significantly impacting UX
|
||||
- Non-urgent monitoring alerts
|
||||
- Development/test environment issues
|
||||
|
||||
#### Response Requirements
|
||||
| Metric | Requirement |
|
||||
|--------|-------------|
|
||||
| **Response Time** | 2 hours (business hours) |
|
||||
| **After Hours Response** | Next business day |
|
||||
| **Team Assignment** | Within 4 hours |
|
||||
| **Status Page Update** | Optional |
|
||||
| **Internal Notification** | Within 2 hours |
|
||||
|
||||
#### Escalation Path
|
||||
1. **Immediate**: Assigned Engineer
|
||||
2. **4 hours**: Team Lead
|
||||
3. **1 business day**: Engineering Manager (if needed)
|
||||
|
||||
#### Communication Requirements
|
||||
- **Frequency**: At key milestones only
|
||||
- **Channels**: Slack, Email
|
||||
- **Recipients**: Assigned team, team lead
|
||||
- **Template**: SEV3 Minor Impact Template
|
||||
|
||||
---
|
||||
|
||||
### SEV4 - Low Impact
|
||||
|
||||
**Definition:** Minimal impact, cosmetic issues, or planned maintenance
|
||||
|
||||
#### Impact Criteria
|
||||
- Cosmetic bugs
|
||||
- Documentation issues
|
||||
- Logging or monitoring gaps
|
||||
- Performance issues with no user impact
|
||||
- Development/test environment issues
|
||||
- Feature requests or enhancements
|
||||
|
||||
#### Response Requirements
|
||||
| Metric | Requirement |
|
||||
|--------|-------------|
|
||||
| **Response Time** | 1-2 business days |
|
||||
| **Assignment** | Next sprint planning |
|
||||
| **Tracking** | Standard ticket system |
|
||||
| **Escalation** | None required |
|
||||
|
||||
#### Communication Requirements
|
||||
- **Frequency**: Standard development cycle updates
|
||||
- **Channels**: Ticket system
|
||||
- **Recipients**: Product owner, assigned developer
|
||||
- **Template**: Standard issue template
|
||||
|
||||
## Classification Guidelines
|
||||
|
||||
### User Impact Assessment
|
||||
|
||||
| Impact Scope | Description | Typical Severity |
|
||||
|--------------|-------------|------------------|
|
||||
| **All Users** | 100% of users affected | SEV1 |
|
||||
| **Major Subset** | 50-75% of users affected | SEV1/SEV2 |
|
||||
| **Significant Subset** | 25-50% of users affected | SEV2 |
|
||||
| **Limited Users** | 5-25% of users affected | SEV2/SEV3 |
|
||||
| **Few Users** | < 5% of users affected | SEV3/SEV4 |
|
||||
| **No User Impact** | Internal only | SEV4 |
|
||||
|
||||
### Business Impact Assessment
|
||||
|
||||
| Business Impact | Description | Severity Boost |
|
||||
|-----------------|-------------|----------------|
|
||||
| **Revenue Loss** | Direct revenue impact | +1 severity level |
|
||||
| **SLA Breach** | Contract violations | +1 severity level |
|
||||
| **Regulatory** | Compliance implications | +1 severity level |
|
||||
| **Brand Damage** | Public-facing issues | +1 severity level |
|
||||
| **Security** | Data or system security | +2 severity levels |
|
||||
|
||||
### Duration Considerations
|
||||
|
||||
| Duration | Impact on Classification |
|
||||
|----------|--------------------------|
|
||||
| **< 15 minutes** | May reduce severity by 1 level |
|
||||
| **15-60 minutes** | Standard classification |
|
||||
| **1-4 hours** | May increase severity by 1 level |
|
||||
| **> 4 hours** | Significant severity increase |
|
||||
|
||||
## Decision Tree
|
||||
|
||||
```
|
||||
1. Is this a security incident with data exposure?
|
||||
→ YES: SEV1 (regardless of user count)
|
||||
→ NO: Continue to step 2
|
||||
|
||||
2. Are revenue-generating services completely down?
|
||||
→ YES: SEV1
|
||||
→ NO: Continue to step 3
|
||||
|
||||
3. What percentage of users are affected?
|
||||
→ > 75%: SEV1
|
||||
→ 25-75%: SEV2
|
||||
→ 5-25%: SEV3
|
||||
→ < 5%: SEV4
|
||||
|
||||
4. Apply business impact modifiers
|
||||
5. Consider duration factors
|
||||
6. When in doubt, err on higher severity
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### SEV1 Examples
|
||||
- Payment processing system completely down
|
||||
- All user authentication failing
|
||||
- Database corruption causing data loss
|
||||
- Security breach with customer data exposed
|
||||
- Website returning 500 errors for all users
|
||||
|
||||
### SEV2 Examples
|
||||
- Payment processing slow (30-second delays)
|
||||
- Search functionality returning incomplete results
|
||||
- API rate limits causing partner integration issues
|
||||
- Dashboard displaying stale data (> 1 hour old)
|
||||
- Mobile app crashing for 40% of users
|
||||
|
||||
### SEV3 Examples
|
||||
- Single feature in admin panel not working
|
||||
- Email notifications delayed by 1 hour
|
||||
- Non-critical API endpoint returning errors
|
||||
- Cosmetic UI bug in settings page
|
||||
- Development environment deployment failing
|
||||
|
||||
### SEV4 Examples
|
||||
- Typo in help documentation
|
||||
- Log format change needed for analysis
|
||||
- Non-critical performance optimization
|
||||
- Internal tool enhancement request
|
||||
- Test data cleanup needed
|
||||
|
||||
## Escalation Triggers
|
||||
|
||||
### Automatic Escalation
|
||||
- SEV1 incidents automatically escalate every 30 minutes if unresolved
|
||||
- SEV2 incidents escalate after 2 hours without significant progress
|
||||
- Any incident with expanding scope increases severity
|
||||
- Customer escalation to support triggers severity review
|
||||
|
||||
### Manual Escalation
|
||||
- Incident Commander can escalate at any time
|
||||
- Technical leads can request escalation
|
||||
- Business stakeholders can request severity review
|
||||
- External factors (media attention, regulatory) trigger escalation
|
||||
|
||||
## Communication Templates
|
||||
|
||||
### SEV1 Executive Alert
|
||||
```
|
||||
Subject: 🚨 CRITICAL INCIDENT - [Service] Complete Outage
|
||||
|
||||
URGENT: Customer-facing service outage requiring immediate attention
|
||||
|
||||
Service: [Service Name]
|
||||
Start Time: [Timestamp]
|
||||
Impact: [Description of customer impact]
|
||||
Estimated Affected Users: [Number/Percentage]
|
||||
Business Impact: [Revenue/SLA/Brand implications]
|
||||
|
||||
Incident Commander: [Name] ([Contact])
|
||||
Response Team: [Team members engaged]
|
||||
|
||||
Current Status: [Brief status update]
|
||||
Next Update: [Timestamp - 15 minutes from now]
|
||||
War Room: [Bridge/Chat link]
|
||||
|
||||
This is a customer-impacting incident requiring executive awareness.
|
||||
```
|
||||
|
||||
### SEV2 Major Impact
|
||||
```
|
||||
Subject: ⚠️ [SEV2] [Service] - Major Performance Impact
|
||||
|
||||
Major service degradation affecting user experience
|
||||
|
||||
Service: [Service Name]
|
||||
Start Time: [Timestamp]
|
||||
Impact: [Description of user impact]
|
||||
Scope: [Affected functionality/users]
|
||||
|
||||
Response Team: [Team Lead] + [Team members]
|
||||
Status: [Current mitigation efforts]
|
||||
Workaround: [If available]
|
||||
|
||||
Next Update: 30 minutes
|
||||
Status Page: [Link if updated]
|
||||
```
|
||||
|
||||
## Review and Updates
|
||||
|
||||
This severity matrix should be reviewed quarterly and updated based on:
|
||||
- Incident response learnings
|
||||
- Business priority changes
|
||||
- Service architecture evolution
|
||||
- Regulatory requirement changes
|
||||
- Customer feedback and SLA updates
|
||||
|
||||
**Last Updated:** February 2026
|
||||
**Next Review:** May 2026
|
||||
**Owner:** Engineering Leadership
|
||||
@@ -0,0 +1,562 @@
|
||||
# Root Cause Analysis (RCA) Frameworks Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This guide provides detailed instructions for applying various Root Cause Analysis frameworks during Post-Incident Reviews. Each framework offers a different perspective and approach to identifying underlying causes of incidents.
|
||||
|
||||
## Framework Selection Guidelines
|
||||
|
||||
| Incident Type | Recommended Framework | Why |
|
||||
|---------------|----------------------|-----|
|
||||
| **Process Failure** | 5 Whys | Simple, direct cause-effect chain |
|
||||
| **Complex System Failure** | Fishbone + Timeline | Multiple contributing factors |
|
||||
| **Human Error** | Fishbone | Systematic analysis of contributing factors |
|
||||
| **Extended Incidents** | Timeline Analysis | Understanding decision points |
|
||||
| **High-Risk Incidents** | Bow Tie | Comprehensive barrier analysis |
|
||||
| **Recurring Issues** | 5 Whys + Fishbone | Deep dive into systemic issues |
|
||||
|
||||
---
|
||||
|
||||
## 5 Whys Analysis Framework
|
||||
|
||||
### Purpose
|
||||
Iteratively drill down through cause-effect relationships to identify root causes.
|
||||
|
||||
### When to Use
|
||||
- Simple, linear cause-effect chains
|
||||
- Time-pressured analysis
|
||||
- Process-related failures
|
||||
- Individual component failures
|
||||
|
||||
### Process Steps
|
||||
|
||||
#### Step 1: Problem Statement
|
||||
Write a clear, specific problem statement.
|
||||
|
||||
**Good Example:**
|
||||
> "The payment API returned 500 errors for 2 hours on March 15, affecting 80% of checkout attempts."
|
||||
|
||||
**Poor Example:**
|
||||
> "The system was broken."
|
||||
|
||||
#### Step 2: First Why
|
||||
Ask why the problem occurred. Focus on immediate, observable causes.
|
||||
|
||||
**Example:**
|
||||
- **Why 1:** Why did the payment API return 500 errors?
|
||||
- **Answer:** The database connection pool was exhausted.
|
||||
|
||||
#### Step 3: Subsequent Whys
|
||||
For each answer, ask "why" again. Continue until you reach a root cause.
|
||||
|
||||
**Example Chain:**
|
||||
- **Why 2:** Why was the database connection pool exhausted?
|
||||
- **Answer:** The application was creating more connections than usual.
|
||||
|
||||
- **Why 3:** Why was the application creating more connections?
|
||||
- **Answer:** A new feature wasn't properly closing connections.
|
||||
|
||||
- **Why 4:** Why wasn't the feature properly closing connections?
|
||||
- **Answer:** Code review missed the connection leak pattern.
|
||||
|
||||
- **Why 5:** Why did code review miss this pattern?
|
||||
- **Answer:** We don't have automated checks for connection pooling best practices.
|
||||
|
||||
#### Step 4: Validation
|
||||
Verify that addressing the root cause would prevent the original problem.
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Ask at least 3 "whys"** - Surface causes are rarely root causes
|
||||
2. **Focus on process failures, not people** - Avoid blame, focus on system improvements
|
||||
3. **Use evidence** - Support each answer with data or observations
|
||||
4. **Consider multiple paths** - Some problems have multiple root causes
|
||||
5. **Test the logic** - Work backwards from root cause to problem
|
||||
|
||||
### Common Pitfalls
|
||||
|
||||
- **Stopping too early** - First few whys often reveal symptoms, not causes
|
||||
- **Single-cause assumption** - Complex systems often have multiple contributing factors
|
||||
- **Blame focus** - Focusing on individual mistakes rather than system failures
|
||||
- **Vague answers** - Use specific, actionable answers
|
||||
|
||||
### 5 Whys Template
|
||||
|
||||
```markdown
|
||||
## 5 Whys Analysis
|
||||
|
||||
**Problem Statement:** [Clear description of the incident]
|
||||
|
||||
**Why 1:** [First why question]
|
||||
**Answer:** [Specific, evidence-based answer]
|
||||
**Evidence:** [Supporting data, logs, observations]
|
||||
|
||||
**Why 2:** [Second why question]
|
||||
**Answer:** [Specific answer based on Why 1]
|
||||
**Evidence:** [Supporting evidence]
|
||||
|
||||
[Continue for 3-7 iterations]
|
||||
|
||||
**Root Cause(s) Identified:**
|
||||
1. [Primary root cause]
|
||||
2. [Secondary root cause if applicable]
|
||||
|
||||
**Validation:** [Confirm that addressing root causes would prevent recurrence]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fishbone (Ishikawa) Diagram Framework
|
||||
|
||||
### Purpose
|
||||
Systematically analyze potential causes across multiple categories to identify contributing factors.
|
||||
|
||||
### When to Use
|
||||
- Complex incidents with multiple potential causes
|
||||
- When human factors are suspected
|
||||
- Systemic or organizational issues
|
||||
- When 5 Whys doesn't reveal clear root causes
|
||||
|
||||
### Categories
|
||||
|
||||
#### People (Human Factors)
|
||||
- **Training and Skills**
|
||||
- Insufficient training on new systems
|
||||
- Lack of domain expertise
|
||||
- Skill gaps in team
|
||||
- Knowledge not shared across team
|
||||
|
||||
- **Communication**
|
||||
- Poor communication between teams
|
||||
- Unclear responsibilities
|
||||
- Information not reaching right people
|
||||
- Language/cultural barriers
|
||||
|
||||
- **Decision Making**
|
||||
- Decisions made under pressure
|
||||
- Insufficient information for decisions
|
||||
- Risk assessment inadequate
|
||||
- Approval processes bypassed
|
||||
|
||||
#### Process (Procedures and Workflows)
|
||||
- **Documentation**
|
||||
- Outdated procedures
|
||||
- Missing runbooks
|
||||
- Unclear instructions
|
||||
- Process not documented
|
||||
|
||||
- **Change Management**
|
||||
- Inadequate change review
|
||||
- Rushed deployments
|
||||
- Insufficient testing
|
||||
- Rollback procedures unclear
|
||||
|
||||
- **Review and Approval**
|
||||
- Code review gaps
|
||||
- Architecture review skipped
|
||||
- Security review insufficient
|
||||
- Performance review missing
|
||||
|
||||
#### Technology (Systems and Tools)
|
||||
- **Architecture**
|
||||
- Single points of failure
|
||||
- Insufficient redundancy
|
||||
- Scalability limitations
|
||||
- Tight coupling between systems
|
||||
|
||||
- **Monitoring and Alerting**
|
||||
- Missing monitoring
|
||||
- Alert fatigue
|
||||
- Inadequate thresholds
|
||||
- Poor alert routing
|
||||
|
||||
- **Tools and Automation**
|
||||
- Manual processes prone to error
|
||||
- Tool limitations
|
||||
- Automation gaps
|
||||
- Integration issues
|
||||
|
||||
#### Environment (External Factors)
|
||||
- **Infrastructure**
|
||||
- Hardware failures
|
||||
- Network issues
|
||||
- Capacity limitations
|
||||
- Geographic dependencies
|
||||
|
||||
- **Dependencies**
|
||||
- Third-party service failures
|
||||
- External API changes
|
||||
- Vendor issues
|
||||
- Supply chain problems
|
||||
|
||||
- **External Pressure**
|
||||
- Time pressure from business
|
||||
- Resource constraints
|
||||
- Regulatory changes
|
||||
- Market conditions
|
||||
|
||||
### Process Steps
|
||||
|
||||
#### Step 1: Define the Problem
|
||||
Place the incident at the "head" of the fishbone diagram.
|
||||
|
||||
#### Step 2: Brainstorm Causes
|
||||
For each category, brainstorm potential contributing factors.
|
||||
|
||||
#### Step 3: Drill Down
|
||||
For each factor, ask what caused that factor (sub-causes).
|
||||
|
||||
#### Step 4: Identify Primary Causes
|
||||
Mark the most likely contributing factors based on evidence.
|
||||
|
||||
#### Step 5: Validate
|
||||
Gather evidence to support or refute each suspected cause.
|
||||
|
||||
### Fishbone Template
|
||||
|
||||
```markdown
|
||||
## Fishbone Analysis
|
||||
|
||||
**Problem:** [Incident description]
|
||||
|
||||
### People
|
||||
**Training/Skills:**
|
||||
- [Factor 1]: [Evidence/likelihood]
|
||||
- [Factor 2]: [Evidence/likelihood]
|
||||
|
||||
**Communication:**
|
||||
- [Factor 1]: [Evidence/likelihood]
|
||||
|
||||
**Decision Making:**
|
||||
- [Factor 1]: [Evidence/likelihood]
|
||||
|
||||
### Process
|
||||
**Documentation:**
|
||||
- [Factor 1]: [Evidence/likelihood]
|
||||
|
||||
**Change Management:**
|
||||
- [Factor 1]: [Evidence/likelihood]
|
||||
|
||||
**Review/Approval:**
|
||||
- [Factor 1]: [Evidence/likelihood]
|
||||
|
||||
### Technology
|
||||
**Architecture:**
|
||||
- [Factor 1]: [Evidence/likelihood]
|
||||
|
||||
**Monitoring:**
|
||||
- [Factor 1]: [Evidence/likelihood]
|
||||
|
||||
**Tools:**
|
||||
- [Factor 1]: [Evidence/likelihood]
|
||||
|
||||
### Environment
|
||||
**Infrastructure:**
|
||||
- [Factor 1]: [Evidence/likelihood]
|
||||
|
||||
**Dependencies:**
|
||||
- [Factor 1]: [Evidence/likelihood]
|
||||
|
||||
**External Factors:**
|
||||
- [Factor 1]: [Evidence/likelihood]
|
||||
|
||||
### Primary Contributing Factors
|
||||
1. [Factor with highest evidence/impact]
|
||||
2. [Second most significant factor]
|
||||
3. [Third most significant factor]
|
||||
|
||||
### Root Cause Hypothesis
|
||||
[Synthesized explanation of how factors combined to cause incident]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Timeline Analysis Framework
|
||||
|
||||
### Purpose
|
||||
Analyze the chronological sequence of events to identify decision points, missed opportunities, and process gaps.
|
||||
|
||||
### When to Use
|
||||
- Extended incidents (> 1 hour)
|
||||
- Complex multi-phase incidents
|
||||
- When response effectiveness is questioned
|
||||
- Communication or coordination failures
|
||||
|
||||
### Analysis Dimensions
|
||||
|
||||
#### Detection Analysis
|
||||
- **Time to Detection:** How long from onset to first alert?
|
||||
- **Detection Method:** How was the incident first identified?
|
||||
- **Alert Effectiveness:** Were the right people notified quickly?
|
||||
- **False Negatives:** What signals were missed?
|
||||
|
||||
#### Response Analysis
|
||||
- **Time to Response:** How long from detection to first response action?
|
||||
- **Escalation Timing:** Were escalations timely and appropriate?
|
||||
- **Resource Mobilization:** How quickly were the right people engaged?
|
||||
- **Decision Points:** What key decisions were made and when?
|
||||
|
||||
#### Communication Analysis
|
||||
- **Internal Communication:** How effective was team coordination?
|
||||
- **External Communication:** Were stakeholders informed appropriately?
|
||||
- **Communication Gaps:** Where did information flow break down?
|
||||
- **Update Frequency:** Were updates provided at appropriate intervals?
|
||||
|
||||
#### Resolution Analysis
|
||||
- **Mitigation Strategy:** Was the chosen approach optimal?
|
||||
- **Alternative Paths:** What other options were considered?
|
||||
- **Resource Allocation:** Were resources used effectively?
|
||||
- **Verification:** How was resolution confirmed?
|
||||
|
||||
### Process Steps
|
||||
|
||||
#### Step 1: Event Reconstruction
|
||||
Create comprehensive timeline with all available events.
|
||||
|
||||
#### Step 2: Phase Identification
|
||||
Identify distinct phases (detection, triage, escalation, mitigation, resolution).
|
||||
|
||||
#### Step 3: Gap Analysis
|
||||
Identify time gaps and analyze their causes.
|
||||
|
||||
#### Step 4: Decision Point Analysis
|
||||
Examine key decision points and alternative paths.
|
||||
|
||||
#### Step 5: Effectiveness Assessment
|
||||
Evaluate the overall effectiveness of the response.
|
||||
|
||||
### Timeline Template
|
||||
|
||||
```markdown
|
||||
## Timeline Analysis
|
||||
|
||||
### Incident Phases
|
||||
1. **Detection** ([start] - [end], [duration])
|
||||
2. **Triage** ([start] - [end], [duration])
|
||||
3. **Escalation** ([start] - [end], [duration])
|
||||
4. **Mitigation** ([start] - [end], [duration])
|
||||
5. **Resolution** ([start] - [end], [duration])
|
||||
|
||||
### Key Decision Points
|
||||
**[Timestamp]:** [Decision made]
|
||||
- **Context:** [Situation at time of decision]
|
||||
- **Alternatives:** [Other options considered]
|
||||
- **Outcome:** [Result of decision]
|
||||
- **Assessment:** [Was this optimal?]
|
||||
|
||||
### Communication Timeline
|
||||
**[Timestamp]:** [Communication event]
|
||||
- **Channel:** [Slack/Email/Phone/etc.]
|
||||
- **Audience:** [Who was informed]
|
||||
- **Content:** [What was communicated]
|
||||
- **Effectiveness:** [Assessment]
|
||||
|
||||
### Gaps and Delays
|
||||
**[Time Period]:** [Description of gap]
|
||||
- **Duration:** [Length of gap]
|
||||
- **Cause:** [Why did gap occur]
|
||||
- **Impact:** [Effect on incident response]
|
||||
|
||||
### Response Effectiveness
|
||||
**Strengths:**
|
||||
- [What went well]
|
||||
- [Effective decisions/actions]
|
||||
|
||||
**Weaknesses:**
|
||||
- [What could be improved]
|
||||
- [Missed opportunities]
|
||||
|
||||
### Root Causes from Timeline
|
||||
1. [Process-based root cause]
|
||||
2. [Communication-based root cause]
|
||||
3. [Decision-making root cause]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Bow Tie Analysis Framework
|
||||
|
||||
### Purpose
|
||||
Analyze both preventive measures (left side) and protective measures (right side) around an incident.
|
||||
|
||||
### When to Use
|
||||
- High-severity incidents (SEV1)
|
||||
- Security incidents
|
||||
- Safety-critical systems
|
||||
- When comprehensive barrier analysis is needed
|
||||
|
||||
### Components
|
||||
|
||||
#### Hazards
|
||||
What conditions create the potential for incidents?
|
||||
|
||||
**Examples:**
|
||||
- High traffic loads
|
||||
- Software deployments
|
||||
- Human interactions with critical systems
|
||||
- Third-party dependencies
|
||||
|
||||
#### Top Event
|
||||
What actually went wrong? This is the center of the bow tie.
|
||||
|
||||
**Examples:**
|
||||
- "Database became unresponsive"
|
||||
- "Payment processing failed"
|
||||
- "User authentication service crashed"
|
||||
|
||||
#### Threats (Left Side)
|
||||
What specific causes could lead to the top event?
|
||||
|
||||
**Examples:**
|
||||
- Code defects in new deployment
|
||||
- Database connection pool exhaustion
|
||||
- Network connectivity issues
|
||||
- DDoS attack
|
||||
|
||||
#### Consequences (Right Side)
|
||||
What are the potential impacts of the top event?
|
||||
|
||||
**Examples:**
|
||||
- Revenue loss
|
||||
- Customer churn
|
||||
- Regulatory violations
|
||||
- Brand damage
|
||||
- Data loss
|
||||
|
||||
#### Barriers
|
||||
What controls exist (or could exist) to prevent threats or mitigate consequences?
|
||||
|
||||
**Preventive Barriers (Left Side):**
|
||||
- Code reviews
|
||||
- Automated testing
|
||||
- Load testing
|
||||
- Input validation
|
||||
- Rate limiting
|
||||
|
||||
**Protective Barriers (Right Side):**
|
||||
- Circuit breakers
|
||||
- Failover systems
|
||||
- Backup procedures
|
||||
- Customer communication
|
||||
- Rollback capabilities
|
||||
|
||||
### Process Steps
|
||||
|
||||
#### Step 1: Define the Top Event
|
||||
Clearly state what went wrong.
|
||||
|
||||
#### Step 2: Identify Threats
|
||||
Brainstorm all possible causes that could lead to the top event.
|
||||
|
||||
#### Step 3: Identify Consequences
|
||||
List all potential impacts of the top event.
|
||||
|
||||
#### Step 4: Map Existing Barriers
|
||||
Identify current controls for each threat and consequence.
|
||||
|
||||
#### Step 5: Assess Barrier Effectiveness
|
||||
Evaluate how well each barrier worked (or failed).
|
||||
|
||||
#### Step 6: Recommend Additional Barriers
|
||||
Identify new controls needed to prevent recurrence.
|
||||
|
||||
### Bow Tie Template
|
||||
|
||||
```markdown
|
||||
## Bow Tie Analysis
|
||||
|
||||
**Top Event:** [What went wrong]
|
||||
|
||||
### Threats (Potential Causes)
|
||||
1. **[Threat 1]**
|
||||
- Likelihood: [High/Medium/Low]
|
||||
- Current Barriers: [Preventive controls]
|
||||
- Barrier Effectiveness: [Assessment]
|
||||
|
||||
2. **[Threat 2]**
|
||||
- Likelihood: [High/Medium/Low]
|
||||
- Current Barriers: [Preventive controls]
|
||||
- Barrier Effectiveness: [Assessment]
|
||||
|
||||
### Consequences (Potential Impacts)
|
||||
1. **[Consequence 1]**
|
||||
- Severity: [High/Medium/Low]
|
||||
- Current Barriers: [Protective controls]
|
||||
- Barrier Effectiveness: [Assessment]
|
||||
|
||||
2. **[Consequence 2]**
|
||||
- Severity: [High/Medium/Low]
|
||||
- Current Barriers: [Protective controls]
|
||||
- Barrier Effectiveness: [Assessment]
|
||||
|
||||
### Barrier Analysis
|
||||
**Effective Barriers:**
|
||||
- [Barrier that worked well]
|
||||
- [Why it was effective]
|
||||
|
||||
**Failed Barriers:**
|
||||
- [Barrier that failed]
|
||||
- [Why it failed]
|
||||
- [How to improve]
|
||||
|
||||
**Missing Barriers:**
|
||||
- [Needed preventive control]
|
||||
- [Needed protective control]
|
||||
|
||||
### Recommendations
|
||||
**Preventive Measures:**
|
||||
1. [New barrier to prevent threat]
|
||||
2. [Improvement to existing barrier]
|
||||
|
||||
**Protective Measures:**
|
||||
1. [New barrier to mitigate consequence]
|
||||
2. [Improvement to existing barrier]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Framework Comparison
|
||||
|
||||
| Framework | Time Required | Complexity | Best For | Output |
|
||||
|-----------|---------------|------------|----------|---------|
|
||||
| **5 Whys** | 30-60 minutes | Low | Simple, linear causes | Clear cause chain |
|
||||
| **Fishbone** | 1-2 hours | Medium | Complex, multi-factor | Comprehensive factor map |
|
||||
| **Timeline** | 2-3 hours | Medium | Extended incidents | Process improvements |
|
||||
| **Bow Tie** | 2-4 hours | High | High-risk incidents | Barrier strategy |
|
||||
|
||||
## Combining Frameworks
|
||||
|
||||
### 5 Whys + Fishbone
|
||||
Use 5 Whys for initial analysis, then Fishbone to explore contributing factors.
|
||||
|
||||
### Timeline + 5 Whys
|
||||
Use Timeline to identify key decision points, then 5 Whys on critical failures.
|
||||
|
||||
### Fishbone + Bow Tie
|
||||
Use Fishbone to identify causes, then Bow Tie to develop comprehensive prevention strategy.
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
- [ ] Root causes address systemic issues, not symptoms
|
||||
- [ ] Analysis is backed by evidence, not assumptions
|
||||
- [ ] Multiple perspectives considered (technical, process, human)
|
||||
- [ ] Recommendations are specific and actionable
|
||||
- [ ] Analysis focuses on prevention, not blame
|
||||
- [ ] Findings are validated against incident timeline
|
||||
- [ ] Contributing factors are prioritized by impact
|
||||
- [ ] Root causes link clearly to preventive actions
|
||||
|
||||
## Common Anti-Patterns
|
||||
|
||||
- **Human Error as Root Cause** - Dig deeper into why human error occurred
|
||||
- **Single Root Cause** - Complex systems usually have multiple contributing factors
|
||||
- **Technology-Only Focus** - Consider process and organizational factors
|
||||
- **Blame Assignment** - Focus on system improvements, not individual fault
|
||||
- **Generic Recommendations** - Provide specific, measurable actions
|
||||
- **Surface-Level Analysis** - Ensure you've reached true root causes
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** February 2026
|
||||
**Next Review:** August 2026
|
||||
**Owner:** SRE Team + Engineering Leadership
|
||||
@@ -0,0 +1,914 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Incident Classifier
|
||||
|
||||
Analyzes incident descriptions and outputs severity levels, recommended response teams,
|
||||
initial actions, and communication templates.
|
||||
|
||||
This tool uses pattern matching and keyword analysis to classify incidents according to
|
||||
SEV1-4 criteria and provide structured response guidance.
|
||||
|
||||
Usage:
|
||||
python incident_classifier.py --input incident.json
|
||||
echo "Database is down" | python incident_classifier.py --format text
|
||||
python incident_classifier.py --interactive
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from typing import Dict, List, Tuple, Optional, Any
|
||||
|
||||
|
||||
class IncidentClassifier:
|
||||
"""
|
||||
Classifies incidents based on description, impact metrics, and business context.
|
||||
Provides severity assessment, team recommendations, and response templates.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize the classifier with rules and templates."""
|
||||
self.severity_rules = self._load_severity_rules()
|
||||
self.team_mappings = self._load_team_mappings()
|
||||
self.communication_templates = self._load_communication_templates()
|
||||
self.action_templates = self._load_action_templates()
|
||||
|
||||
def _load_severity_rules(self) -> Dict[str, Dict]:
|
||||
"""Load severity classification rules and keywords."""
|
||||
return {
|
||||
"sev1": {
|
||||
"keywords": [
|
||||
"down", "outage", "offline", "unavailable", "crashed", "failed",
|
||||
"critical", "emergency", "dead", "broken", "timeout", "500 error",
|
||||
"data loss", "corrupted", "breach", "security incident",
|
||||
"revenue impact", "customer facing", "all users", "complete failure"
|
||||
],
|
||||
"impact_indicators": [
|
||||
"100%", "all users", "entire service", "complete",
|
||||
"revenue loss", "sla violation", "customer churn",
|
||||
"security breach", "data corruption", "regulatory"
|
||||
],
|
||||
"duration_threshold": 0, # Immediate classification
|
||||
"response_time": 300, # 5 minutes
|
||||
"description": "Complete service failure affecting all users or critical business functions"
|
||||
},
|
||||
"sev2": {
|
||||
"keywords": [
|
||||
"degraded", "slow", "performance", "errors", "partial",
|
||||
"intermittent", "high latency", "timeouts", "some users",
|
||||
"feature broken", "api errors", "database slow"
|
||||
],
|
||||
"impact_indicators": [
|
||||
"50%", "25-75%", "many users", "significant",
|
||||
"performance degradation", "feature unavailable",
|
||||
"support tickets", "user complaints"
|
||||
],
|
||||
"duration_threshold": 300, # 5 minutes
|
||||
"response_time": 900, # 15 minutes
|
||||
"description": "Significant degradation affecting subset of users or non-critical functions"
|
||||
},
|
||||
"sev3": {
|
||||
"keywords": [
|
||||
"minor", "cosmetic", "single feature", "workaround available",
|
||||
"edge case", "rare issue", "non-critical", "internal tool",
|
||||
"logging issue", "monitoring gap"
|
||||
],
|
||||
"impact_indicators": [
|
||||
"<25%", "few users", "limited impact",
|
||||
"workaround exists", "internal only",
|
||||
"development environment"
|
||||
],
|
||||
"duration_threshold": 3600, # 1 hour
|
||||
"response_time": 7200, # 2 hours
|
||||
"description": "Limited impact with workarounds available"
|
||||
},
|
||||
"sev4": {
|
||||
"keywords": [
|
||||
"cosmetic", "documentation", "typo", "minor bug",
|
||||
"enhancement", "nice to have", "low priority",
|
||||
"test environment", "dev tools"
|
||||
],
|
||||
"impact_indicators": [
|
||||
"no impact", "cosmetic only", "documentation",
|
||||
"development", "testing", "non-production"
|
||||
],
|
||||
"duration_threshold": 86400, # 24 hours
|
||||
"response_time": 172800, # 2 days
|
||||
"description": "Minimal impact, cosmetic issues, or planned maintenance"
|
||||
}
|
||||
}
|
||||
|
||||
def _load_team_mappings(self) -> Dict[str, List[str]]:
|
||||
"""Load team assignment rules based on service/component keywords."""
|
||||
return {
|
||||
"database": ["Database Team", "SRE", "Backend Engineering"],
|
||||
"frontend": ["Frontend Team", "UX Engineering", "Product Engineering"],
|
||||
"api": ["API Team", "Backend Engineering", "Platform Team"],
|
||||
"infrastructure": ["SRE", "DevOps", "Platform Team"],
|
||||
"security": ["Security Team", "SRE", "Compliance Team"],
|
||||
"network": ["Network Engineering", "SRE", "Infrastructure Team"],
|
||||
"authentication": ["Identity Team", "Security Team", "Backend Engineering"],
|
||||
"payment": ["Payments Team", "Finance Engineering", "Compliance Team"],
|
||||
"mobile": ["Mobile Team", "API Team", "QA Engineering"],
|
||||
"monitoring": ["SRE", "Platform Team", "DevOps"],
|
||||
"deployment": ["DevOps", "Release Engineering", "SRE"],
|
||||
"data": ["Data Engineering", "Analytics Team", "Backend Engineering"]
|
||||
}
|
||||
|
||||
def _load_communication_templates(self) -> Dict[str, Dict]:
|
||||
"""Load communication templates for each severity level."""
|
||||
return {
|
||||
"sev1": {
|
||||
"subject": "🚨 [SEV1] {service} - {brief_description}",
|
||||
"body": """CRITICAL INCIDENT ALERT
|
||||
|
||||
Incident Details:
|
||||
- Start Time: {timestamp}
|
||||
- Severity: SEV1 - Critical Outage
|
||||
- Service: {service}
|
||||
- Impact: {impact_description}
|
||||
- Current Status: Investigating
|
||||
|
||||
Customer Impact:
|
||||
{customer_impact}
|
||||
|
||||
Response Team:
|
||||
- Incident Commander: TBD (assigning now)
|
||||
- Primary Responder: {primary_responder}
|
||||
- SMEs Required: {subject_matter_experts}
|
||||
|
||||
Immediate Actions Taken:
|
||||
{initial_actions}
|
||||
|
||||
War Room: {war_room_link}
|
||||
Status Page: Will be updated within 15 minutes
|
||||
Next Update: {next_update_time}
|
||||
|
||||
This is a customer-impacting incident requiring immediate attention.
|
||||
|
||||
{incident_commander_contact}"""
|
||||
},
|
||||
"sev2": {
|
||||
"subject": "⚠️ [SEV2] {service} - {brief_description}",
|
||||
"body": """MAJOR INCIDENT NOTIFICATION
|
||||
|
||||
Incident Details:
|
||||
- Start Time: {timestamp}
|
||||
- Severity: SEV2 - Major Impact
|
||||
- Service: {service}
|
||||
- Impact: {impact_description}
|
||||
- Current Status: Investigating
|
||||
|
||||
User Impact:
|
||||
{customer_impact}
|
||||
|
||||
Response Team:
|
||||
- Primary Responder: {primary_responder}
|
||||
- Supporting Team: {supporting_teams}
|
||||
- Incident Commander: {incident_commander}
|
||||
|
||||
Initial Assessment:
|
||||
{initial_assessment}
|
||||
|
||||
Next Steps:
|
||||
{next_steps}
|
||||
|
||||
Updates will be provided every 30 minutes.
|
||||
Status page: {status_page_link}
|
||||
|
||||
{contact_information}"""
|
||||
},
|
||||
"sev3": {
|
||||
"subject": "ℹ️ [SEV3] {service} - {brief_description}",
|
||||
"body": """MINOR INCIDENT NOTIFICATION
|
||||
|
||||
Incident Details:
|
||||
- Start Time: {timestamp}
|
||||
- Severity: SEV3 - Minor Impact
|
||||
- Service: {service}
|
||||
- Impact: {impact_description}
|
||||
- Status: {current_status}
|
||||
|
||||
Details:
|
||||
{incident_details}
|
||||
|
||||
Assigned Team: {assigned_team}
|
||||
Estimated Resolution: {eta}
|
||||
|
||||
Workaround: {workaround}
|
||||
|
||||
This incident has limited customer impact and is being addressed during normal business hours.
|
||||
|
||||
{team_contact}"""
|
||||
},
|
||||
"sev4": {
|
||||
"subject": "[SEV4] {service} - {brief_description}",
|
||||
"body": """LOW PRIORITY ISSUE
|
||||
|
||||
Issue Details:
|
||||
- Reported: {timestamp}
|
||||
- Severity: SEV4 - Low Impact
|
||||
- Component: {service}
|
||||
- Description: {description}
|
||||
|
||||
This issue will be addressed in the normal development cycle.
|
||||
|
||||
Assigned to: {assigned_team}
|
||||
Target Resolution: {target_date}
|
||||
|
||||
{standard_contact}"""
|
||||
}
|
||||
}
|
||||
|
||||
def _load_action_templates(self) -> Dict[str, List[Dict]]:
|
||||
"""Load initial action templates for each severity level."""
|
||||
return {
|
||||
"sev1": [
|
||||
{
|
||||
"action": "Establish incident command",
|
||||
"priority": 1,
|
||||
"timeout_minutes": 5,
|
||||
"description": "Page incident commander and establish war room"
|
||||
},
|
||||
{
|
||||
"action": "Create incident ticket",
|
||||
"priority": 1,
|
||||
"timeout_minutes": 2,
|
||||
"description": "Create tracking ticket with all known details"
|
||||
},
|
||||
{
|
||||
"action": "Update status page",
|
||||
"priority": 2,
|
||||
"timeout_minutes": 15,
|
||||
"description": "Post initial status page update acknowledging incident"
|
||||
},
|
||||
{
|
||||
"action": "Notify executives",
|
||||
"priority": 2,
|
||||
"timeout_minutes": 15,
|
||||
"description": "Alert executive team of customer-impacting outage"
|
||||
},
|
||||
{
|
||||
"action": "Engage subject matter experts",
|
||||
"priority": 3,
|
||||
"timeout_minutes": 10,
|
||||
"description": "Page relevant SMEs based on affected systems"
|
||||
},
|
||||
{
|
||||
"action": "Begin technical investigation",
|
||||
"priority": 3,
|
||||
"timeout_minutes": 5,
|
||||
"description": "Start technical diagnosis and mitigation efforts"
|
||||
}
|
||||
],
|
||||
"sev2": [
|
||||
{
|
||||
"action": "Assign incident commander",
|
||||
"priority": 1,
|
||||
"timeout_minutes": 30,
|
||||
"description": "Assign IC and establish coordination channel"
|
||||
},
|
||||
{
|
||||
"action": "Create incident tracking",
|
||||
"priority": 1,
|
||||
"timeout_minutes": 5,
|
||||
"description": "Create incident ticket with details and timeline"
|
||||
},
|
||||
{
|
||||
"action": "Assess customer impact",
|
||||
"priority": 2,
|
||||
"timeout_minutes": 15,
|
||||
"description": "Determine scope and severity of user impact"
|
||||
},
|
||||
{
|
||||
"action": "Engage response team",
|
||||
"priority": 2,
|
||||
"timeout_minutes": 30,
|
||||
"description": "Page appropriate technical responders"
|
||||
},
|
||||
{
|
||||
"action": "Begin investigation",
|
||||
"priority": 3,
|
||||
"timeout_minutes": 15,
|
||||
"description": "Start technical analysis and debugging"
|
||||
},
|
||||
{
|
||||
"action": "Plan status communication",
|
||||
"priority": 3,
|
||||
"timeout_minutes": 30,
|
||||
"description": "Determine if status page update is needed"
|
||||
}
|
||||
],
|
||||
"sev3": [
|
||||
{
|
||||
"action": "Assign to appropriate team",
|
||||
"priority": 1,
|
||||
"timeout_minutes": 120,
|
||||
"description": "Route to team with relevant expertise"
|
||||
},
|
||||
{
|
||||
"action": "Create tracking ticket",
|
||||
"priority": 1,
|
||||
"timeout_minutes": 30,
|
||||
"description": "Document issue in standard ticketing system"
|
||||
},
|
||||
{
|
||||
"action": "Assess scope and impact",
|
||||
"priority": 2,
|
||||
"timeout_minutes": 60,
|
||||
"description": "Understand full scope of the issue"
|
||||
},
|
||||
{
|
||||
"action": "Identify workarounds",
|
||||
"priority": 2,
|
||||
"timeout_minutes": 60,
|
||||
"description": "Find temporary solutions if possible"
|
||||
},
|
||||
{
|
||||
"action": "Plan resolution approach",
|
||||
"priority": 3,
|
||||
"timeout_minutes": 120,
|
||||
"description": "Develop plan for permanent fix"
|
||||
}
|
||||
],
|
||||
"sev4": [
|
||||
{
|
||||
"action": "Create backlog item",
|
||||
"priority": 1,
|
||||
"timeout_minutes": 1440, # 24 hours
|
||||
"description": "Add to team backlog for future sprint planning"
|
||||
},
|
||||
{
|
||||
"action": "Triage and prioritize",
|
||||
"priority": 2,
|
||||
"timeout_minutes": 2880, # 2 days
|
||||
"description": "Review and prioritize against other work"
|
||||
},
|
||||
{
|
||||
"action": "Assign owner",
|
||||
"priority": 3,
|
||||
"timeout_minutes": 4320, # 3 days
|
||||
"description": "Assign to appropriate developer when capacity allows"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
def classify_incident(self, incident_data: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Main classification method that analyzes incident data and returns
|
||||
comprehensive response recommendations.
|
||||
|
||||
Args:
|
||||
incident_data: Dictionary containing incident information
|
||||
|
||||
Returns:
|
||||
Dictionary with classification results and recommendations
|
||||
"""
|
||||
# Extract key information from incident data
|
||||
description = incident_data.get('description', '').lower()
|
||||
affected_users = incident_data.get('affected_users', '0%')
|
||||
business_impact = incident_data.get('business_impact', 'unknown')
|
||||
service = incident_data.get('service', 'unknown service')
|
||||
duration = incident_data.get('duration_minutes', 0)
|
||||
|
||||
# Classify severity
|
||||
severity = self._classify_severity(description, affected_users, business_impact, duration)
|
||||
|
||||
# Determine response teams
|
||||
response_teams = self._determine_teams(description, service)
|
||||
|
||||
# Generate initial actions
|
||||
initial_actions = self._generate_initial_actions(severity, incident_data)
|
||||
|
||||
# Create communication template
|
||||
communication = self._generate_communication(severity, incident_data)
|
||||
|
||||
# Calculate response timeline
|
||||
timeline = self._generate_timeline(severity)
|
||||
|
||||
# Determine escalation path
|
||||
escalation = self._determine_escalation(severity, business_impact)
|
||||
|
||||
return {
|
||||
"classification": {
|
||||
"severity": severity.upper(),
|
||||
"confidence": self._calculate_confidence(description, affected_users, business_impact),
|
||||
"reasoning": self._explain_classification(severity, description, affected_users),
|
||||
"timestamp": datetime.now(timezone.utc).isoformat()
|
||||
},
|
||||
"response": {
|
||||
"primary_team": response_teams[0] if response_teams else "General Engineering",
|
||||
"supporting_teams": response_teams[1:] if len(response_teams) > 1 else [],
|
||||
"all_teams": response_teams,
|
||||
"response_time_minutes": self.severity_rules[severity]["response_time"] // 60
|
||||
},
|
||||
"initial_actions": initial_actions,
|
||||
"communication": communication,
|
||||
"timeline": timeline,
|
||||
"escalation": escalation,
|
||||
"incident_data": {
|
||||
"service": service,
|
||||
"description": incident_data.get('description', ''),
|
||||
"affected_users": affected_users,
|
||||
"business_impact": business_impact,
|
||||
"duration_minutes": duration
|
||||
}
|
||||
}
|
||||
|
||||
def _classify_severity(self, description: str, affected_users: str,
|
||||
business_impact: str, duration: int) -> str:
|
||||
"""Classify incident severity based on multiple factors."""
|
||||
scores = {"sev1": 0, "sev2": 0, "sev3": 0, "sev4": 0}
|
||||
|
||||
# Keyword analysis
|
||||
for severity, rules in self.severity_rules.items():
|
||||
for keyword in rules["keywords"]:
|
||||
if keyword in description:
|
||||
scores[severity] += 2
|
||||
|
||||
for indicator in rules["impact_indicators"]:
|
||||
if indicator.lower() in description or indicator.lower() in affected_users.lower():
|
||||
scores[severity] += 3
|
||||
|
||||
# Business impact weighting
|
||||
if business_impact.lower() in ['critical', 'high', 'severe']:
|
||||
scores["sev1"] += 5
|
||||
scores["sev2"] += 3
|
||||
elif business_impact.lower() in ['medium', 'moderate']:
|
||||
scores["sev2"] += 3
|
||||
scores["sev3"] += 2
|
||||
elif business_impact.lower() in ['low', 'minimal']:
|
||||
scores["sev3"] += 2
|
||||
scores["sev4"] += 3
|
||||
|
||||
# User impact analysis
|
||||
if '%' in affected_users:
|
||||
try:
|
||||
percentage = float(re.findall(r'\d+', affected_users)[0])
|
||||
if percentage >= 75:
|
||||
scores["sev1"] += 4
|
||||
elif percentage >= 25:
|
||||
scores["sev2"] += 4
|
||||
elif percentage >= 5:
|
||||
scores["sev3"] += 3
|
||||
else:
|
||||
scores["sev4"] += 2
|
||||
except (IndexError, ValueError):
|
||||
pass
|
||||
|
||||
# Duration consideration
|
||||
if duration > 0:
|
||||
if duration >= 3600: # 1 hour
|
||||
scores["sev1"] += 2
|
||||
scores["sev2"] += 1
|
||||
elif duration >= 1800: # 30 minutes
|
||||
scores["sev2"] += 2
|
||||
scores["sev3"] += 1
|
||||
|
||||
# Return highest scoring severity
|
||||
return max(scores, key=scores.get)
|
||||
|
||||
def _determine_teams(self, description: str, service: str) -> List[str]:
|
||||
"""Determine which teams should respond based on affected systems."""
|
||||
teams = set()
|
||||
text_to_analyze = f"{description} {service}".lower()
|
||||
|
||||
for component, team_list in self.team_mappings.items():
|
||||
if component in text_to_analyze:
|
||||
teams.update(team_list)
|
||||
|
||||
# Default teams if no specific match
|
||||
if not teams:
|
||||
teams = {"General Engineering", "SRE"}
|
||||
|
||||
return list(teams)
|
||||
|
||||
def _generate_initial_actions(self, severity: str, incident_data: Dict) -> List[Dict]:
|
||||
"""Generate prioritized initial actions based on severity."""
|
||||
base_actions = self.action_templates[severity].copy()
|
||||
|
||||
# Customize actions based on incident details
|
||||
for action in base_actions:
|
||||
if severity in ["sev1", "sev2"]:
|
||||
action["urgency"] = "immediate" if severity == "sev1" else "high"
|
||||
else:
|
||||
action["urgency"] = "normal" if severity == "sev3" else "low"
|
||||
|
||||
return base_actions
|
||||
|
||||
def _generate_communication(self, severity: str, incident_data: Dict) -> Dict:
|
||||
"""Generate communication template filled with incident data."""
|
||||
template = self.communication_templates[severity]
|
||||
|
||||
# Fill template with incident data
|
||||
now = datetime.now(timezone.utc)
|
||||
service = incident_data.get('service', 'Unknown Service')
|
||||
description = incident_data.get('description', 'Incident detected')
|
||||
|
||||
communication = {
|
||||
"subject": template["subject"].format(
|
||||
service=service,
|
||||
brief_description=description[:50] + "..." if len(description) > 50 else description
|
||||
),
|
||||
"body": template["body"],
|
||||
"urgency": severity,
|
||||
"recipients": self._determine_recipients(severity),
|
||||
"channels": self._determine_channels(severity),
|
||||
"frequency_minutes": self._get_update_frequency(severity)
|
||||
}
|
||||
|
||||
return communication
|
||||
|
||||
def _generate_timeline(self, severity: str) -> Dict:
|
||||
"""Generate expected response timeline."""
|
||||
rules = self.severity_rules[severity]
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
milestones = []
|
||||
if severity == "sev1":
|
||||
milestones = [
|
||||
{"milestone": "Incident Commander assigned", "minutes": 5},
|
||||
{"milestone": "War room established", "minutes": 10},
|
||||
{"milestone": "Initial status page update", "minutes": 15},
|
||||
{"milestone": "Executive notification", "minutes": 15},
|
||||
{"milestone": "First customer update", "minutes": 30}
|
||||
]
|
||||
elif severity == "sev2":
|
||||
milestones = [
|
||||
{"milestone": "Response team assembled", "minutes": 15},
|
||||
{"milestone": "Initial assessment complete", "minutes": 30},
|
||||
{"milestone": "Stakeholder notification", "minutes": 60},
|
||||
{"milestone": "Status page update (if needed)", "minutes": 60}
|
||||
]
|
||||
elif severity == "sev3":
|
||||
milestones = [
|
||||
{"milestone": "Team assignment", "minutes": 120},
|
||||
{"milestone": "Initial triage complete", "minutes": 240},
|
||||
{"milestone": "Resolution plan created", "minutes": 480}
|
||||
]
|
||||
else: # sev4
|
||||
milestones = [
|
||||
{"milestone": "Backlog creation", "minutes": 1440},
|
||||
{"milestone": "Priority assessment", "minutes": 2880}
|
||||
]
|
||||
|
||||
return {
|
||||
"response_time_minutes": rules["response_time"] // 60,
|
||||
"milestones": milestones,
|
||||
"update_frequency_minutes": self._get_update_frequency(severity)
|
||||
}
|
||||
|
||||
def _determine_escalation(self, severity: str, business_impact: str) -> Dict:
|
||||
"""Determine escalation requirements and triggers."""
|
||||
escalation_rules = {
|
||||
"sev1": {
|
||||
"immediate": ["Incident Commander", "Engineering Manager"],
|
||||
"15_minutes": ["VP Engineering", "Customer Success"],
|
||||
"30_minutes": ["CTO"],
|
||||
"60_minutes": ["CEO", "All C-Suite"],
|
||||
"triggers": ["Extended outage", "Revenue impact", "Media attention"]
|
||||
},
|
||||
"sev2": {
|
||||
"immediate": ["Team Lead", "On-call Engineer"],
|
||||
"30_minutes": ["Engineering Manager"],
|
||||
"120_minutes": ["VP Engineering"],
|
||||
"triggers": ["No progress", "Expanding scope", "Customer escalation"]
|
||||
},
|
||||
"sev3": {
|
||||
"immediate": ["Assigned Engineer"],
|
||||
"240_minutes": ["Team Lead"],
|
||||
"triggers": ["Issue complexity", "Multiple teams needed"]
|
||||
},
|
||||
"sev4": {
|
||||
"immediate": ["Product Owner"],
|
||||
"triggers": ["Customer request", "Stakeholder priority"]
|
||||
}
|
||||
}
|
||||
|
||||
return escalation_rules.get(severity, escalation_rules["sev4"])
|
||||
|
||||
def _determine_recipients(self, severity: str) -> List[str]:
|
||||
"""Determine who should receive notifications."""
|
||||
recipients = {
|
||||
"sev1": ["on-call", "engineering-leadership", "executives", "customer-success"],
|
||||
"sev2": ["on-call", "engineering-leadership", "product-team"],
|
||||
"sev3": ["assigned-team", "team-lead"],
|
||||
"sev4": ["assigned-engineer"]
|
||||
}
|
||||
return recipients.get(severity, recipients["sev4"])
|
||||
|
||||
def _determine_channels(self, severity: str) -> List[str]:
|
||||
"""Determine communication channels to use."""
|
||||
channels = {
|
||||
"sev1": ["pager", "phone", "slack", "email", "status-page"],
|
||||
"sev2": ["pager", "slack", "email"],
|
||||
"sev3": ["slack", "email"],
|
||||
"sev4": ["ticket-system"]
|
||||
}
|
||||
return channels.get(severity, channels["sev4"])
|
||||
|
||||
def _get_update_frequency(self, severity: str) -> int:
|
||||
"""Get recommended update frequency in minutes."""
|
||||
frequencies = {"sev1": 15, "sev2": 30, "sev3": 240, "sev4": 0}
|
||||
return frequencies.get(severity, 0)
|
||||
|
||||
def _calculate_confidence(self, description: str, affected_users: str, business_impact: str) -> float:
|
||||
"""Calculate confidence score for the classification."""
|
||||
confidence = 0.5 # Base confidence
|
||||
|
||||
# Higher confidence with more specific information
|
||||
if '%' in affected_users and any(char.isdigit() for char in affected_users):
|
||||
confidence += 0.2
|
||||
|
||||
if business_impact.lower() in ['critical', 'high', 'medium', 'low']:
|
||||
confidence += 0.15
|
||||
|
||||
if len(description.split()) > 5: # Detailed description
|
||||
confidence += 0.15
|
||||
|
||||
return min(confidence, 1.0)
|
||||
|
||||
def _explain_classification(self, severity: str, description: str, affected_users: str) -> str:
|
||||
"""Provide explanation for the classification decision."""
|
||||
rules = self.severity_rules[severity]
|
||||
|
||||
matched_keywords = []
|
||||
for keyword in rules["keywords"]:
|
||||
if keyword in description.lower():
|
||||
matched_keywords.append(keyword)
|
||||
|
||||
explanation = f"Classified as {severity.upper()} based on: "
|
||||
reasons = []
|
||||
|
||||
if matched_keywords:
|
||||
reasons.append(f"keywords: {', '.join(matched_keywords[:3])}")
|
||||
|
||||
if '%' in affected_users:
|
||||
reasons.append(f"user impact: {affected_users}")
|
||||
|
||||
if not reasons:
|
||||
reasons.append("default classification based on available information")
|
||||
|
||||
return explanation + "; ".join(reasons)
|
||||
|
||||
|
||||
def format_json_output(result: Dict) -> str:
|
||||
"""Format result as pretty JSON."""
|
||||
return json.dumps(result, indent=2, ensure_ascii=False)
|
||||
|
||||
|
||||
def format_text_output(result: Dict) -> str:
|
||||
"""Format result as human-readable text."""
|
||||
classification = result["classification"]
|
||||
response = result["response"]
|
||||
actions = result["initial_actions"]
|
||||
communication = result["communication"]
|
||||
|
||||
output = []
|
||||
output.append("=" * 60)
|
||||
output.append("INCIDENT CLASSIFICATION REPORT")
|
||||
output.append("=" * 60)
|
||||
output.append("")
|
||||
|
||||
# Classification section
|
||||
output.append("CLASSIFICATION:")
|
||||
output.append(f" Severity: {classification['severity']}")
|
||||
output.append(f" Confidence: {classification['confidence']:.1%}")
|
||||
output.append(f" Reasoning: {classification['reasoning']}")
|
||||
output.append(f" Timestamp: {classification['timestamp']}")
|
||||
output.append("")
|
||||
|
||||
# Response section
|
||||
output.append("RECOMMENDED RESPONSE:")
|
||||
output.append(f" Primary Team: {response['primary_team']}")
|
||||
if response['supporting_teams']:
|
||||
output.append(f" Supporting Teams: {', '.join(response['supporting_teams'])}")
|
||||
output.append(f" Response Time: {response['response_time_minutes']} minutes")
|
||||
output.append("")
|
||||
|
||||
# Actions section
|
||||
output.append("INITIAL ACTIONS:")
|
||||
for i, action in enumerate(actions[:5], 1): # Show first 5 actions
|
||||
output.append(f" {i}. {action['action']} (Priority {action['priority']})")
|
||||
output.append(f" Timeout: {action['timeout_minutes']} minutes")
|
||||
output.append(f" {action['description']}")
|
||||
output.append("")
|
||||
|
||||
# Communication section
|
||||
output.append("COMMUNICATION:")
|
||||
output.append(f" Subject: {communication['subject']}")
|
||||
output.append(f" Urgency: {communication['urgency'].upper()}")
|
||||
output.append(f" Recipients: {', '.join(communication['recipients'])}")
|
||||
output.append(f" Channels: {', '.join(communication['channels'])}")
|
||||
if communication['frequency_minutes'] > 0:
|
||||
output.append(f" Update Frequency: Every {communication['frequency_minutes']} minutes")
|
||||
output.append("")
|
||||
|
||||
output.append("=" * 60)
|
||||
|
||||
return "\n".join(output)
|
||||
|
||||
|
||||
def parse_input_text(text: str) -> Dict[str, Any]:
|
||||
"""Parse free-form text input into structured incident data."""
|
||||
# Basic parsing - in a real system, this would be more sophisticated
|
||||
incident_data = {
|
||||
"description": text.strip(),
|
||||
"service": "unknown service",
|
||||
"affected_users": "unknown",
|
||||
"business_impact": "unknown"
|
||||
}
|
||||
|
||||
# Try to extract service name
|
||||
service_patterns = [
|
||||
r'(?:service|api|database|server|application)\s+(\w+)',
|
||||
r'(\w+)(?:\s+(?:is|has|service|api|database))',
|
||||
r'(?:^|\s)(\w+)\s+(?:down|failed|broken)'
|
||||
]
|
||||
|
||||
for pattern in service_patterns:
|
||||
match = re.search(pattern, text.lower())
|
||||
if match:
|
||||
incident_data["service"] = match.group(1)
|
||||
break
|
||||
|
||||
# Try to extract user impact
|
||||
impact_patterns = [
|
||||
r'(\d+%)\s+(?:of\s+)?(?:users?|customers?)',
|
||||
r'(?:all|every|100%)\s+(?:users?|customers?)',
|
||||
r'(?:some|many|several)\s+(?:users?|customers?)'
|
||||
]
|
||||
|
||||
for pattern in impact_patterns:
|
||||
match = re.search(pattern, text.lower())
|
||||
if match:
|
||||
incident_data["affected_users"] = match.group(1) if match.group(1) else match.group(0)
|
||||
break
|
||||
|
||||
# Try to infer business impact
|
||||
if any(word in text.lower() for word in ['critical', 'urgent', 'emergency', 'down', 'outage']):
|
||||
incident_data["business_impact"] = "high"
|
||||
elif any(word in text.lower() for word in ['slow', 'degraded', 'performance']):
|
||||
incident_data["business_impact"] = "medium"
|
||||
elif any(word in text.lower() for word in ['minor', 'cosmetic', 'small']):
|
||||
incident_data["business_impact"] = "low"
|
||||
|
||||
return incident_data
|
||||
|
||||
|
||||
def interactive_mode():
|
||||
"""Run in interactive mode, prompting user for input."""
|
||||
classifier = IncidentClassifier()
|
||||
|
||||
print("🚨 Incident Classifier - Interactive Mode")
|
||||
print("=" * 50)
|
||||
print("Enter incident details (or 'quit' to exit):")
|
||||
print()
|
||||
|
||||
while True:
|
||||
try:
|
||||
description = input("Incident description: ").strip()
|
||||
if description.lower() in ['quit', 'exit', 'q']:
|
||||
break
|
||||
|
||||
if not description:
|
||||
print("Please provide an incident description.")
|
||||
continue
|
||||
|
||||
service = input("Affected service (optional): ").strip() or "unknown"
|
||||
affected_users = input("Affected users (e.g., '50%', 'all users'): ").strip() or "unknown"
|
||||
business_impact = input("Business impact (high/medium/low): ").strip() or "unknown"
|
||||
|
||||
incident_data = {
|
||||
"description": description,
|
||||
"service": service,
|
||||
"affected_users": affected_users,
|
||||
"business_impact": business_impact
|
||||
}
|
||||
|
||||
result = classifier.classify_incident(incident_data)
|
||||
print("\n" + "=" * 50)
|
||||
print(format_text_output(result))
|
||||
print("=" * 50)
|
||||
print()
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n\nExiting...")
|
||||
break
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function with argument parsing and execution."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Classify incidents and provide response recommendations",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
python incident_classifier.py --input incident.json
|
||||
echo "Database is down" | python incident_classifier.py --format text
|
||||
python incident_classifier.py --interactive
|
||||
|
||||
Input JSON format:
|
||||
{
|
||||
"description": "Database connection timeouts",
|
||||
"service": "user-service",
|
||||
"affected_users": "80%",
|
||||
"business_impact": "high"
|
||||
}
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--input", "-i",
|
||||
help="Input file path (JSON format) or '-' for stdin"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--format", "-f",
|
||||
choices=["json", "text"],
|
||||
default="json",
|
||||
help="Output format (default: json)"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--interactive",
|
||||
action="store_true",
|
||||
help="Run in interactive mode"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--output", "-o",
|
||||
help="Output file path (default: stdout)"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Interactive mode
|
||||
if args.interactive:
|
||||
interactive_mode()
|
||||
return
|
||||
|
||||
classifier = IncidentClassifier()
|
||||
|
||||
try:
|
||||
# Read input
|
||||
if args.input == "-" or (not args.input and not sys.stdin.isatty()):
|
||||
# Read from stdin
|
||||
input_text = sys.stdin.read().strip()
|
||||
if not input_text:
|
||||
parser.error("No input provided")
|
||||
|
||||
# Try to parse as JSON first, then as text
|
||||
try:
|
||||
incident_data = json.loads(input_text)
|
||||
except json.JSONDecodeError:
|
||||
incident_data = parse_input_text(input_text)
|
||||
|
||||
elif args.input:
|
||||
# Read from file
|
||||
with open(args.input, 'r') as f:
|
||||
incident_data = json.load(f)
|
||||
else:
|
||||
parser.error("No input specified. Use --input, --interactive, or pipe data to stdin.")
|
||||
|
||||
# Validate required fields
|
||||
if not isinstance(incident_data, dict):
|
||||
parser.error("Input must be a JSON object")
|
||||
|
||||
if "description" not in incident_data:
|
||||
parser.error("Input must contain 'description' field")
|
||||
|
||||
# Classify incident
|
||||
result = classifier.classify_incident(incident_data)
|
||||
|
||||
# Format output
|
||||
if args.format == "json":
|
||||
output = format_json_output(result)
|
||||
else:
|
||||
output = format_text_output(result)
|
||||
|
||||
# Write output
|
||||
if args.output:
|
||||
with open(args.output, 'w') as f:
|
||||
f.write(output)
|
||||
f.write('\n')
|
||||
else:
|
||||
print(output)
|
||||
|
||||
except FileNotFoundError as e:
|
||||
print(f"Error: File not found - {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"Error: Invalid JSON - {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print(f"Error: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
1638
engineering-team/incident-commander/scripts/pir_generator.py
Normal file
1638
engineering-team/incident-commander/scripts/pir_generator.py
Normal file
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user