claude-skills-reference/engineering-team/incident-commander/references/sla-management-guide.md

SLA Management Guide

Comprehensive reference for Service Level Agreements, Objectives, and Indicators. Designed for incident commanders who must understand, protect, and communicate SLA status during and after incidents.

---

1. Definitions & Relationships

Service Level Indicator (SLI)

An SLI is the quantitative measurement of a specific aspect of service quality. SLIs are the raw data that feed everything above them. They must be precisely defined, automatically collected, and unambiguous.

Common SLI types by service:

Service Type	SLI	Measurement Method
Web Application	Request latency (p50, p95, p99)	Server-side histogram
Web Application	Availability (successful responses / total requests)	Load balancer logs
REST API	Error rate (5xx responses / total responses)	API gateway metrics
REST API	Throughput (requests per second)	Counter metric
Database	Query latency (p99)	Slow query log + APM
Database	Replication lag (seconds)	Replica monitoring
Message Queue	End-to-end delivery latency	Timestamp comparison
Message Queue	Message loss rate	Producer vs consumer counts
Storage	Durability (objects lost / objects stored)	Integrity checksums
CDN	Cache hit ratio	Edge server logs

SLI specification formula:

SLI = (good events / total events) x 100

For availability: SLI = (successful requests / total requests) x 100 For latency: SLI = (requests faster than threshold / total requests) x 100

Service Level Objective (SLO)

An SLO is the target value or range for an SLI. It defines the acceptable level of reliability. SLOs are internal goals that engineering teams commit to.

Setting meaningful SLOs:

1. Measure the current baseline over 30 days minimum
2. Subtract a safety margin (typically 0.05%-0.1% below actual performance)
3. Validate against user expectations and business requirements
4. Never set an SLO higher than what the system can sustain without heroics

Common pitfall: Setting 99.99% availability when 99.9% meets every user need. The jump from 99.9% to 99.99% is a 10x reduction in allowed downtime and typically requires 3-5x the engineering investment.

SLO examples:

• 99.9% of HTTP requests return a non-5xx response within each calendar month
• 95% of API requests complete in under 200ms (p95 latency)
• 99.95% of messages are delivered within 30 seconds of production

Service Level Agreement (SLA)

An SLA is a formal contract between a service provider and its customers that specifies consequences for failing to meet defined service levels. SLAs must always be looser than SLOs to provide a buffer zone.

Rule of thumb: If your SLO is 99.95%, your SLA should be 99.9% or lower. The gap between SLO and SLA is your safety margin.

The Hierarchy

  SLA (99.9%)     ← Contract with customers, financial penalties
    ↑ backs
  SLO (99.95%)    ← Internal target, triggers error budget policy
    ↑ targets
  SLI (measured)  ← Raw metric: actual uptime = 99.97% this month

Standard combinations by tier:

Tier	SLI (Metric)	SLO (Target)	SLA (Contract)	Allowed Downtime/Month
Critical (payments)	Availability	99.99%	99.95%	SLO: 4.38 min / SLA: 21.9 min
High (core API)	Availability	99.95%	99.9%	SLO: 21.9 min / SLA: 43.8 min
Standard (dashboard)	Availability	99.9%	99.5%	SLO: 43.8 min / SLA: 3.65 hrs
Low (internal tools)	Availability	99.5%	99.0%	SLO: 3.65 hrs / SLA: 7.3 hrs

---

2. Error Budget Policy

What Is an Error Budget

An error budget is the maximum amount of unreliability a service can have within a given period while still meeting its SLO. It is calculated as:

Error Budget = 1 - SLO target

For a 99.9% SLO over a 30-day month (43,200 minutes):

Error Budget = 1 - 0.999 = 0.001 = 0.1%
Allowed Downtime = 43,200 x 0.001 = 43.2 minutes

Downtime Allowances by SLO

SLO	Error Budget	Monthly Downtime	Quarterly Downtime	Annual Downtime
99.0%	1.0%	7 hrs 18 min	21 hrs 54 min	3 days 15 hrs
99.5%	0.5%	3 hrs 39 min	10 hrs 57 min	1 day 19 hrs
99.9%	0.1%	43.8 min	2 hrs 11 min	8 hrs 46 min
99.95%	0.05%	21.9 min	1 hr 6 min	4 hrs 23 min
99.99%	0.01%	4.38 min	13.1 min	52.6 min
99.999%	0.001%	26.3 sec	78.9 sec	5.26 min

Error Budget Consumption Tracking

Track budget consumption as a percentage of the total budget used so far in the current window:

Budget Consumed (%) = (actual bad minutes / allowed bad minutes) x 100

Example: SLO is 99.9% (43.8 min budget/month). On day 10, you have had 15 minutes of downtime.

Budget Consumed = (15 / 43.8) x 100 = 34.2%
Expected consumption at day 10 = (10/30) x 100 = 33.3%
Status: Slightly over pace (34.2% consumed at 33.3% of month elapsed)

Burn Rate

Burn rate measures how fast the error budget is being consumed relative to the steady-state rate:

Burn Rate = (error rate observed / error rate allowed by SLO)

A burn rate of 1.0 means the budget will be exactly exhausted by the end of the window. A burn rate of 10 means the budget will be exhausted in 1/10th of the window.

Burn rate to time-to-exhaustion (30-day month):

Burn Rate	Budget Exhausted In	Urgency
1x	30 days	On pace, monitoring only
2x	15 days	Elevated attention
6x	5 days	Active investigation required
14.4x	2.08 days (~50 hours)	Immediate page
36x	20 hours	Critical, all-hands
720x	1 hour	Total outage scenario

Error Budget Exhaustion Policy

When the error budget is consumed, the following actions trigger based on threshold:

Tier 1 - Budget at 75% consumed (Yellow):

• Notify service team lead via automated alert
• Freeze non-critical deployments to the affected service
• Conduct pre-emptive review of upcoming changes for risk
• Increase monitoring sensitivity (lower alert thresholds)

Tier 2 - Budget at 100% consumed (Orange):

• Hard feature freeze on the affected service
• Mandatory reliability sprint: all engineering effort redirected to reliability
• Daily status updates to engineering leadership
• Postmortem required for the incidents that consumed the budget
• Freeze lasts until budget replenishes to 50% or systemic fixes are verified

Tier 3 - Budget at 150% consumed / SLA breach imminent (Red):

• Escalation to VP Engineering and CTO
• Cross-team war room if dependencies are involved
• Customer communication prepared and staged
• Legal and finance teams briefed on potential SLA credit obligations
• Recovery plan with specific milestones required within 24 hours

Error Budget Policy Template

SERVICE: [service-name]
SLO: [target]% availability over [rolling 30-day / calendar month] window
ERROR BUDGET: [calculated] minutes per window

BUDGET THRESHOLDS:
  - 50% consumed: Team notification, increased vigilance
  - 75% consumed: Feature freeze for this service, reliability focus
  - 100% consumed: Full feature freeze, reliability sprint mandatory
  - SLA threshold crossed: Executive escalation, customer communication

REVIEW CADENCE: Monthly budget review on [day], quarterly SLO adjustment

EXCEPTIONS: Planned maintenance windows excluded if communicated 72+ hours in advance
            and within agreed maintenance allowance.

APPROVED BY: [Engineering Lead] / [Product Lead] / [Date]

---

3. SLA Breach Handling

Detection Methods

Automated detection (primary):

• Real-time monitoring dashboards with SLA burn-rate alerts
• Automated SLA compliance calculations running every 5 minutes
• Threshold-based alerts when cumulative downtime approaches SLA limits
• Synthetic monitoring (external probes) for customer-perspective validation

Manual review (secondary):

• Monthly SLA compliance reports generated on the 1st of each month
• Customer-reported incidents cross-referenced with internal metrics
• Quarterly audits comparing measured SLIs against contracted SLAs
• Discrepancy review between internal metrics and customer-perceived availability

Breach Classification

Minor Breach:

• SLA missed by less than 0.05 percentage points (e.g., 99.85% vs 99.9% SLA)
• Fewer than 3 discrete incidents contributed
• No single incident exceeded 30 minutes
• Customer impact was limited or partial degradation only
• Financial credit: typically 5-10% of monthly service fee

Major Breach:

• SLA missed by 0.05 to 0.5 percentage points
• Extended outage of 1-4 hours in a single incident, or multiple significant incidents
• Clear customer impact with support tickets generated
• Financial credit: typically 10-25% of monthly service fee

Critical Breach:

• SLA missed by more than 0.5 percentage points
• Total outage exceeding 4 hours, or repeated major incidents in same window
• Data loss, security incident, or compliance violation involved
• Financial credit: typically 25-100% of monthly service fee
• May trigger contract termination clauses

Response Protocol

For Minor Breach (within 3 business days):

1. Generate SLA compliance report with exact metrics
2. Document contributing incidents with root causes
3. Send proactive notification to customer success manager
4. Issue service credits if contractually required (do not wait for customer to ask)
5. File internal improvement ticket with 30-day remediation target

For Major Breach (within 24 hours):

1. Incident commander confirms SLA impact calculation
2. Draft customer communication (see template below)
3. Executive sponsor reviews and approves communication
4. Issue service credits with detailed breakdown
5. Schedule root cause review with customer within 5 business days
6. Produce remediation plan with committed timelines

For Critical Breach (immediate):

1. Activate executive escalation chain
2. Legal team reviews contractual exposure
3. Finance team calculates credit obligations
4. Customer communication from VP or C-level within 4 hours
5. Dedicated remediation task force assigned
6. Weekly status updates to customer until remediation complete
7. Formal postmortem document shared with customer within 10 business days

Customer Communication Template

Subject: Service Level Update - [Service Name] - [Month Year]

Dear [Customer Name],

We are writing to inform you that [Service Name] did not meet the committed
service level of [SLA target]% availability during [time period].

MEASURED PERFORMANCE: [actual]% availability
COMMITTED SLA: [SLA target]% availability
SHORTFALL: [delta] percentage points

CONTRIBUTING FACTORS:
- [Date/Time]: [Brief description of incident] ([duration] impact)
- [Date/Time]: [Brief description of incident] ([duration] impact)

SERVICE CREDIT: In accordance with our agreement, a credit of [amount/percentage]
will be applied to your next invoice.

REMEDIATION ACTIONS:
1. [Specific technical fix with completion date]
2. [Process improvement with implementation date]
3. [Monitoring enhancement with deployment date]

We take our service commitments seriously. [Name], [Title] is personally
overseeing the remediation and is available to discuss further at your convenience.

Sincerely,
[Name, Title]

Legal and Compliance Considerations

• Maintain auditable records of all SLA measurements for the full contract term plus 2 years
• SLA calculations must use the measurement methodology defined in the contract, not internal approximations
• Force majeure clauses typically exclude natural disasters, but verify per contract
• Planned maintenance exclusions must match the exact notification procedures in the contract
• Multi-region SLAs may have separate calculations per region; verify aggregation method

---

4. Incident-to-SLA Mapping

Downtime Calculation Methodologies

Full outage: Service completely unavailable. Every minute counts as a full minute of downtime.

Downtime = End Time - Start Time (in minutes)

Partial degradation: Service available but impaired. Apply a degradation factor:

Effective Downtime = Actual Duration x Degradation Factor

Degradation Level	Factor	Description
Complete outage	1.0	Service fully unavailable
Severe degradation	0.75	>50% of requests failing or >10x latency
Moderate degradation	0.5	10-50% of requests affected or 3-10x latency
Minor degradation	0.25	<10% of requests affected or <3x latency increase
Cosmetic / non-functional	0.0	No impact on core SLI metrics

Note: The exact degradation factors must be agreed upon in the SLA contract. The above are industry-standard starting points.

Planned vs Unplanned Downtime

Most SLAs exclude pre-announced maintenance windows from availability calculations, subject to conditions:

• Notification provided N hours/days in advance (commonly 72 hours)
• Maintenance occurs within an agreed window (e.g., Sunday 02:00-06:00 UTC)
• Total planned downtime does not exceed the monthly maintenance allowance (e.g., 4 hours/month)
• Any overrun beyond the planned window counts as unplanned downtime

SLA Availability = (Total Minutes - Excluded Maintenance - Unplanned Downtime) / (Total Minutes - Excluded Maintenance) x 100

Multi-Service SLA Composition

When a customer-facing product depends on multiple services, composite SLA is calculated as:

Serial dependency (all must be up):

Composite SLA = SLA_A x SLA_B x SLA_C
Example: 99.9% x 99.95% x 99.99% = 99.84%

Parallel / redundant (any one must be up):

Composite Availability = 1 - ((1 - SLA_A) x (1 - SLA_B))
Example: 1 - ((1 - 0.999) x (1 - 0.999)) = 1 - 0.000001 = 99.9999%

This is critical during incidents: an outage in a shared dependency may breach SLAs for multiple customer-facing products simultaneously.

Worked Examples

Example 1: Simple outage

• Service: Core API (SLA: 99.9%)
• Month: 30 days = 43,200 minutes
• Incident: Full outage from 14:23 to 14:38 UTC on the 12th (15 minutes)
• No other incidents this month

Availability = (43,200 - 15) / 43,200 x 100 = 99.965%
SLA Status: PASS (99.965% > 99.9%)
Error Budget Consumed: 15 / 43.2 = 34.7%

Example 2: Partial degradation

• Service: Payment Processing (SLA: 99.95%)
• Month: 30 days = 43,200 minutes
• Incident: 50% of transactions failing for 4 hours (240 minutes)
• Degradation factor: 0.5 (moderate - 50% of requests affected)

Effective Downtime = 240 x 0.5 = 120 minutes
Availability = (43,200 - 120) / 43,200 x 100 = 99.722%
SLA Status: FAIL (99.722% < 99.95%)
Shortfall: 0.228 percentage points → Major Breach

Example 3: Multiple incidents

• Service: Dashboard (SLA: 99.5%)
• Month: 31 days = 44,640 minutes
• Incident A: 45-minute full outage on the 5th
• Incident B: 2-hour severe degradation (factor 0.75) on the 18th
• Incident C: 30-minute full outage on the 25th

Total Effective Downtime = 45 + (120 x 0.75) + 30 = 45 + 90 + 30 = 165 minutes
Availability = (44,640 - 165) / 44,640 x 100 = 99.630%
SLA Status: PASS (99.630% > 99.5%)
Error Budget Consumed: 165 / 223.2 = 73.9% → Yellow threshold, feature freeze recommended

---

5. SLO Best Practices

Start with User Journeys

Do not set SLOs based on infrastructure metrics. Start from what users experience:

1. Identify critical user journeys (e.g., "User completes checkout")
2. Map each journey to the services and dependencies involved
3. Define what "good" looks like for each journey (fast, error-free, complete)
4. Select the SLIs that most directly measure that user experience
5. Set SLO targets that reflect the minimum acceptable user experience

A database with 99.99% uptime is meaningless if the API in front of it has a bug causing 5% error rates.

The Four Golden Signals as SLI Sources

From Google SRE, the four golden signals provide comprehensive service health:

Signal	SLI Example	Typical SLO
Latency	p99 request duration < 500ms	99% of requests under threshold
Traffic	Requests per second	N/A (capacity planning, not SLO)
Errors	5xx rate as % of total requests	< 0.1% error rate over rolling window
Saturation	CPU/memory/queue depth	< 80% utilization (capacity SLI)

For most services, latency and error rate are the two most important SLIs to back with SLOs.

Setting SLO Targets

1. Collect 90 days of historical SLI data
2. Calculate the 5th percentile performance (worst 5% of days)
3. Set SLO slightly above that baseline (this ensures the SLO is achievable without heroics)
4. Validate: would a breach at this level actually impact users negatively?
5. Adjust upward only if user impact analysis demands it

Never set SLOs by aspiration. A 99.99% SLO on a service that has historically achieved 99.93% is a guaranteed source of perpetual firefighting with no reliability improvement.

Review Cadence

• Weekly: Review current error budget burn rate, flag services approaching thresholds
• Monthly: Full SLO compliance review, adjust alert thresholds if needed
• Quarterly: Reassess SLO targets based on 90-day data, review SLA contract alignment
• Annually: Strategic SLO review tied to product roadmap and infrastructure investments

Anti-Patterns

Anti-Pattern	Problem	Fix
Vanity SLOs	Setting 99.99% to impress, then ignoring breaches	Set achievable targets, enforce budget policy
SLO Inflation	Ratcheting SLOs up whenever performance is good	Only increase SLOs when users demonstrably need it
Unmeasured SLAs	Committing contractual SLAs without actual SLI measurement	Instrument SLIs before signing SLA contracts
Copy-Paste SLOs	Same SLO for every service regardless of criticality	Tier services by business impact, set SLOs accordingly
Ignoring Dependencies	Setting aggressive SLOs without accounting for dependency reliability	Calculate composite SLA; your SLO cannot exceed dependency chain
Alert-Free SLOs	Having SLOs but no automated alerting on budget consumption	Every SLO must have corresponding burn rate alerts

---

6. Monitoring & Alerting for SLAs

Multi-Window Burn Rate Alerting

The Google SRE approach uses multiple time windows to balance speed of detection against alert noise. Each alert condition requires both a short window (for speed) and a long window (for confirmation):

Alert configuration matrix:

Severity	Short Window	Short Threshold	Long Window	Long Threshold	Action
Critical (Page)	1 hour	> 14.4x burn rate	5 minutes	> 14.4x burn rate	Wake someone up
High (Page)	6 hours	> 6x burn rate	30 minutes	> 6x burn rate	Page on-call within 30 min
Medium (Ticket)	3 days	> 1x burn rate	6 hours	> 1x burn rate	Create ticket, next business day

Why these specific numbers:

• 14.4x burn rate over 1 hour consumes 2% of monthly budget in that hour. At this rate, the entire 30-day budget is gone in ~50 hours. This demands immediate human attention.
• 6x burn rate over 6 hours consumes 5% of monthly budget. The budget will be exhausted in 5 days. Urgent but not wake-up-at-3am urgent.
• 1x burn rate over 3 days means you are on pace to exactly exhaust the budget. This needs investigation but is not an emergency.

Burn Rate Alert Formulas

For a given time window, calculate the burn rate:

burn_rate = (error_count_in_window / request_count_in_window) / (1 - SLO_target)

Example for a 99.9% SLO, observing 50 errors out of 10,000 requests in a 1-hour window:

observed_error_rate = 50 / 10,000 = 0.005 (0.5%)
allowed_error_rate = 1 - 0.999 = 0.001 (0.1%)
burn_rate = 0.005 / 0.001 = 5.0

A burn rate of 5.0 means the error budget is being consumed 5 times faster than the sustainable rate.

Alert Severity to SLA Risk Mapping

Burn Rate	Budget Impact	SLA Risk	Response
< 1x	Under budget pace	None	Routine monitoring
1x - 3x	On pace or slightly over	Low	Investigate next business day
3x - 6x	Budget will exhaust in 5-10 days	Moderate	Investigate within 4 hours
6x - 14.4x	Budget will exhaust in 2-5 days	High	Page on-call, respond in 30 min
> 14.4x	Budget will exhaust in < 2 days	Critical	Immediate page, incident declared
> 100x	Active major outage	SLA breach imminent	All-hands incident response

Dashboard Design for SLA Tracking

Every SLA-tracked service should have a dashboard with these panels:

Row 1 - Current Status:

• Current availability (real-time, rolling 5-minute window)
• Current error rate (real-time)
• Current p99 latency (real-time)

Row 2 - Budget Status:

• Error budget remaining (% of monthly budget, gauge visualization)
• Budget consumption timeline (line chart, actual vs expected burn)
• Budget burn rate (current 1h, 6h, and 3d burn rates)

Row 3 - Historical Context:

• 30-day availability trend (daily granularity)
• SLA compliance status for current and previous 3 months
• Incident markers overlaid on availability timeline

Row 4 - Dependencies:

• Upstream dependency availability (services this service depends on)
• Downstream impact scope (services that depend on this service)
• Composite SLA calculation for customer-facing products

Alert Fatigue Prevention

Alert fatigue is the primary reason SLA monitoring fails in practice. Mitigation strategies:

1. Require dual-window confirmation. Never page on a single short window. Always require both the short window (for speed) and long window (for persistence) to fire simultaneously.

2. Separate page-worthy from ticket-worthy. Only two conditions should wake someone up: >14.4x burn rate sustained, or >6x burn rate sustained. Everything else is a ticket.

3. Deduplicate aggressively. If the same service triggers both a latency and error rate alert for the same underlying issue, group them into a single notification.

4. Auto-resolve. Alerts must auto-resolve when the burn rate drops below threshold. Never leave stale alerts open.

5. Review alert quality monthly. Track the ratio of actionable alerts to total alerts. Target >80% actionable rate. If an alert fires and no human action is needed, tune or remove it.

6. Escalation, not repetition. If an alert is not acknowledged within the response window, escalate to the next tier. Do not re-send the same alert every 5 minutes.

Practical Monitoring Stack

Layer	Tool Category	Purpose
Collection	Prometheus, OpenTelemetry, StatsD	Gather SLI metrics from services
Storage	Prometheus TSDB, Thanos, Mimir	Retain metrics for SLO window + 90 days
Calculation	Prometheus recording rules, Sloth	Pre-compute burn rates and budget consumption
Alerting	Alertmanager, PagerDuty, OpsGenie	Route alerts by severity and schedule
Visualization	Grafana, Datadog	Dashboards for real-time and historical SLA views
Reporting	Custom scripts, SLO generators	Monthly SLA compliance reports for customers

Retention requirement: SLI data must be retained for at least the SLA reporting period (typically monthly or quarterly) plus a 90-day dispute window. Annual SLA reviews require 12 months of data at daily granularity minimum.

---

Last updated: February 2026 For use with: incident-commander skill Maintainer: Engineering Team