AI Prompts Library

Draft an incident postmortem for the following:

Incident: [BRIEF DESCRIPTION]
Date/Time: [WHEN]
Duration: [HOW LONG]
Impact: [WHO/WHAT WAS AFFECTED]
Detection: [HOW DID YOU FIND OUT]
Resolution: [WHAT FIXED IT]
Root cause: [WHAT BROKE]

Write a blameless postmortem covering:
1. Summary (3 sentences)
2. Timeline (UTC, key events)
3. Root cause analysis (5 whys, with concrete causal chain - not generic phrases like "human error")
4. What went well during response
5. What went poorly
6. Action items (specific, owned, dated)

Maintain a blameless tone - focus on systems and processes, not individuals.

You are a site reliability engineer who creates comprehensive incident response playbooks. Write a detailed playbook for the described system and incident type.

**System:** [SYSTEM_NAME and DESCRIPTION]
**Infrastructure:** [Cloud provider, key services, architecture overview]
**Team:** [On-call structure, team size, timezone coverage]
**Incident Type:** [Full outage / Partial degradation / Data breach / Security incident / Performance degradation / Data corruption / Dependency failure]

**Generate an Incident Response Playbook:**

1. **Severity Classification:**
   - SEV1 (Critical): Complete service outage affecting all users
   - SEV2 (Major): Significant degradation affecting >25% of users
   - SEV3 (Minor): Limited impact, workaround available
   - SEV4 (Low): Cosmetic or minor issue, no user impact
   - Classification criteria and examples for this specific system

2. **Detection:**
   - Automated alerts that trigger this playbook
   - Alert thresholds and conditions
   - Monitoring dashboards to check first
   - Customer-reported incident identification

3. **Initial Response (first 15 minutes):**
   - On-call engineer actions checklist
   - Communication templates (status page, Slack, PagerDuty)
   - Stakeholder notification matrix (who to page at each severity)
   - War room setup procedure
   - Incident commander role and responsibilities

4. **Diagnosis Procedure:**
   - Step-by-step troubleshooting decision tree
   - Key metrics to check and where to find them
   - Common failure modes and their signatures
   - Log queries for each failure mode
   - Dependency health checks
   - Database connection and query performance checks

5. **Mitigation Actions:**
   - For each common failure mode:
     - Immediate mitigation steps
     - Rollback procedures (with exact commands)
     - Failover procedures
     - Scaling actions
     - Feature flag toggles
   - Safe vs. risky actions and approval requirements

6. **Communication During Incident:**
   - Status update frequency by severity
   - Internal update template
   - External customer communication template
   - Escalation criteria and procedure

7. **Post-Incident:**
   - Incident timeline documentation template
   - Post-mortem template (blameless format)
   - Action items tracking
   - Metrics to report (MTTD, MTTR, customer impact)
   - Follow-up communication to customers

**Output**: Complete playbook document ready to add to the team's runbook wiki.

You are an observability engineer designing a comprehensive monitoring and alerting system. Create a monitoring setup plan for the described application.

**Application:** [APP_NAME and DESCRIPTION]
**Architecture:** [Monolith / Microservices / Serverless / Hybrid]
**Infrastructure:** [AWS / GCP / Azure / On-premise]
**Current Monitoring Tools:** [Datadog / Grafana / CloudWatch / Prometheus / New Relic / None]
**SLA Requirements:** [e.g., 99.9% uptime, <200ms p95 latency]
**On-call Team Size:** [NUMBER]

**Generate a Monitoring Plan:**

1. **The Four Golden Signals:**
   For each service/component:
   - **Latency**: What to measure, threshold values (p50, p95, p99)
   - **Traffic**: Request rate, throughput metrics
   - **Errors**: Error rate, error types, 5xx vs 4xx
   - **Saturation**: CPU, memory, disk, connection pool utilization

2. **Infrastructure Metrics:**
   - Server/container health (CPU, memory, disk, network)
   - Database metrics (connections, query time, replication lag, deadlocks)
   - Cache metrics (hit rate, memory usage, evictions)
   - Queue metrics (depth, processing rate, age of oldest message)
   - Load balancer metrics (healthy hosts, response time, error rate)

3. **Application Metrics (custom):**
   - Business metrics (signups, purchases, key user actions)
   - Feature usage metrics
   - Background job metrics (success rate, duration, queue depth)
   - Third-party API metrics (latency, error rate, availability)

4. **Alert Configuration:**
   For each alert:
   - Alert name and description
   - Condition (threshold, anomaly, or rate of change)
   - Severity (P1-P4)
   - Notification channel (PagerDuty, Slack, email)
   - Runbook link
   - Auto-resolution criteria
   - Alert fatigue prevention (grouping, deduplication, silence windows)

5. **Dashboard Design:**
   - Executive dashboard (top-level health)
   - Service dashboard (per-service detail)
   - On-call dashboard (active alerts, recent incidents)
   - Capacity planning dashboard (trend lines)
   - For each dashboard: Layout, key widgets, refresh interval

6. **SLO/SLI Framework:**
   - Define SLIs for each critical user journey
   - Set SLO targets with error budget
   - Burn rate alerting configuration
   - Error budget policy (what happens when budget is exhausted)

7. **Logging Strategy:**
   - Structured logging format
   - Log levels and when to use each
   - Correlation IDs for distributed tracing
   - Log retention and archival policy

**Output**: Complete monitoring specification document with alert definitions, dashboard wireframes, and SLO framework.

Incident response plan for [SYSTEM/SERVICE]:

**Severity Levels**:
- **P0 (Critical)**: Total outage, data breach
- **P1 (High)**: Major feature down, perf degradation
- **P2 (Medium)**: Minor feature issue
- **P3 (Low)**: Cosmetic, no user impact

**Response Timeline**:
- P0: Immediate response, 15min updates
- P1: 30min response, hourly updates
- P2: 4hr response, daily updates
- P3: Best effort

**Incident Response Steps**:

**1. Detection & Alerting** (0-5min)
- Monitor alerts trigger
- On-call engineer paged
- Acknowledge alert

**2. Assessment** (5-15min)
- Determine severity
- Identify affected systems/users
- Assemble incident team
- Create incident channel (#incident-123)

**3. Communication** (15-30min)
- Post status page update
- Notify stakeholders
- Start incident log (timeline)

**4. Mitigation** (Ongoing)
- Implement temporary fix
- Monitor metrics
- Update stakeholders regularly

**5. Resolution**
- Deploy permanent fix
- Verify resolution
- Monitor for regression
- Close incident

**6. Post-Mortem** (Within 48hr)
- Timeline of events
- Root cause analysis
- Impact assessment
- Action items (with owners)
- Lessons learned
- Preventive measures

**Communication Templates**:
- Initial: "We're investigating [issue]."
- Update: "Still working. ETA: [time]. Impact: [scope]."
- Resolved: "Issue resolved at [time]. RCA to follow."

**Tools**:
- Monitoring: [Datadog/New Relic]
- On-call: [PagerDuty/Opsgenie]
- Status: [StatusPage]
- Comms: [Slack/Teams]