AI Prompts Library

You are an observability engineer designing a comprehensive monitoring and alerting system. Create a monitoring setup plan for the described application.

**Application:** [APP_NAME and DESCRIPTION]
**Architecture:** [Monolith / Microservices / Serverless / Hybrid]
**Infrastructure:** [AWS / GCP / Azure / On-premise]
**Current Monitoring Tools:** [Datadog / Grafana / CloudWatch / Prometheus / New Relic / None]
**SLA Requirements:** [e.g., 99.9% uptime, <200ms p95 latency]
**On-call Team Size:** [NUMBER]

**Generate a Monitoring Plan:**

1. **The Four Golden Signals:**
   For each service/component:
   - **Latency**: What to measure, threshold values (p50, p95, p99)
   - **Traffic**: Request rate, throughput metrics
   - **Errors**: Error rate, error types, 5xx vs 4xx
   - **Saturation**: CPU, memory, disk, connection pool utilization

2. **Infrastructure Metrics:**
   - Server/container health (CPU, memory, disk, network)
   - Database metrics (connections, query time, replication lag, deadlocks)
   - Cache metrics (hit rate, memory usage, evictions)
   - Queue metrics (depth, processing rate, age of oldest message)
   - Load balancer metrics (healthy hosts, response time, error rate)

3. **Application Metrics (custom):**
   - Business metrics (signups, purchases, key user actions)
   - Feature usage metrics
   - Background job metrics (success rate, duration, queue depth)
   - Third-party API metrics (latency, error rate, availability)

4. **Alert Configuration:**
   For each alert:
   - Alert name and description
   - Condition (threshold, anomaly, or rate of change)
   - Severity (P1-P4)
   - Notification channel (PagerDuty, Slack, email)
   - Runbook link
   - Auto-resolution criteria
   - Alert fatigue prevention (grouping, deduplication, silence windows)

5. **Dashboard Design:**
   - Executive dashboard (top-level health)
   - Service dashboard (per-service detail)
   - On-call dashboard (active alerts, recent incidents)
   - Capacity planning dashboard (trend lines)
   - For each dashboard: Layout, key widgets, refresh interval

6. **SLO/SLI Framework:**
   - Define SLIs for each critical user journey
   - Set SLO targets with error budget
   - Burn rate alerting configuration
   - Error budget policy (what happens when budget is exhausted)

7. **Logging Strategy:**
   - Structured logging format
   - Log levels and when to use each
   - Correlation IDs for distributed tracing
   - Log retention and archival policy

**Output**: Complete monitoring specification document with alert definitions, dashboard wireframes, and SLO framework.

Incident response plan for [SYSTEM/SERVICE]:

**Severity Levels**:
- **P0 (Critical)**: Total outage, data breach
- **P1 (High)**: Major feature down, perf degradation
- **P2 (Medium)**: Minor feature issue
- **P3 (Low)**: Cosmetic, no user impact

**Response Timeline**:
- P0: Immediate response, 15min updates
- P1: 30min response, hourly updates
- P2: 4hr response, daily updates
- P3: Best effort

**Incident Response Steps**:

**1. Detection & Alerting** (0-5min)
- Monitor alerts trigger
- On-call engineer paged
- Acknowledge alert

**2. Assessment** (5-15min)
- Determine severity
- Identify affected systems/users
- Assemble incident team
- Create incident channel (#incident-123)

**3. Communication** (15-30min)
- Post status page update
- Notify stakeholders
- Start incident log (timeline)

**4. Mitigation** (Ongoing)
- Implement temporary fix
- Monitor metrics
- Update stakeholders regularly

**5. Resolution**
- Deploy permanent fix
- Verify resolution
- Monitor for regression
- Close incident

**6. Post-Mortem** (Within 48hr)
- Timeline of events
- Root cause analysis
- Impact assessment
- Action items (with owners)
- Lessons learned
- Preventive measures

**Communication Templates**:
- Initial: "We're investigating [issue]."
- Update: "Still working. ETA: [time]. Impact: [scope]."
- Resolved: "Issue resolved at [time]. RCA to follow."

**Tools**:
- Monitoring: [Datadog/New Relic]
- On-call: [PagerDuty/Opsgenie]
- Status: [StatusPage]
- Comms: [Slack/Teams]