Monitoring and Alerting Setup Guide

You are an observability engineer designing a comprehensive monitoring and alerting system. Create a monitoring setup plan for the described application.

**Application:** [APP_NAME and DESCRIPTION]
**Architecture:** [Monolith / Microservices / Serverless / Hybrid]
**Infrastructure:** [AWS / GCP / Azure / On-premise]
**Current Monitoring Tools:** [Datadog / Grafana / CloudWatch / Prometheus / New Relic / None]
**SLA Requirements:** [e.g., 99.9% uptime, <200ms p95 latency]
**On-call Team Size:** [NUMBER]

**Generate a Monitoring Plan:**

1. **The Four Golden Signals:**
   For each service/component:
   - **Latency**: What to measure, threshold values (p50, p95, p99)
   - **Traffic**: Request rate, throughput metrics
   - **Errors**: Error rate, error types, 5xx vs 4xx
   - **Saturation**: CPU, memory, disk, connection pool utilization

2. **Infrastructure Metrics:**
   - Server/container health (CPU, memory, disk, network)
   - Database metrics (connections, query time, replication lag, deadlocks)
   - Cache metrics (hit rate, memory usage, evictions)
   - Queue metrics (depth, processing rate, age of oldest message)
   - Load balancer metrics (healthy hosts, response time, error rate)

3. **Application Metrics (custom):**
   - Business metrics (signups, purchases, key user actions)
   - Feature usage metrics
   - Background job metrics (success rate, duration, queue depth)
   - Third-party API metrics (latency, error rate, availability)

4. **Alert Configuration:**
   For each alert:
   - Alert name and description
   - Condition (threshold, anomaly, or rate of change)
   - Severity (P1-P4)
   - Notification channel (PagerDuty, Slack, email)
   - Runbook link
   - Auto-resolution criteria
   - Alert fatigue prevention (grouping, deduplication, silence windows)

5. **Dashboard Design:**
   - Executive dashboard (top-level health)
   - Service dashboard (per-service detail)
   - On-call dashboard (active alerts, recent incidents)
   - Capacity planning dashboard (trend lines)
   - For each dashboard: Layout, key widgets, refresh interval

6. **SLO/SLI Framework:**
   - Define SLIs for each critical user journey
   - Set SLO targets with error budget
   - Burn rate alerting configuration
   - Error budget policy (what happens when budget is exhausted)

7. **Logging Strategy:**
   - Structured logging format
   - Log levels and when to use each
   - Correlation IDs for distributed tracing
   - Log retention and archival policy

**Output**: Complete monitoring specification document with alert definitions, dashboard wireframes, and SLO framework.