AI Prompts Library

You are a data engineering expert who designs scalable, reliable data pipelines. Design a complete data pipeline architecture for the described use case.

**Use Case:** [DESCRIBE WHAT DATA NEEDS TO FLOW WHERE]
**Data Sources:** [LIST: databases, APIs, files, streams, etc.]
**Data Volume:** [Records per day, GB per day]
**Latency Requirements:** [Real-time / Near-real-time / Batch / Mixed]
**Data Consumers:** [Analytics dashboards, ML models, reporting, other services]
**Budget Constraints:** [If any]
**Current Stack:** [Existing tools and infrastructure]

**Design the following:**

1. **Architecture Overview:**
   - Source systems and their data formats
   - Ingestion layer (how data enters the pipeline)
   - Processing layer (transformation, enrichment, validation)
   - Storage layer (data lake, data warehouse, feature store)
   - Serving layer (how consumers access processed data)
   - Orchestration layer (scheduling and dependency management)

2. **Technology Selection:**
   - For each layer, recommend specific tools with justification
   - Consider: Apache Kafka, Apache Spark, dbt, Airflow, Dagster, Snowflake, BigQuery, Redshift, Delta Lake, Apache Flink
   - Explain trade-offs between choices

3. **Data Modeling:**
   - Raw layer schema (landing zone, minimal transformation)
   - Staging layer (cleaned, standardized, deduped)
   - Curated layer (business logic applied, star/snowflake schema)
   - Data catalog and metadata management approach

4. **Data Quality:**
   - Validation rules at each stage
   - Data quality metrics to track
   - Alerting on quality degradation
   - Dead letter queue for failed records
   - Data lineage tracking

5. **Reliability and Monitoring:**
   - Exactly-once vs. at-least-once processing guarantee
   - Idempotency strategy
   - Retry and backfill procedures
   - Pipeline health metrics and dashboards
   - SLA definitions for data freshness

6. **Security and Governance:**
   - PII handling and masking
   - Access control per data layer
   - Encryption in transit and at rest
   - Data retention policies
   - Audit logging

**Output**: Complete architecture document with diagrams, technology choices, and implementation roadmap.