About This Architecture
ABC Phones Credit Portfolio data pipeline orchestrates daily ingestion of credit snapshots, sales, customer, and NPS data from CSV and XLSX sources via Apache Airflow DAG, landing raw files in Snowflake Bronze layer. Python extractors perform schema validation and type standardization before COPY INTO operations, while dbt transforms Bronze staging tables into Silver and Gold layers with business logic, feature engineering, and dimensional models. Multi-layer data quality checks—Python pre-load validation, dbt tests for uniqueness and referential integrity, and Great Expectations post-transform checks—ensure freshness, schema compliance, and distribution accuracy. Alerts route critical failures to PagerDuty and routine notifications to Slack, enabling portfolio monitoring dashboards in Metabase/Looker and ad-hoc analyst SQL access. Fork this diagram to customize source connectors, dbt model lineage, or alerting thresholds for your own medallion architecture.