About This Architecture
Oracle-to-Snowflake migration pipeline combining batch and real-time CDC ingestion into a medallion architecture. Oracle Redo Logs feed Oracle CDC (LogMiner/GoldenGate) and Debezium Connector, which streams changes via Apache Kafka to Snowpipe Streaming, while Airflow orchestrates Apache Spark batch extracts to S3/ADLS staging for bulk COPY INTO loads. Raw data lands in Snowflake's Raw Layer, transforms via dbt into curated Gold Layer tables, with Query History and Alerts monitoring end-to-end pipeline health. This architecture minimizes downtime during cutover, supports incremental syncs post-migration, and leverages Snowflake's native Snowpipe for zero-copy ingestion. Fork this diagram to customize connector configs, adjust batch schedules, or add data quality checks between layers.