About This Architecture
Enterprise data lakehouse combining AWS S3 Iceberg, EMR Spark ETL, and legacy Cloudera CDP for unified analytics across internal (SAP, Veeva, Fieldglass) and external sources (CT.gov, CROs). Data flows from SFTP, MSK Kafka, and API ingestion into zoned S3 data lake, processed by auto-scaling EMR clusters (3–20 nodes) orchestrated via MWAA Airflow across multi-AZ VPC with Redshift Serverless and ECS Fargate. AWS Glue Data Catalog and Lake Formation provide metadata governance while Athena enables SQL analytics, with KMS encryption and CloudWatch monitoring securing the pipeline. Fork this diagram to customize ingestion sources, adjust EMR scaling policies, or extend to additional AWS regions using S3 Cross-Region Replication. The architecture demonstrates hybrid cloud data integration patterns suitable for regulated industries managing complex data lineage and compliance requirements.