About This Architecture
End-to-end GCP data ingestion pipeline orchestrating REST API collection, EDI file processing, and event-driven loads across three specialized GKE clusters in a VPC with isolated subnets. Cloud Composer triggers API Fetch Jobs in the Ingestion cluster, which write raw data to GCS staging; EDI Processing cluster handles malware scanning, DLP checks, decryption, and format conversion before publishing to Pub/Sub topics. Snowpipe consumers in Snowflake DEV and PROD ingest cleaned data, while Secret Manager and Workload Identity enforce least-privilege access across all stages. Fork this diagram to customize subnet ranges, add Cloud Dataflow transforms, or integrate additional data sources and destinations.