About This Architecture

Kitchen Center's serverless data platform on GCP orchestrates multi-source ingestion from SAP APIs and production databases through Cloud Run Jobs triggered by Cloud Scheduler, landing raw data in a GCS Bronze zone partitioned by year. Data flows through BigQuery's medallion architecture—Silver layer mirrors structured data with field-level governance, Gold layer serves materialized and logical views for BI consumption—while Dataform manages transformation logic, lineage tracking, and data quality tests across SQLX definitions. Dataplex Knowledge Catalog provides unified data discovery, governance, and IAM controls across GCS and BigQuery assets, eliminating manual metadata management. The entire infrastructure is version-controlled via GitHub branches (dev/qa/prd) and deployed through Terraform IaC modules with Workload Identity Federation, eliminating JSON key management and enabling secure CI/CD. Fork this diagram to customize ingestion schedules, add new source pipelines, or adapt the medallion schema for your own multi-tenant data platform.

People also ask

How do you build a scalable, governed data platform on GCP with serverless ingestion, medallion architecture, and automated CI/CD?

Kitchen Center's platform uses Cloud Run Jobs scheduled by Cloud Scheduler to ingest from SAP APIs and databases into a GCS Bronze zone, transforms data through BigQuery Silver and Gold layers using Dataform for lineage and quality, and governs all assets via Dataplex. Infrastructure is managed as code with Terraform and deployed securely using Workload Identity Federation, eliminating key managem