About This Architecture

Enterprise ETL platform on Azure combining API Management, EventStoreDB, and multi-stage data lake architecture with Argo Workflows orchestration. Organisation users authenticate via Microsoft Entra ID and VPN Gateway, then interact with Project API, App API, and ETL API services fronted by API Management. Raw data uploads to Azure Blob Storage trigger Azure Event Grid events that orchestrate Argo Workflows, executing containerised Validation, Import, and Transformation jobs across Bronze, Silver, and Gold data lake tiers. Observability spans Azure Monitor job logs, Log Analytics Workspace, and Application Insights, with secrets managed by Key Vault. Fork this diagram to customise your own multi-stage data pipeline, adjust job orchestration logic, or integrate additional Azure services like Synapse or Databricks.

People also ask

How do I design a scalable ETL pipeline on Azure with event-driven job orchestration and multi-tier data lake storage?

This diagram shows a production Azure ETL platform where API Management routes authenticated users to Project, App, and ETL APIs; raw uploads trigger Azure Event Grid events that orchestrate Argo Workflows to run containerised Validation, Import, and Transformation jobs across Bronze (raw), Silver (curated), and Gold (aggregated) data lake tiers. Azure Monitor and Log Analytics provide end-to-end