Batch and Streaming Data Platform Flow

azure · flowchart diagram.

About This Architecture

Hybrid batch and streaming data platform on Azure Databricks ingests JSON data through Service Bus and Event Hub, landing raw payloads in Blob Storage before transformation. Batch jobs run on schedule via Databricks, while streaming workflows use Autoloader and Event Grid to detect and process new files continuously into Bronze and Silver Delta tables. Both flows apply auto-flatten logic to normalize nested JSON to the first level, enabling unified downstream consumption. Fork this diagram to customize ingestion schedules, schema validation rules, or add Gold layer aggregations. This architecture demonstrates Azure's event-driven medallion pattern for real-time and batch workloads in a single platform.

People also ask

How do I build a hybrid batch and streaming data pipeline on Azure Databricks?

This diagram shows a medallion architecture where batch jobs ingest JSON from Azure Service Bus on schedule, while streaming workflows continuously pull events from Event Hub via Autoloader. Both flows land raw data in Blob Storage, then use Event Grid and Databricks Autoloader to detect new files and load them into Bronze Delta tables, which are auto-flattened to Silver for downstream consumption

Batch and Streaming Data Platform Flow

AzureintermediateAzure Databricksdata engineeringbatch streamingDelta Lakemedallion architectureevent-driven
Domain: Data EngineeringAudience: Data engineers building hybrid batch-streaming pipelines on Azure Databricks
1 views0 favoritesPublic

Created by

March 19, 2026

Updated

March 23, 2026 at 7:33 AM

Type

flowchart

Need a custom architecture diagram?

Describe your architecture in plain English and get a production-ready Draw.io diagram in seconds. Works for AWS, Azure, GCP, Kubernetes, and more.

Generate with AI