About This Architecture

Lambda-driven batch and speed layer architecture ingests data from IoT Core, RDS, and S3 into an immutable master dataset, then splits processing into two parallel paths. The batch layer uses Glue ETL and EMR to build processed views in S3, while the speed layer streams data through Lambda and Kinesis to populate ElastiCache and OpenSearch for real-time queries. The serving layer merges both paths via Redshift, Athena, DynamoDB, and API Gateway, feeding analytics dashboards in QuickSight and SageMaker. This design demonstrates the Lambda Architecture pattern for handling high-velocity, high-volume data with both historical accuracy and real-time responsiveness. Fork and customize this diagram to adapt batch intervals, streaming frameworks, or serving stores for your use case.

People also ask

How do I design an AWS data architecture that handles both batch and real-time processing with a unified query layer?

The Lambda Architecture splits data flow into batch and speed layers: batch uses Glue ETL and EMR to build historical views in S3 and Redshift, while speed uses Lambda and Kinesis to populate real-time caches (ElastiCache, OpenSearch, DynamoDB). Both layers feed a serving layer that merges results via Athena and API Gateway for analytics and applications.