WHC Blackwater Mine Unified Data Architecture

aws · data pipeline diagram.

About This Architecture

WHC Blackwater Mine Unified Data Architecture ingests 6K PI Historian tags, Ampla IQT/DAS sheets, and Citect SCADA signals into an AWS data lake with dual hot and cold paths. The hot path streams ~200 records/second through Kinesis Data Streams to Lambda consumers, populating DynamoDB (P1 KPIs <10ms), Timestream (P2 Trends <2s), and EventBridge (P3 Recommendations <5s) for real-time dashboards. The cold path batches data via Kinesis Firehose into S3 Raw Zone, then Glue ETL curates 3.13B PI rows into Apache Iceberg tables, enabling Athena ad-hoc queries and SageMaker ML feature engineering. Step Functions orchestrates the entire pipeline while Lake Formation, IAM, and VPC enforce governance across Blackwater and future Daunia sites. Fork this diagram to customize ingestion sources, adjust streaming thresholds, or extend to multi-region mining operations.

People also ask

How do you build a real-time data pipeline for mining operations that ingests SCADA and historian data into a data lake with sub-second KPI dashboards and ML feature engineering?

WHC Blackwater Mine architecture uses Kinesis Data Streams (~200 rec/s) to split traffic: hot path routes to DynamoDB (P1 KPIs <10ms), Timestream (P2 Trends), and EventBridge (P3 Recommendations); cold path batches via Firehose into S3 Raw Zone, then Glue ETL curates 3.13B PI rows into Apache Iceberg for Athena queries and SageMaker ML training windows.

WHC Blackwater Mine Unified Data Architecture

AWSadvanceddata-engineeringKinesisApache Icebergmining-operationsreal-time-streaming
Domain: Data EngineeringAudience: Data engineers designing real-time mining operations data pipelines on AWS
0 views0 favoritesPublic

Created by

March 29, 2026

Updated

March 29, 2026 at 11:50 AM

Type

data pipeline

Need a custom architecture diagram?

Describe your architecture in plain English and get a production-ready Draw.io diagram in seconds. Works for AWS, Azure, GCP, Kubernetes, and more.

Generate with AI