About This Architecture

WHC Blackwater Mine Unified Data Architecture ingests 6K PI Historian tags, Ampla IQT/DAS sheets, and Citect SCADA signals into an AWS data lake with dual hot and cold paths. The hot path streams ~200 records/second through Kinesis Data Streams to Lambda consumers, populating DynamoDB (P1 KPIs <10ms), Timestream (P2 Trends <2s), and EventBridge (P3 Recommendations <5s) for real-time dashboards. The cold path batches data via Kinesis Firehose into S3 Raw Zone, then Glue ETL curates 3.13B PI rows into Apache Iceberg tables, enabling Athena ad-hoc queries and SageMaker ML feature engineering. Step Functions orchestrates the entire pipeline while Lake Formation, IAM, and VPC enforce governance across Blackwater and future Daunia sites. Fork this diagram to customize ingestion sources, adjust streaming thresholds, or extend to multi-region mining operations.

People also ask

How do you build a real-time data pipeline for mining operations that ingests SCADA and historian data into a data lake with sub-second KPI dashboards and ML feature engineering?

WHC Blackwater Mine architecture uses Kinesis Data Streams (~200 rec/s) to split traffic: hot path routes to DynamoDB (P1 KPIs <10ms), Timestream (P2 Trends), and EventBridge (P3 Recommendations); cold path batches via Firehose into S3 Raw Zone, then Glue ETL curates 3.13B PI rows into Apache Iceberg for Athena queries and SageMaker ML training windows.