About This Architecture
Europa Pipeline is a fully serverless data ingestion and enrichment workflow orchestrated by AWS Step Functions, triggered daily by EventBridge Scheduler at 07:00 UTC. Data flows from the SEDIA API through a three-stage Lambda-driven pipeline: fetch and deduplicate items in DynamoDB, map parallel processors across up to 5 concurrent workers enriching content with Bedrock Titan embeddings and Claude Haiku analysis, then finalize results and upsert vectors to S3. The architecture demonstrates best practices for fault tolerance, including a watchdog EventBridge Rule that monitors Step Functions timeouts and triggers SNS notifications on failure. Fork this diagram to customize concurrency limits, embedding models, or add additional enrichment stages without managing servers.