Real-Time Streaming Data Platform

Context

Continuous data must be collected, checked, and made usable without losing track of invalid events. This project uses synthetic user events from the Random User API to demonstrate a distributed data platform, from ingestion to real-time analytics.

Summary

Designed and developed a real-time data platform built on a distributed event-streaming architecture. Events are collected, validated, and processed continuously. Usable data and aggregations are stored in an S3-compatible data lake. Invalid events are kept separately to support investigation.

Contributions

• Ingestion and data contracts: Orchestration with Airflow, event publishing to a three-broker Kafka cluster, Avro serialization, and schema compatibility management with Schema Registry.
• Processing and analytics: Validation with Spark Structured Streaming on a cluster with one master and two workers; storage of valid records as Parquet files in MinIO and continuous aggregation.
• Reliability and operations: Quarantine of invalid events with diagnostic information, monitoring with Prometheus and Grafana, Kafka authentication and ACLs, automated tests, and CI checks with GitHub Actions.

Results

• End-to-end pipeline: Event ingestion, validation, storage, and analysis in one platform.
• Three MinIO outputs: Validated data, quarantined events, and analytical aggregations.
• Reproducible environment: Local demonstration with Docker Compose, three Kafka brokers, and a Spark cluster with one master and two workers.

Value

• More reliable data: Schema and quality checks separate usable events from invalid data.
• Traceable anomalies: The quarantine retains rejected events to support diagnosis.
• Reusable analytics: Validated data and aggregations stored as Parquet can support downstream analytical use cases.
• Easier operations: Metrics, dashboards, and automated checks make pipeline behavior more visible.