Flight-delay big data pipeline

The Big Data Programming module's project, in two distinct parts. First, a written Lambda-architecture proposal for a real-time and historical analytics platform: Kafka for ingestion, HDFS for storage, Spark and Flink for batch and stream processing, Cassandra as the serving store, argued and designed on paper against a Kappa-architecture alternative.

Second, the actual hands-on analysis, run in a Hadoop/Spark/Pig/Hive Docker environment against real flight-delay data copied into HDFS: MapReduce preprocessing and analysis jobs, Hive and Pig queries, and Spark itself for a further pass, giving a direct comparison between the older MapReduce-era tooling and the newer Spark stack on identical input. A Spark MLlib model, trained and run in Zeppelin, sat on top of that, and the results were presented in Power BI rather than left as notebook output.

Keeping the architecture proposal and the hands-on build honestly separate mattered more here than making them look like one continuous system.