Cloud-native data pipeline analyzing real-time parking availability on GCP
Designed and deployed a cloud-native, end-to-end data pipeline to analyze real-time and static parking availability datasets using Google Cloud Platform (GCP). Ingested and preprocessed raw datasets via OpenRefine, managed scalable storage using GCP Cloud Storage Buckets, and deployed Dataproc clusters to orchestrate distributed processing across Apache Hive, Apache Spark, and BigQuery. Transformed raw datasets into structured analytical tables and ran comparative performance benchmarking across all three platforms, identifying Apache Spark as the fastest execution engine at 1.9ms execution time — a result that directly informed infrastructure optimization decisions.
The benchmarking wasn't a one-off test: the same transformation logic ran against each engine under Dataproc's managed cluster model, so the comparison reflected real processing conditions rather than isolated synthetic queries. Apache Hive served as the baseline batch-processing layer, Apache Spark handled in-memory distributed transformations, and BigQuery was evaluated as a serverless, SQL-native alternative — giving a practical picture of where each tool's strengths and trade-offs actually show up when working with parking-occupancy-style datasets, the kind municipal and smart-city systems rely on to track space availability at scale.
This project demonstrates hands-on expertise in cloud storage architecture, managed cluster deployment, data warehousing, and platform performance analysis — skills that directly transfer to cloud security monitoring pipelines, SIEM data ingestion, and AI-driven infrastructure observability. The pipeline mirrors real-world SOC data flows: ingest, normalize, store, and analyze at scale — without breaking the cloud bill.