Data warehousing and analytics

Data-warehousing work at Understory spanned two real eras: an earlier Python pipeline on PySpark, Pandas, and Boto3 against AWS Glue, since superseded, and a current TypeScript ETL pipeline on Node.js that the live Athena/Glue catalog infrastructure actually points at today.

I built the current service with strict validation on every data model rather than trusting upstream data to already be clean, with every run traceable end to end so a bad run was debuggable after the fact instead of a black box that either worked or didn't. This fed into near-real-time analytics on AWS Glue (Spark), Athena, and Redshift, and I contributed to the underlying data-catalog wiring, Glue crawlers and Athena workgroups, that made the data actually queryable once it landed.

The emphasis throughout was data integrity over speed: a fast pipeline that silently corrupts data is worse than a slower one that doesn't.