Changing Jobs in the Age of AI
A reflection on career and change as a data engineer with six years of experience, navigating the age of AI.
Why You Shouldn’t Always Use dropDuplicates for Deduplication in PySpark
This article analyzes why dropDuplicates() is slow for deduplicating large PySpark datasets (due to its first aggregation over all columns and memory overhead), and shares an optimization case study d
Relationship Safety Also Needs a Checkup
To prevent repeated dating violence and relationship-related tragedies,I built a simple test that helps anyone check for subtle warning signs in their romantic relationships.