Topic
Shuffle
Learn Shuffle as a connected topic across chapters, concepts, simulations, and interview reasoning.
10 Concepts4 Articles1h 51m
Overview
Learn Shuffle as a connected topic across chapters, concepts, simulations, and interview reasoning.
How this topic helps
#apache Spark
Performance
Big Data
Aqe
Learning Path in this Topic
Series that contain articles from Shuffle. Select a path to filter the article list.
Articles
4 matched articles
Article 1Shuffles in Spark: Why groupBy Kills PerformanceTLDR: A Spark shuffle is the most expensive operation in any distributed job — it moves every matching key across the network, writes temporary sorted files to disk, and forces a hard synchronization 31 minArticle 2Spark Architecture: Driver, Executors, DAG Scheduler, and Task Scheduler ExplainedTLDR: Spark's architecture is a precise chain of responsibility. The Driver converts user code into a DAG, the DAGScheduler breaks it into stages at shuffle boundaries, the TaskScheduler dispatches ta28 minArticle 3Spark Adaptive Query Execution: Dynamic Coalescing, Pruning, and Skew HandlingTLDR: Before AQE, Spark compiled your entire query into a static physical plan using size estimates that were frequently wrong — and a wrong estimate at planning time meant a skewed join, 800 small ta39 minArticle 4Consistent Hashing: Scaling Without ChaosTLDR: Standard hashing (key % N) breaks when \(N\) changes — adding or removing a server reshuffles almost all keys. Consistent Hashing maps both servers and keys onto a ring (0–360°). When a server i13 min