Topic
Dataframes
Learn Dataframes as a connected topic across chapters, concepts, simulations, and interview reasoning.
10 Concepts2 Articles43 min
Overview
Learn Dataframes as a connected topic across chapters, concepts, simulations, and interview reasoning.
How this topic helps
#apache Spark
Big Data
Catalyst
Data Engineering
Learning Path in this Topic
Series that contain articles from Dataframes. Select a path to filter the article list.
Articles
2 matched articles
Article 1Spark DataFrames and Spark SQL: Schema, DDL, and the Catalyst OptimizerTLDR: Catalyst is Spark's query compiler. It takes any DataFrame operation or SQL string, parses it into an abstract syntax tree, resolves column references against the catalog, applies a library of a24 minArticle 2Apache Spark for Data Engineers: RDDs, DataFrames, and Structured StreamingTLDR: Apache Spark distributes Python DataFrame jobs across a cluster of executors, using lazy evaluation and the Catalyst query optimizer to process terabytes with the same code that works on gigabyt19 min