Developing Code for Data Processing using Python and SQL
Objective: Write and optimize production code with the Spark, Python and SQL APIs.
- Python project structure for Declarative Automation Bundles (formerly Databricks Asset Bundles / DABs)
- Managing third-party dependencies: PyPI packages, local wheels and source archives
- Pandas/Python UDFs, streaming tables vs. materialized views
- AUTO CDC APIs (formerly APPLY CHANGES) to simplify CDC in Lakeflow Spark Declarative Pipelines
- Control flow operators (if/else, for/each) and testing with assertDataFrameEqual/assertSchemaEqual
Hands-on Lab: Refactor a batch pipeline into Lakeflow Spark Declarative Pipelines with AUTO CDC APIs and assertDataFrameEqual tests.
Outcome: Idiomatic production-grade Python/SQL code, ready for the highest-weighted block of the exam.