Chapter 2

Building Data Pipelines

Data engineering is the infrastructure that makes modeling possible. This chapter covers the ETL pattern, pipeline orchestration with DAGs, distributed processing with Spark, the major cloud warehouses, and how to design a pipeline from source to dashboard.

• 1. What is Data Engineering?→
• 2. ETL: The Central Abstraction→
• 3. Orchestration and DAGs→
• 4. Big Data: Hadoop, Spark, and Hive→
• 5. The Cloud Warehouses→
• 6. Designing a Pipeline→