Roadmap › ETL & Data Warehousing Concepts › Module: Building Reliable Pipelines
Lessons in this course (12)
12. Orchestration and scheduling
IntermediateMediumLesson 12 of 12
- SQL Server Agent — schedules SSIS packages/T-SQL jobs on-prem, with steps, retries and notifications.
- Azure Data Factory / Fabric pipelines — triggers (schedule, tumbling window, storage event), activities (Copy, Data Flow, Stored Procedure, Notebook), dependencies, parameters, integration runtimes (Azure, self-hosted for on-prem sources, Azure-SSIS IR to run SSIS packages in the cloud).
- Apache Airflow — DAGs in Python.
- Design for dependencies (dimensions before facts), retries with backoff, timeouts, and alerting.