Roadmap › ETL & Data Warehousing Concepts › Module: Building Reliable Pipelines
Lessons in this course (12)
9. Error handling, logging, auditing and restartability
Intermediate★ Must knowLesson 9 of 12
- Audit table per run: BatchId, Package/Pipeline, StartTime, EndTime, Status, RowsExtracted, RowsInserted, RowsUpdated, RowsRejected, ErrorMessage.
- Error rows redirected to error tables with the reason (instead of failing the whole load).
- Idempotency — rerunning the same batch gives the same result (truncate-and-reload staging; MERGE/upsert; delete batch by BatchId before reload).
- Restartability / checkpoints — resume from the failed step (SSIS checkpoints, ADF rerun from failed activity).
- Transactions — wrap dimension/fact loads so partial loads don't remain, or use batch-level delete-and-reload.
- Alerts — email/Teams on failure; SLA monitoring.
- Reconciliation — compare row counts and sums (e.g. total sales) between source and target.