Roadmap › System Design — High-Level Design (HLD) from Basics to Case Studies › Module: Operating Large Systems
Lessons in this course (14)
14. Reliability, observability and disaster recovery
IntermediateMediumLesson 14 of 14
- Redundancy everywhere: multiple instances, availability zones, replicated databases, no single point of failure.
- Failure handling: timeouts, retries with backoff + jitter, circuit breakers, bulkheads, graceful degradation (show cached data), health checks.
- Observability: logs, metrics (golden signals: latency, traffic, errors, saturation), distributed traces; SLOs and alerts on symptoms users feel.
- Deployments: blue-green, canary releases, feature flags, automatic rollback.
- Disaster recovery: RPO (how much data you can lose) and RTO (how long you can be down) drive the design — backups (RPO hours), async geo-replication (minutes), sync replication (zero); regular restore drills.
- Security: authentication at the edge, least privilege between services, encryption in transit and at rest, secrets in a vault, WAF, DDoS protection.