What is the difference between availability and reliability in software systems?
KnownRevise
Availability is the share of time (or requests) in which a system is up and able to serve users, for example 99.9 percent of requests succeeding over a month. Reliability is the probability that the system performs correctly, without failure, over a period of time under stated conditions, often expressed through mean time between failures (MTBF) and error rates. A system can be highly available but unreliable (it is always up but often returns wrong results or fails briefly many times), or reliable but less available (it rarely fails, but each outage lasts a long time). Durability, a third related property, is whether stored data survives failures.
Read full answerHide full answer
Applies toweb services, APIs, databases, storage systems and the SLOs used to manage them
MTBF (reliability): how long the system usually runs between failures.
MTTR (repair speed): how long it takes to restore service.
So availability depends on both how often you fail and how quickly you recover.
def availability(mtbf_hours, mttr_hours):
return mtbf_hours / (mtbf_hours + mttr_hours)
# Fails rarely but takes long to fix
print(f"{availability(1000, 10):.4%}") # 99.0099%
# Fails often but recovers in minutes
print(f"{availability(100, 0.05):.4%}") # 99.9500%
The second system is less reliable (fails ten times as often) but more available, because it recovers quickly.
Examples
Available but unreliable: an API that is always reachable but returns intermittent 500 errors or occasionally wrong data. Users can reach it, but cannot trust it.
Reliable but less available: a batch system that rarely fails but is down for a planned four-hour maintenance window every week.
Available but not durable: a cache that is always up but loses data on restart (acceptable for a cache, not for a database).
Why the Distinction Matters
Different engineering levers: availability improves most with redundancy and fast recovery; reliability improves with correctness, testing, safe deployments and fault prevention.
Different SLOs: an availability SLO (successful requests ratio) and a correctness or latency SLO may both be needed.
User perception: users care about both “is it up?” and “does it work correctly when it is up?”.
Measuring Each
Availability: successful requests divided by total valid requests, or minutes in which health checks pass.
Durability: replication and backup verification, restore tests, checksum failures.
Interview Tip
Define availability as “up and serving” and reliability as “working correctly over time”, link them through MTBF and MTTR, and give an example of a system that has one but not the other. Mentioning durability as a separate property makes the answer complete.
2
How do you define system availability, and what do you measure for it?
KnownRevise
System availability is the proportion of time, or of requests, in which the system successfully serves users. The modern, user-focused definition is request-based: availability = successful valid requests / total valid requests over a window such as 30 days. The key components are SLIs (what you measure, such as the success ratio of an endpoint), SLOs (the target, such as 99.9 percent over 30 days), the error budget (the allowed failure, 0.1 percent), and the SLA (a contractual promise, usually looser than the SLO). Measurement should reflect what users experience, from load balancer logs or synthetic probes, not just whether servers are running.
Read full answerHide full answer
Applies toweb services, APIs, internal platforms, databases and anything with uptime commitments
Two Ways to Define It
Approach
Formula
Strengths
Weaknesses
Time-based
Uptime / (uptime + downtime)
Simple, good for whole-system outages
Misses partial outages; what counts as “down”?
Request-based
Successful requests / valid requests
Reflects user experience; captures partial failures
Low traffic windows give noisy numbers
Key Components
Component
Meaning
Example
SLI (indicator)
A measured ratio of good events to valid events
Share of GET /checkout requests returning non-5xx within 1 second
SLO (objective)
Internal target for the SLI
99.9 percent over a rolling 30 days
Error budget
100 percent minus SLO
0.1 percent of requests, about 43 minutes of full outage per 30 days
SLA (agreement)
External, contractual commitment with penalties
99.5 percent monthly, with service credits if missed
What Counts as “Good”
Define precisely:
Success: HTTP status not 5xx (and some 429s if caused by your own overload), correct response body.
Latency: a request slower than a threshold (for example 1 second) is often counted as failed for availability purposes.
Valid requests: exclude client errors (4xx due to bad input) and health-check traffic.
Where to Measure
Source
What it shows
Load balancer or API gateway logs
Real user requests, status codes and latency; closest to the user
Application metrics
Per-endpoint success, error types
Synthetic probes from several regions
Detects outages even with no traffic, including DNS and network problems
Real user monitoring (browser or app)
End-to-end experience including CDN and client network
Components in series multiply: an API that depends on a 99.9 percent database and a 99.9 percent auth service can be at most about 99.8 percent available (0.999 x 0.999).
Redundant components in parallel improve it: two independent 99 percent replicas give 1 - (1 - 0.99)^2 = 99.99 percent, if failover works and failures are independent.
Common Mistakes
Measuring only “process is running” instead of successful user requests.
Ignoring partial outages (one region, one endpoint, one customer segment).
Averaging across endpoints so a broken checkout is hidden by healthy static pages.
Setting SLOs at 100 percent, which leaves no room for change and is unachievable.
Interview Tip
Give the request-based definition, explain SLI, SLO, error budget and SLA with numbers, and say where you would measure (load balancer logs plus synthetic probes). Showing the series and parallel availability maths demonstrates depth.
3
What does five nines mean, and how does it relate to availability?
KnownRevise
“Five nines” means 99.999 percent availability: the system may be unavailable for only about 5.26 minutes per year, or roughly 26 seconds per month. Each additional nine cuts allowed downtime by a factor of ten and usually raises cost and complexity sharply, because it requires redundancy across failure domains, automated failover in seconds, zero-downtime deployments, and highly reliable dependencies. Most services do not need five nines; the right target comes from user needs and business impact, and it must account for every dependency in series.
Read full answerHide full answer
Applies tosetting SLOs and SLAs for telecom, payments, cloud infrastructure, healthcare and other critical services
Detection alone takes time: if alerting needs a minute to fire, a single incident consumes the monthly budget.
Humans are too slow: recovery must be automatic (failover, rollback) within seconds.
Dependencies multiply: a service calling five components each at 99.999 percent can be at most about 99.995 percent (0.99999^5) unless calls are redundant.
Deployments and maintenance must cause no downtime: rolling, blue-green or canary releases, online schema migrations.
Correlated failures (bad configuration pushed everywhere, regional outages, DNS issues) must be contained.
What It Typically Requires
Area
Practice
Redundancy
Active-active across zones and usually regions
Failover
Automated, tested, seconds not minutes
Releases
Progressive rollout with automatic rollback
Dependencies
Each critical dependency at least as available, or redundant and degradable
Data
Synchronous replication or carefully designed conflict handling
Operations
Chaos testing, game days, strong on-call and runbooks
Choosing the Right Target
User expectations: users often cannot tell the difference between 99.99 and 99.999 percent because their own networks and devices are less reliable.
Cost vs benefit: each extra nine can multiply infrastructure and engineering costs.
Error budgets: a target below 100 percent leaves room for releases and experiments. Google’s SRE practice treats the error budget as a resource to spend on change.
Different targets per feature: login and payments may need higher availability than reports or recommendations.
Interview Tip
Quote the downtime numbers (about 5 minutes a year, 26 seconds a month), explain why each nine is ten times harder, and show how dependencies in series reduce availability. Arguing that the target should follow user needs rather than prestige shows good engineering judgement.
4
How does redundancy improve the reliability of a system?
KnownRevise
Redundancy means having more than one component able to do the same job, so that when one fails, another takes over and users keep being served. It improves reliability and availability by removing single points of failure: with independent redundant components, the system fails only if all of them fail at once. Redundancy comes in several forms (active-active, active-passive, N+1 capacity, data replication, geographic spread), and it only helps if failures are independent and failover actually works, which must be tested.
Read full answerHide full answer
Applies toservers and instances, databases, load balancers, network paths, power and cooling, availability zones and regions, and data storage
The Maths
For independent components with availability a:
In series (all required): A = a1 x a2 x … (each extra dependency lowers availability).
In parallel (any one is enough): A = 1 - (1 - a)^n.
def parallel(a, n):
return 1 - (1 - a) ** n
for n in (1, 2, 3):
print(n, f"{parallel(0.99, n):.4%}")
# 1 99.0000%
# 2 99.9900%
# 3 99.9999%
Two independent 99 percent replicas give 99.99 percent in theory. In practice, correlated failures (shared power, the same bad deploy, a common dependency) and failover delays reduce the benefit, which is why redundancy is spread across failure domains.
Types of Redundancy
Type
How it works
Example
Active-active
All copies serve traffic; failures just shrink capacity
Several app instances behind a load balancer; multi-region active-active
Active-passive (standby)
A standby takes over when the primary fails
Database primary with a hot standby and automatic failover
N+1 or N+2 capacity
Extra capacity so losing one or two units still meets demand
Three availability zones; secondary region for disaster recovery
Network and power redundancy
Dual links, redundant switches, multiple power feeds
Two network providers, dual power supplies
Diversity
Different implementations or vendors
Multi-CDN or multi-DNS providers to avoid a single vendor outage
What Makes Redundancy Effective
Independence: place copies in separate failure domains (hosts, racks, zones, regions).
Detection: health checks and failure detectors notice failures quickly.
Automatic failover: traffic moves without human action; DNS TTLs, connection draining and leader election tuned for speed.
Enough capacity: surviving copies must handle the full load (N+1), or failover causes overload and a cascade.
Consistent state: replicas must have up-to-date data, or failover loses writes (see RPO).
Regular testing: failover drills and chaos experiments prove it works.
Pitfalls
False redundancy: two instances on the same host or in the same zone, or both depending on one database.
Split brain: two primaries accepting writes after a faulty failover; prevented with quorums and fencing.
Complexity: more components and failover logic mean new failure modes.
Correlated software bugs: identical replicas share the same bug; a bad configuration pushed everywhere defeats redundancy, so roll out changes progressively.
Cost: idle standby capacity and duplicated infrastructure.
Interview Tip
Explain redundancy as removing single points of failure, show the parallel availability formula, and stress independence of failure domains, automatic failover, spare capacity and testing. Mentioning correlated failures such as a bad deploy shows you understand its limits.
5
What is a single point of failure, and how can it be removed?
KnownRevise
A single point of failure (SPOF) is any component whose failure alone makes the whole system, or a critical function, unavailable. It can be hardware (one database server, one load balancer), software (one cron host, one leader without failover), infrastructure (one availability zone, one DNS provider), data (one copy of a database), or even people and processes (one engineer who knows how to restore backups). You remove SPOFs by finding them through dependency mapping and failure testing, then adding redundancy, automatic failover, isolation and graceful degradation.
Read full answerHide full answer
Applies toarchitecture reviews, reliability planning, cloud deployments and incident prevention
Common SPOFs and Mitigations
SPOF
Mitigation
Single application server
Several instances behind a load balancer, autoscaling
Single load balancer
Managed or redundant load balancers (active-active or with floating IP failover)
Single database primary
Replicas with automatic failover (Patroni, RDS Multi-AZ), or distributed databases
One availability zone
Spread instances and replicas across three zones
One region
Multi-region active-active or warm standby for critical systems
Single DNS or CDN provider
Multiple providers or a provider with strong redundancy, sensible TTLs
Shared configuration or secrets service
Caching of config, highly available clusters, safe defaults
Cron or batch job on one host
Distributed scheduler with leader election, or idempotent jobs on several nodes with locking
Message broker with one node
Clustered brokers with replicated queues or partitions
Certificate or credential expiry
Automated renewal and expiry monitoring
One person with critical knowledge
Runbooks, shared on-call, documentation, access for several people
Third-party API
Timeouts, fallbacks, caching, an alternative provider where justified
How to Find SPOFs
Draw the request path: from DNS and CDN through load balancers, services, databases, queues and third parties.
Ask “what if this fails?” for every box and arrow, including control-plane services (identity, config, service discovery).
Check shared dependencies: a “redundant” pair that uses one disk, one network switch, one zone or one database is not redundant.
Run failure tests: terminate instances, block a zone, fail a dependency (chaos engineering, game days).
Review incidents: past outages often reveal hidden SPOFs.
Example
A web app runs three instances across zones, but all of them read session data from one Redis node.
Problem: if Redis fails, every user is logged out or every request errors. Redis is a SPOF.
Fixes: Redis with replicas and automatic failover (Sentinel, Redis Cluster or a managed service across zones); or signed cookies or tokens so sessions do not need Redis; and a fallback that degrades gracefully instead of failing every request.
Beyond Redundancy
Graceful degradation: if recommendations fail, show the page without them.
Bulkheads: isolate resources so one failing dependency cannot consume all threads or connections.
Timeouts and circuit breakers to stop a slow dependency from becoming a SPOF through resource exhaustion.
Progressive rollouts: a bad configuration pushed everywhere at once acts as a SPOF in time; staged rollouts limit the damage.
Trade-offs
Removing every SPOF costs money and adds complexity. Prioritise by impact and likelihood: eliminate SPOFs on critical paths (checkout, login), and accept some on internal or low-impact tools with good recovery plans.
Interview Tip
Define a SPOF, give examples across infrastructure, data, software and people, explain how to find them (dependency mapping, failure testing) and how to remove them (redundancy, failover, degradation, bulkheads). Pointing out hidden shared dependencies shows real-world experience.
6
Why is mean time between failures (MTBF) important in reliability engineering?
KnownRevise
Mean Time Between Failures (MTBF) is the average operating time between one failure and the next for a repairable system. It is a measure of reliability: a higher MTBF means failures happen less often. MTBF matters because it helps predict failure frequency, plan redundancy, maintenance and spare capacity, compare designs or vendors, and, together with MTTR, calculate availability = MTBF / (MTBF + MTTR). It is an average, so it says nothing precise about when a particular component will fail, and it is only meaningful for consistent definitions of “failure”.
uptime_hours = 720 # a 30-day month
failures = 3
mttr_hours = 0.5 # average time to restore
mtbf = uptime_hours / failures
availability = mtbf / (mtbf + mttr_hours)
print(f"MTBF={mtbf:.0f} h, availability={availability:.3%}")
# MTBF=240 h, availability=99.792%
Why It Matters
Use
How MTBF helps
Availability planning
Combined with MTTR it predicts expected uptime
Redundancy design
Knowing how often components fail shows how many spares or replicas you need
Capacity planning
In large fleets, failures are routine: 10,000 disks with an MTBF of 1 million hours means a disk failure roughly every 100 hours
Maintenance scheduling
Replace or patch components before failure rates rise
Vendor and design comparison
Compare hardware or architectures on failure frequency
Tracking improvement
A rising MTBF shows that reliability work (testing, safer deploys) is paying off
Fleet Example
disks = 10_000
disk_mtbf_hours = 1_000_000
hours_between_failures = disk_mtbf_hours / disks
print(f"one disk failure about every {hours_between_failures:.0f} hours")
# one disk failure about every 100 hours
At scale, “rare” failures become constant, so systems must handle them automatically.
Related Metrics
Metric
Meaning
MTTF (mean time to failure)
For non-repairable components, average time until failure
MTTR (mean time to repair or restore)
Average time to restore service
MTTD (mean time to detect)
Time from failure to detection
Failure rate
Approximately 1 / MTBF for a constant-rate model
Limitations
Averages hide patterns: hardware failure rates follow a “bathtub curve” (early failures, a stable period, wear-out), so failures are not evenly spread.
Definition matters: is a brief error spike a failure? A partial outage? Inconsistent definitions make MTBF meaningless.
Vendor figures are lab estimates: field failure rates can be noticeably higher (Google’s and Backblaze’s large disk studies found real-world rates differ from datasheets).
Software is different: failures often come from changes (deploys, configuration) rather than wear; tracking change failure rate may be more useful.
Small samples: with few incidents, MTBF fluctuates widely.
Capacity headroom, rate limiting and load shedding to avoid overload failures.
Eliminating known failure modes found in post-mortems.
Reliable hardware and proactive replacement.
Interview Tip
Define MTBF, show how it combines with MTTR to give availability, and use a fleet example to show that failures become routine at scale. Pointing out its limits (averages, definitions, the bathtub curve) and that in software, improving MTTR is often cheaper, shows mature thinking.
7
What role does mean time to repair (MTTR) play in system availability?
KnownRevise
Mean Time to Repair (or Restore), MTTR is the average time from a failure until service is restored. It is one half of the availability equation, availability = MTBF / (MTBF + MTTR), so cutting recovery time directly raises availability, often more cheaply than preventing every failure. MTTR includes detection, diagnosis, mitigation and recovery, and modern reliability practice focuses heavily on reducing it through fast alerting, automated failover and rollback, good observability, runbooks and practised incident response.
Read full answerHide full answer
Applies toSRE and operations teams, incident management, SLOs and error budgets, and system design for fast recovery
Practice: game days and chaos experiments so teams and tools are ready.
Small, frequent changes: easier to identify and revert the change that caused a problem.
MTTR vs MTBF Investments
Approach
Example
Effect
Increase MTBF
More testing, safer infrastructure
Fewer incidents
Decrease MTTR
Rollback automation, failover, better alerts
Shorter incidents
Complex systems always fail eventually, so many teams find that reducing MTTR gives faster gains, while MTBF work removes repeat causes found in post-mortems.
Caveats
MTTR is an average; a single long outage can dominate user impact, so also track the distribution and the worst incidents.
Definitions vary (repair, restore, recover, respond); agree on one and measure consistently.
Mitigation first: restore users quickly, then investigate the root cause.
Interview Tip
Explain MTTR’s phases, show numerically how lowering it raises availability, and list concrete ways to cut it (automated failover, rollback, feature flags, observability, runbooks). Stressing “mitigate first, then find the root cause” shows incident-response maturity.
8
What is the difference between high availability and fault tolerance?
KnownRevise
High availability (HA) means the system is designed to minimise downtime: when a component fails, it recovers quickly, often with a brief interruption (seconds to minutes) while failover happens. Fault tolerance (FT) means the system continues operating with no interruption and no lost work when a component fails, because redundant components are running in lockstep or fully active and take over instantly. FT is stricter, more expensive and more complex; most web systems aim for HA, while FT is used where even a brief interruption is unacceptable.
Read full answerHide full answer
Applies todatabases, servers and virtual machines, network equipment, aircraft and industrial control systems, payment and telecom infrastructure
Comparison
Aspect
High availability
Fault tolerance
Goal
Minimise downtime
Zero interruption on component failure
Failure effect
Short disruption during failover
No visible disruption
In-flight requests
May fail and need a retry
Continue, or are transparently completed
Typical design
Active-passive or active-active with failover
Fully redundant, synchronized components, lockstep execution or replicated state machines
Cost
Moderate
High (duplicate hardware, synchronization overhead)
Complexity
Moderate
High
Examples
Database with automatic standby failover; app servers behind a load balancer
Aircraft flight control computers, VMware vSphere Fault Tolerance, RAID 1 mirrored disks, redundant power supplies
Example: Database Failover vs Fault Tolerance
HA: a PostgreSQL primary with a synchronous standby and automatic failover (Patroni or a managed Multi-AZ service). When the primary fails, failover usually takes from several seconds to a couple of minutes; open connections drop and clients reconnect and retry.
FT-like behaviour: a consensus-based database (CockroachDB, Spanner) or a storage layer that writes to several replicas synchronously, so a single node failure does not lose committed data, and the remaining nodes keep serving. Even there, requests in flight on the failed node may need retrying, so it is strictly “highly available with no data loss” rather than perfect fault tolerance.
Lockstep or triple modular redundancy: several processors execute the same instructions and vote on results (used in aerospace and some safety systems).
Replicated state machines: every replica applies the same ordered log (Raft, Paxos).
Transparent retries and idempotency: the system hides individual failures from callers.
High Availability Techniques
Redundant instances with health checks and load balancing.
Automatic failover of databases, caches and brokers.
Multi-zone deployment and fast DNS or anycast routing changes.
Graceful degradation when parts of the system are down.
How They Relate
FT is a stronger form of availability engineering; a fault-tolerant component contributes to a highly available system.
Many systems combine both: fault-tolerant storage (replicated disks) underneath highly available services (stateless instances that can be replaced).
Neither protects against correlated failures such as a bad deployment or configuration pushed to all replicas; that needs progressive rollouts and isolation.
Choosing
Requirement
Choice
A few seconds of disruption are acceptable
HA
Interruption could endanger lives or cause major financial loss
FT for those components
Cost-sensitive web applications
HA with retries and idempotent operations
Interview Tip
Say that HA minimises downtime with quick recovery, while FT continues with no interruption, then compare cost, complexity and examples. Noting that most distributed systems achieve “HA plus client retries” rather than true FT shows a realistic view.
9
How would you architect a system for high availability?
KnownRevise
To architect for high availability, remove single points of failure at every layer, spread redundant components across independent failure domains (zones, then regions if needed), make failure detected and handled automatically, and keep the system useful while degraded. Concretely: stateless services behind load balancers, replicated data stores with automatic failover, health checks, timeouts, retries with backoff and circuit breakers, safe deployments, capacity headroom and load shedding, and observability with SLOs, all verified by failure testing.
Read full answerHide full answer
Applies toweb applications, APIs, microservices platforms and data services on cloud or on-premises infrastructure
Reference Architecture (Single Region, Three Zones)
Users -> DNS (health-checked) -> CDN / edge
-> Load balancer (spans zones A, B, C)
-> Stateless app instances in A, B, C (autoscaled, N+1 capacity)
-> Cache cluster with replicas across zones
-> Database: primary in A, synchronous standby in B, replica in C, automatic failover
-> Message queue cluster replicated across zones
-> Object storage (replicated by the provider)
Layer-by-Layer Decisions
Layer
HA technique
DNS and edge
Managed DNS with health checks, CDN serving static content and cached pages
Load balancing
Managed or redundant load balancers across zones; health checks remove bad instances
Application
Stateless instances in each zone; sessions in signed tokens or a replicated store; autoscaling
Durable queues; idempotent consumers; retries with dead-letter queues
Dependencies
Timeouts, retries with exponential backoff and jitter, circuit breakers, fallbacks
Capacity
N+1 per zone (survive one zone loss at peak), rate limiting, load shedding
Releases
Rolling, blue-green or canary deployments with automatic rollback; feature flags
Operations
SLOs, alerts on error budget burn, runbooks, on-call
Availability Maths to Guide the Design
Dependencies in series multiply: an app (99.95 percent) that needs a database (99.95 percent) and a cache (99.9 percent) is at most about 99.8 percent available, unless the cache is optional.
Parallel redundancy helps: three independent zones each at 99.5 percent give 1 - 0.005^3, well above 99.99 percent, if failover is automatic and capacity suffices.
Make non-critical dependencies optional (degrade instead of failing) so they leave the series chain.
Multi-Region (When Needed)
Active-passive (warm standby): second region with replicated data; fail over on regional outage. Lower cost; RTO of minutes.
Active-active: users served from several regions; needs data designed for multi-region writes or a home-region per record. Highest availability and complexity. The consistency choices during inter-region partitions follow the CAP trade-off.
Resilience Patterns
Timeouts on every network call.
Retries only for idempotent operations, with exponential backoff and jitter and a retry budget, to avoid retry storms.
Circuit breakers to stop calling a failing dependency.
Bulkheads to isolate resource pools per dependency.
Graceful degradation: hide recommendations or show cached data rather than failing the page.
Verify It Works
Chaos experiments: terminate instances, fail a zone, inject latency.
Regular failover drills for databases and regions.
Restore tests for backups.
Interview Tip
Walk layer by layer from DNS to data, emphasise multi-zone redundancy, automatic failover, resilience patterns and safe deployments, and use the series and parallel availability maths to justify choices. Ending with failure testing shows you know designs must be proven.
10
Which design patterns are commonly used to improve system availability?
KnownRevise
Common availability patterns fall into four groups: redundancy and failover (active-active, active-passive, leader election, replication), failure isolation (bulkheads, cell-based architecture, circuit breakers, timeouts), controlled recovery (retries with exponential backoff and jitter, idempotency, queue-based load levelling, health checks with self-healing), and graceful degradation under load (rate limiting, load shedding, fallbacks, caching, feature flags). Combined with safe deployment patterns (canary, blue-green) they reduce both how often users are affected and for how long.
Read full answerHide full answer
Applies todistributed systems, microservices, cloud platforms and client libraries
Pattern Catalogue
Pattern
What it does
Example
Active-active redundancy
Several instances or regions serve traffic
App servers in three zones behind a load balancer
Active-passive failover
Standby takes over on primary failure
Database primary with hot standby
Leader election
Exactly one node coordinates, with automatic replacement
etcd or ZooKeeper electing a scheduler leader
Health check and self-healing
Detect and replace unhealthy instances
Kubernetes liveness and readiness probes
Timeout
Bound waiting on a dependency
300 ms timeout on a pricing call
Retry with exponential backoff and jitter
Recover from transient errors without stampedes
100 ms, 200 ms, 400 ms plus random jitter
Circuit breaker
Stop calling a failing dependency, fail fast
Open after 50 percent errors, probe after 30 seconds
Bulkhead
Separate resource pools so one failure cannot exhaust all
Separate connection pools per downstream service
Fallback
Alternative response when a dependency fails
Default recommendations, cached prices
Queue-based load levelling
Buffer bursts; process at a steady rate
Orders written to a queue, workers process asynchronously
Rate limiting and load shedding
Protect the system under overload
Return 429 or 503 for low-priority traffic first
Caching
Serve reads without the origin
CDN with stale-if-error
Cell-based architecture
Split users into independent cells to limit blast radius
Each cell has its own stack serving a subset of customers
Canary and blue-green deployment
Reduce risk of bad releases
1 percent of traffic to the new version, automatic rollback
Feature flags (kill switches)
Disable faulty features instantly
Turn off a new search ranker
Retry With Backoff and Jitter
import random
def backoff_delays(base=0.1, cap=2.0, attempts=5, seed=7):
rng = random.Random(seed)
return [round(rng.uniform(0, min(cap, base * 2 ** i)), 3) for i in range(attempts)]
print(backoff_delays()) # "full jitter": random delay up to an exponentially growing cap
# [0.032, 0.03, 0.26, 0.058, 0.857]
Retries without backoff or idempotency can turn a small outage into a retry storm: every client hammers the recovering service at once. Limit attempts, use jitter, retry only idempotent operations (or use idempotency keys), and prefer a retry budget (for example at most 10 percent extra load).
How Patterns Work Together
A product page calls pricing, inventory and recommendations:
Each call has a timeout.
Transient failures get one or two jittered retries.
A circuit breaker around recommendations opens when it is failing.
The fallback shows popular items instead.
Bulkheads ensure slow recommendations cannot use up threads needed for pricing.
Under overload, load shedding drops low-priority requests (bots, prefetches) first.
Trade-offs
More moving parts and configuration to tune.
Fallbacks can show stale or generic data.
Active-active and cells cost more infrastructure.
Patterns must be tested; an untested failover path often fails when needed.
Interview Tip
Group patterns into redundancy, isolation, recovery and degradation, give a concrete example of several working together, and highlight the retry storm risk with the backoff-and-jitter fix. That shows you understand how patterns interact, not just their names.
11
How does load balancing improve availability, and what are its pitfalls?
KnownRevise
Load balancing improves availability by spreading traffic across several healthy instances, removing failed instances automatically through health checks, and allowing maintenance and deployments without downtime (drain one instance at a time). With instances in several zones, losing one server or even one zone only reduces capacity. The pitfalls are that the load balancer itself can become a single point of failure, poorly designed health checks can remove all instances or keep broken ones, uneven distribution and sticky sessions can overload some nodes, and failover can overload the survivors if there is not enough spare capacity.
Read full answerHide full answer
Applies toL4 and L7 load balancers (AWS ALB and NLB, Google Cloud Load Balancing, NGINX, HAProxy, Envoy), Kubernetes Services and Ingress, DNS and global load balancing
How It Helps Availability
Mechanism
Effect
Distribution across instances
No single server handles all traffic
Health checks
Unhealthy instances stop receiving requests
Multi-zone targets
A zone failure only reduces capacity
Connection draining
Instances finish in-flight requests before removal during deploys
Autoscaling integration
New instances join automatically under load
Global load balancing
Route users to another region on regional failure
Pitfalls and Mitigations
Pitfall
What goes wrong
Mitigation
Load balancer as SPOF
A single NGINX or HAProxy box fails
Managed load balancers, redundant pairs with floating IPs, DNS across several LB nodes
Shallow health checks
/health returns 200 while the app cannot reach its database
Readiness checks that verify critical local dependencies
Over-deep health checks
Health check depends on a shared database; when it slows, every instance is marked unhealthy and removed
Keep checks local; do not fail health on shared dependency outages; use circuit breakers instead
Insufficient capacity
After a zone fails, remaining instances overload and fail too (cascade)
N+1 capacity per zone, load shedding, autoscaling headroom
Sticky sessions
Users pinned to one instance lose sessions when it fails; uneven load
Stateless services with external or token-based sessions
Uneven distribution
Round robin ignores slow or busy instances; long-lived connections (gRPC, WebSockets) stick to old instances
Least-outstanding-requests or least-connections algorithms, connection recycling, client-side balancing
Slow start
New instances with cold caches or JIT get full traffic immediately
Slow-start ramp-up
Retry amplification
LB retries plus client retries multiply load during an incident
Single retry layer, retry budgets, idempotency
DNS caching
Clients keep sending traffic to a failed endpoint
Short TTLs, anycast, health-checked DNS
Misconfigured timeouts
LB timeout shorter than the app’s, causing 502s or duplicate work
Align timeouts across layers
Example: Health Check Design
# Kubernetes probes
livenessProbe: # restart if the process is stuck
httpGet: { path: /livez, port: 8080 }
periodSeconds: 10
failureThreshold: 3
readinessProbe: # stop routing traffic while not ready
httpGet: { path: /readyz, port: 8080 }
periodSeconds: 5
failureThreshold: 2
/livez checks only that the process responds. /readyz checks that the instance has finished starting and can serve (for example its local connection pool is initialised), but it should not fail just because a shared dependency is slow, or every pod would be removed at once.
Algorithms
Algorithm
Best for
Round robin
Similar instances, short requests
Least connections or least outstanding requests
Varying request durations
Weighted
Mixed instance sizes, canary releases
Consistent hashing
Cache affinity or sharded state
Power of two random choices
Large fleets; avoids herding onto one “least loaded” node
Interview Tip
Explain how health checks, distribution and draining improve availability, then cover the pitfalls: LB as SPOF, health check design, capacity after failover, sticky sessions and retry amplification. The “deep health check removes every instance” example shows real operational experience.
12
What role do health checks play in keeping a system available?
KnownRevise
Health checks are periodic probes that tell load balancers, orchestrators and monitoring systems whether an instance or service can do its job. They keep a system available by removing broken instances from traffic, restarting stuck processes, holding back new instances until they are ready, and triggering failover of databases or regions. Their value depends on design: a check that is too shallow keeps broken instances in service, while one that is too deep (depending on shared databases) can remove every instance at once during a dependency problem.
Read full answerHide full answer
Applies toload balancers, Kubernetes probes, service discovery (Consul), DNS failover, database failover managers, and synthetic monitoring
Types of Health Checks
Type
Question
Action on failure
Example
Liveness
Is the process alive and not deadlocked?
Restart the container or process
Kubernetes livenessProbe on /livez
Readiness
Can this instance serve traffic now?
Stop routing traffic to it (do not restart)
Kubernetes readinessProbe, LB target health
Startup
Has slow initialisation finished?
Delay liveness checks until it has
Kubernetes startupProbe for apps with long warm-up
Dependency or deep check
Can it reach its critical dependencies?
Report degraded status; alert
/health/deep used by monitoring, not by the LB
Synthetic (external)
Can users complete key journeys?
Alert, or trigger DNS or regional failover
Probes from several regions logging in and searching
Example Endpoint Design
from http.server import BaseHTTPRequestHandler
STATE = {"started": True, "draining": False, "db_pool_ready": True}
def health_status(path):
if path == "/livez":
return 200 # process responds: alive
if path == "/readyz":
ok = STATE["started"] and not STATE["draining"] and STATE["db_pool_ready"]
return 200 if ok else 503 # route traffic only when ready
return 404
class Health(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(health_status(self.path))
self.end_headers()
print(health_status("/livez"), health_status("/readyz")) # 200 200
STATE["draining"] = True # e.g. during shutdown
print(health_status("/livez"), health_status("/readyz")) # 200 503
During graceful shutdown, readiness fails first so the load balancer stops sending new requests, then the process finishes in-flight work and exits.
Design Guidelines
Liveness should be cheap and local: do not include database checks, or a database outage restarts every pod and makes recovery slower.
Readiness reflects ability to serve: initialisation done, not draining, local resources ready.
Avoid cascading removal: if a shared dependency fails, all instances fail readiness together; prefer circuit breakers and degraded responses over marking everything unhealthy. Some load balancers “fail open” when all targets are unhealthy for this reason.
Use thresholds: require several consecutive failures before removal and several successes before return, to avoid flapping.
Set sensible timeouts and intervals: fast enough to detect failures within seconds, not so frequent that checks add load.
Secure and lightweight: health endpoints should not expose sensitive details publicly or do expensive work.
Measure from outside too: internal checks can all pass while DNS, TLS certificates or the CDN are broken; synthetic probes catch that.
Health Checks in Failover
Database failover managers (Patroni, Orchestrator, cloud Multi-AZ) use health checks plus a quorum to decide when to promote a replica, with care to avoid split brain.
DNS failover (Route 53, Cloudflare) switches traffic to a standby region when health checks fail.
Service discovery removes unhealthy instances from the registry.
Interview Tip
Distinguish liveness, readiness and startup checks, explain what each triggers, and stress the danger of deep checks causing cascading removal or restart loops. Mentioning graceful draining and external synthetic probes rounds out a strong answer.
13
What is the purpose of the circuit breaker pattern in distributed systems?
KnownRevise
A circuit breaker wraps calls to a remote dependency and stops sending requests when that dependency is failing, returning an error or fallback immediately instead. It protects availability in three ways: callers fail fast instead of piling up threads and connections waiting on timeouts, the failing service gets time to recover instead of being flooded, and failures do not cascade through the call chain. After a cool-down period, the breaker lets a few trial requests through and closes again if they succeed.
Read full answerHide full answer
Applies tomicroservices, API clients, database and cache clients, service meshes (Envoy outlier detection), and libraries such as Resilience4j, Polly (.NET), opossum (Node.js) and pybreaker (Python)
The Three States
State
Behaviour
Transition
Closed
Requests pass through; failures are counted
Opens when the failure rate or count crosses a threshold
Open
Requests fail immediately (or use a fallback)
After a wait period, moves to half-open
Half-open
A limited number of trial requests are allowed
Closes if they succeed; reopens if they fail
Example Implementation
import time
class CircuitBreaker:
def __init__(self, failure_threshold=3, recovery_seconds=2.0, clock=time.monotonic):
self.threshold, self.recovery, self.clock = failure_threshold, recovery_seconds, clock
self.state, self.failures, self.opened_at = "closed", 0, 0.0
def call(self, fn, fallback):
if self.state == "open":
if self.clock() - self.opened_at < self.recovery:
return fallback()
self.state = "half_open" # allow a trial request
try:
result = fn()
except Exception:
self._record_failure()
return fallback()
self.state, self.failures = "closed", 0
return result
def _record_failure(self):
self.failures += 1
if self.state == "half_open" or self.failures >= self.threshold:
self.state, self.opened_at = "open", self.clock()
now = [0.0]
cb = CircuitBreaker(clock=lambda: now[0])
healthy = [False]
def pricing():
if not healthy[0]:
raise TimeoutError("pricing timed out")
return "price: 499"
for _ in range(4):
print(cb.state, cb.call(pricing, lambda: "price unavailable"))
now[0] = 3.0; healthy[0] = True
print(cb.call(pricing, lambda: "price unavailable"), cb.state)
# closed price unavailable
# closed price unavailable
# closed price unavailable
# open price unavailable
# price: 499 closed
The fourth call never reaches the pricing service: the breaker is open. After the recovery period, a trial call succeeds and the breaker closes.
Why It Matters: Cascading Failures
Without a breaker, a slow dependency with a 2-second timeout causes every request thread to wait 2 seconds. Thread pools fill, the caller becomes slow too, and its own callers start timing out. The failure spreads upward. The breaker cuts this chain by failing fast.
Configuration Choices
Setting
Consideration
Failure threshold
Rate-based (for example 50 percent of the last 20 calls) is more robust than a fixed count
What counts as failure
Timeouts and 5xx errors, not 4xx client errors
Slow-call threshold
Treat calls slower than a limit as failures (Resilience4j supports this)
Open duration
Long enough for recovery, short enough to resume quickly
Half-open trial count
A few requests, to avoid flooding a recovering service
Scope
One breaker per dependency (and sometimes per endpoint or host)
Use With Other Patterns
Timeouts: the breaker needs failures to detect; timeouts produce them quickly.
Retries: retry only when the breaker is closed, with backoff and jitter; never retry against an open breaker.
Fallbacks: cached data, defaults, or a clear degraded message.
Bulkheads: limit concurrency per dependency.
Monitoring: alert on breaker state changes; open breakers reveal unhealthy dependencies.
Interview Tip
Explain the closed, open and half-open states, the cascading-failure problem it solves, and how it combines with timeouts, retries and fallbacks. Mentioning that failure thresholds should be rate-based and scoped per dependency shows practical experience.
14
Which key indicators would you monitor to ensure system reliability?
KnownRevise
To ensure reliability, monitor user-facing service level indicators first: the four golden signals of latency, traffic, errors and saturation, expressed as SLIs with SLOs and error budget burn rates. Then add dependency health (databases, caches, queues, third-party APIs), infrastructure resources, data correctness and durability signals (replication lag, backup success, data freshness), change-related signals (deploys, configuration changes, change failure rate), and operational metrics (incident count, MTTD, MTTR).
Read full answerHide full answer
Applies toweb services, APIs, data pipelines, databases and platform infrastructure
User-Facing Indicators (Most Important)
Indicator
What to measure
Example SLO
Availability / success rate
Share of valid requests not failing (5xx, timeouts)
99.9 percent over 30 days
Latency
p50, p95 and p99 per endpoint (not averages)
99 percent of checkout requests under 800 ms
Traffic
Requests per second, active users, messages per second
Used for capacity and anomaly detection
Errors
Error rate by type and endpoint
Alert on burn rate, not single errors
Saturation
How “full” the system is: CPU, memory, connections, thread pools, queue depth
Alert before limits, for example 80 percent of the connection pool
Error Budget Burn Rate
Alerting on how fast the error budget is being consumed catches serious problems quickly without noisy alerts:
slo = 0.999
budget = 1 - slo # 0.1 percent of requests may fail
error_rate_last_hour = 0.0144 # 1.44 percent failing right now
burn_rate = error_rate_last_hour / budget
hours_to_exhaust = 30 * 24 / burn_rate
print(f"burn rate {burn_rate:.1f}x, 30-day budget gone in {hours_to_exhaust:.0f} h")
# burn rate 14.4x, 30-day budget gone in 50 h
A burn rate of 14.4 sustained for one hour consumes 2 percent of a 30-day budget; Google’s SRE guidance uses this as a typical fast-burn paging threshold, paired with a slower-burn ticket alert.
Dependency and Data Indicators
Area
Indicators
Databases
Query latency, error rate, connections used, replication lag, deadlocks, disk space
Caches
Hit ratio, evictions, latency, memory usage
Queues and streams
Queue depth, consumer lag, age of oldest message, dead-letter count
Third-party APIs
Latency, error rate, rate-limit responses, circuit breaker state
Data correctness
Reconciliation mismatches, failed validations, data freshness for pipelines
Durability
Backup success and age, restore test results, replication health
Certificate expiry dates and DNS resolution health.
Change and Operational Indicators
Indicator
Why
Deploy and configuration change events
Most incidents follow a change; overlay them on dashboards
Change failure rate
Share of deployments causing incidents or rollbacks
Incident frequency and severity
Reliability trend over time
MTTD and MTTR
Detection and recovery speed
Alert volume and noise
Too many alerts lead to missed real ones
Good Practices
Measure at the edge (load balancer, API gateway) to see what users experience, plus synthetic probes for critical journeys.
Use percentiles and histograms, not averages.
Break down by endpoint, region, customer tier and version to find partial outages.
Correlate metrics, logs and traces via request IDs.
Alert on symptoms (SLO burn) and use cause-level metrics for diagnosis.
Interview Tip
Lead with the golden signals expressed as SLIs and SLOs, explain burn-rate alerting with numbers, then add dependency, data, infrastructure and change indicators. Saying you would page on user-facing symptoms rather than every CPU spike shows mature monitoring practice.
15
How do you build monitoring that accurately reflects system availability?
KnownRevise
Monitoring that accurately reflects availability measures what users experience: it defines SLIs as good events / valid events for key user journeys, collects them from the edge (load balancer or API gateway logs) and from outside (synthetic probes and real-user monitoring), excludes invalid traffic, breaks results down by region, endpoint and customer segment to catch partial outages, and compares them with SLOs and error budgets using burn-rate alerts. Internal host metrics remain useful for diagnosis, but they do not define availability.
Read full answerHide full answer
Applies toSRE and platform teams, SLO programmes, status pages and SLA reporting
Step 1: Define User Journeys and SLIs
Journey
SLI (good / valid)
Log in
Login requests returning success within 1 s / all valid login attempts
Search
Search requests returning non-5xx within 500 ms / all search requests
Checkout
Orders completed without server error / checkout attempts
API
Requests without 5xx or timeout / requests excluding 4xx client errors
Decide what “good” means precisely: status codes, latency thresholds, correctness checks.
Step 2: Measure at the Right Places
Source
Strength
Gap
Load balancer or gateway logs
Real traffic, closest to users inside your infrastructure
Misses DNS, CDN and client network failures
Synthetic probes from several regions
Works with low traffic; detects total outages and DNS or TLS issues
Tests only scripted paths
Real user monitoring (browser or mobile)
True end-to-end view
Noisy, affected by users’ own networks
Application metrics
Detailed per-endpoint and per-error data
Cannot see requests that never reached the app
Using at least two (edge logs plus synthetic probes) avoids blind spots.
Step 3: Compute SLIs and SLOs
# Availability SLI over 30 days from load balancer metrics (Prometheus)
sum(rate(http_requests_total{job="edge", code!~"5.."}[30d]))
/
sum(rate(http_requests_total{job="edge"}[30d]))
Exclude health checks and known synthetic traffic; decide how to treat 429s caused by your own overload (usually as failures).
Step 4: Alert on Burn Rate
Use multi-window, multi-burn-rate alerts:
Alert
Condition (99.9 percent SLO)
Action
Fast burn
Error rate above 14.4x budget over 1 hour and 5 minutes
Page on-call
Slow burn
Error rate above 1x budget over 3 days (often with a 6-hour short window)
Ticket
The short window confirms the problem is still happening, which reduces stale alerts.
Step 5: Make Partial Outages Visible
Break SLIs down by region, zone, endpoint, client version, tenant.
Track per-customer availability for large tenants; an overall 99.95 percent can hide one customer at 90 percent.
Use multiple synthetic locations to see regional or ISP issues.
Step 6: Avoid Common Distortions
Distortion
Fix
Measuring “servers up”
Measure successful requests instead
Averages across endpoints
Separate SLIs for critical journeys
Low-traffic periods
Combine with synthetic probes; use time-based SLIs for rarely used features
Retries hiding failures
Count first-attempt failures too, or measure from the client
Monitoring in the same failure domain
Run probes and alerting outside the production region or provider
Clock and aggregation errors
Consistent time windows; avoid averaging percentiles
Step 7: Report and Act
SLO dashboards with remaining error budget.
Weekly or monthly reliability reviews; error budget policies (slow releases when budget is spent).
A status page driven by the same SLIs for consistency.
Interview Tip
Describe the pipeline: define user-journey SLIs, measure at the edge and with synthetic probes, compute SLOs, alert on burn rates, and break down to catch partial outages. Pointing out that monitoring must run outside the failing system, and that averages hide outages, shows real experience.
16
Why do alerting and on-call rotations matter for system reliability?
KnownRevise
Alerting turns monitoring into action: it tells the right person quickly that users are affected or soon will be, which shortens time to detect and therefore time to restore. On-call rotations make sure someone qualified is always available to respond, while sharing the load fairly so engineers stay effective. Together they directly reduce MTTR and protect SLOs, but only if alerts are actionable, symptom-based and low-noise, and on-call is sustainable, with runbooks, escalation paths and follow-up work to remove recurring causes.
Read full answerHide full answer
Applies toSRE and DevOps teams, incident management, and tools such as PagerDuty, Opsgenie, Grafana OnCall, Alertmanager and incident.io
Why Alerting Matters
Benefit
Explanation
Faster detection
Incidents are found by monitoring, not by angry customers
Lower MTTR
Detection is the first phase of recovery
Protects error budgets
Burn-rate alerts trigger before SLOs are breached
Prevents escalation
Saturation alerts (disk filling, certificate expiring) allow fixes before outages
What Good Alerts Look Like
Symptom-based: page on user impact (SLO burn rate, failed checkouts), not on every CPU spike.
Actionable: the responder can do something; if not, it should be a ticket or dashboard.
Urgent only when urgent: severity levels: page for user impact now, ticket for slow burn or capacity trends.
With context: links to dashboards, runbooks, recent deploys and affected regions.
Deduplicated and grouped: one incident, one page, not 200.
Reviewed regularly: remove or fix alerts that fire without needing action.
# Prometheus alert on fast SLO burn (99.9 percent SLO)
- alert: CheckoutErrorBudgetFastBurn
expr: |
(sum(rate(http_requests_total{route="/checkout", code=~"5.."}[1h]))
/ sum(rate(http_requests_total{route="/checkout"}[1h]))) > (14.4 * 0.001)
and
(sum(rate(http_requests_total{route="/checkout", code=~"5.."}[5m]))
/ sum(rate(http_requests_total{route="/checkout"}[5m]))) > (14.4 * 0.001)
labels:
severity: page
annotations:
summary: "Checkout is burning its error budget about 14x faster than allowed"
runbook: "https://runbooks.internal/checkout-errors"
Why On-Call Rotations Matter
Aspect
Value
Coverage
Someone responds 24/7, including nights and weekends
Ownership
Teams that build a service also operate it (“you build it, you run it”), which motivates reliability improvements
Knowledge sharing
Rotations spread operational knowledge and avoid single human points of failure
Escalation
Primary and secondary on-call, plus clear escalation to experts and managers
Making On-Call Sustainable
Reasonable load: Google’s SRE book suggests no more than about two incidents per 12-hour shift so each one can be handled and followed up properly.
Rotation size: enough people that each person is on call only every few weeks; follow-the-sun across time zones where possible.
Compensation and recovery time after nights with pages.
Handoffs: summaries of ongoing issues at each shift change.
Training: shadowing, game days and incident simulations before taking primary on-call.
Runbooks for known failure modes, kept up to date.
Closing the Loop
Post-incident reviews produce action items that remove causes and noisy alerts.
Track metrics: pages per week, false-positive rate, time to acknowledge, MTTR.
Automate recurring manual fixes so they no longer need a human.
Anti-Patterns
Alert fatigue from noisy, non-actionable alerts, which leads to real alerts being missed.
Paging on causes (high CPU) rather than symptoms (users failing).
One “hero” who knows everything and is always called.
No follow-up: the same incident pages every week.
Interview Tip
Explain that alerting shortens detection and on-call ensures response, then describe good alert design (symptom-based, actionable, burn-rate) and sustainable rotations with runbooks and escalation. Mentioning alert fatigue and the post-incident loop shows operational maturity.
17
What steps would you take to respond to an incident that reduces availability?
KnownRevise
A good incident response follows a clear sequence: detect and declare the incident, assign roles (an incident commander coordinates), assess impact, mitigate first to restore users quickly (roll back, fail over, shed load, disable a feature), communicate regularly with stakeholders and customers, verify recovery, and then run a blameless post-incident review that produces concrete follow-up actions. The priority is stopping user impact, not finding the root cause during the outage.
Read full answerHide full answer
Applies toSRE, DevOps and on-call teams; production outages, partial degradations and severe latency incidents
Step-by-Step Response
1. Detect and declare
An SLO burn-rate alert, synthetic probe or customer report triggers the response.
Declare an incident early with a severity level; it is cheaper to downgrade than to start late.
Open a dedicated channel and an incident document with a timeline.
2. Assign roles
Role
Responsibility
Incident commander (IC)
Coordinates, decides priorities, keeps the team focused; usually does not debug
Operations / subject-matter experts
Investigate and apply mitigations
Communications lead
Status page, customer support, executives
Scribe
Records timeline, decisions and actions
3. Assess impact
What is broken: which journeys, regions, customers, and what percentage of requests?
When did it start? What changed around then (deploys, config, traffic, dependency incidents)?
4. Mitigate first
Mitigation
When to use
Roll back the latest deployment or config change
Impact started after a change
Disable a feature flag
A new feature is failing
Fail over to another zone or region
Infrastructure or regional failure
Scale out or shed load
Overload or traffic spike
Block abusive traffic, tighten rate limits
Bots, DDoS, a runaway client
Open circuit breakers, serve cached or degraded responses
Failing dependency
Restart or replace unhealthy instances
Stuck processes, memory leaks
Prefer reversible, low-risk actions; announce each action in the channel before taking it.
5. Communicate
Post status updates at a regular interval (for example every 30 minutes for a major incident), even if there is no new information.
Be factual: impact, what is being done, next update time.
Keep internal teams (support, sales, leadership) informed.
6. Verify recovery
Confirm SLIs have returned to normal, not just that the fix was applied.
Watch for recurrence; check queues, backlogs and data consistency.
Replay or reconcile work that failed during the incident if needed.
7. Close and review
Declare the incident resolved and record the end time.
Hold a blameless post-incident review within days: timeline, contributing factors, what went well, what was hard.
10:02 Burn-rate alert: checkout 5xx at 6 percent
10:04 Incident declared SEV-2; IC assigned; channel opened
10:08 Deploy of payment-service v412 at 09:58 identified as likely trigger
10:10 Rollback to v411 started (announced in channel)
10:16 Error rate back to 0.05 percent; monitoring for recurrence
10:30 Resolved; status page updated; review scheduled
Common Mistakes
Debugging the root cause for an hour instead of rolling back in five minutes.
Everyone trying fixes at once without coordination.
No communication, so support and customers are left guessing.
Skipping the review, so the same incident happens again.
Interview Tip
Walk through detect, declare, roles, impact, mitigation, communication, verification and review, and stress “mitigate first, root-cause later”. Giving a short example timeline with a rollback shows practical incident experience.
18
How can post-mortem analysis improve future reliability and availability?
KnownRevise
A post-mortem (post-incident review) improves reliability by turning an incident into lasting changes: it reconstructs what happened and why, identifies contributing factors (technical, process and organisational), and produces owned, prioritised action items that prevent recurrence, detect problems sooner, and recover faster. It works only when it is blameless, so people share honest details, and when actions are tracked to completion. Over time, post-mortems reveal patterns across incidents that guide investment in the biggest reliability risks.
Read full answerHide full answer
Applies toSRE and engineering teams after outages, near misses and significant degradations
What a Good Post-Mortem Contains
Section
Content
Summary
What happened, impact, duration in plain language
Impact
Users, requests, revenue, SLO and error budget consumed
Timeline
Detection, key decisions, mitigation and resolution, with times
Contributing factors
Technical triggers plus conditions that allowed impact (missing tests, weak alerts, unclear runbooks)
What went well
Practices worth keeping
What was difficult
Gaps in tooling, knowledge or communication
Action items
Specific, owned, prioritised, with due dates
Lessons learned
Insights for other teams
Blameless Culture
Focus on systems and conditions, not individual mistakes: “the deploy pipeline allowed an unreviewed config change to reach all regions at once” rather than “Sam pushed bad config”.
People who fear blame hide information, so root causes stay hidden.
Ask “how did the system make this easy to get wrong?”
Finding Contributing Factors
Incidents rarely have a single cause. Look at:
Trigger: the change or event (deploy, traffic spike, dependency failure).
Detection: how long until we knew, and why?
Mitigation: what slowed recovery (missing rollback, unclear ownership)?
Blast radius: why did it affect everyone instead of a subset?
Techniques include “five whys” (used carefully, since it can narrow to one cause), timeline analysis, and fault trees.
Turning Findings Into Improvements
Category
Example action
Prevent
Add validation for configuration changes; add tests for the bug class
Detect
Burn-rate alert for the affected journey; synthetic probe for login
Make actions SMART-like: specific, owned, with dates. Track them in the normal backlog and review completion.
Example
Incident: a configuration change set a cache TTL to 0, causing 40 minutes of database overload.
Findings: no validation of config values; change rolled out to all regions at once; alert fired only after 15 minutes; no quick config rollback.
Actions: schema validation and safe ranges for config; staged config rollout with automatic halt on SLO regression; burn-rate alert with a shorter window; config rollback command added to the runbook.
Learning Across Incidents
Tag incidents by cause category (change, capacity, dependency, data, human process).
Review trends quarterly: if 60 percent of incidents follow deploys, invest in safer deployments.
Share post-mortems widely so other teams learn without repeating the incident.
Review near misses too; they are cheaper lessons.
Pitfalls
Blame, which drives information underground.
Writing the document but never completing action items.
Action items that are too vague (“be more careful”).
Only reviewing large incidents.
Interview Tip
Describe the structure (timeline, impact, contributing factors, action items), stress blamelessness and tracking actions to completion, and give an example where the actions cover prevention, detection and mitigation. Mentioning trend analysis across incidents shows a strategic view.
19
How does system scalability affect availability?
KnownRevise
Scalability affects availability because most outages under load are capacity failures: when demand exceeds what the system can handle, latency rises, requests time out, queues grow and components fail, often cascading to others. A system that scales smoothly (adds capacity automatically, has no single bottleneck, and degrades gracefully when it cannot keep up) stays available through traffic spikes and component failures. Scaling also adds more components and coordination, which brings new failure modes, so it must be paired with good isolation and automation.
Read full answerHide full answer
Applies toweb services under traffic spikes, databases, queues and stream processors, autoscaling platforms and capacity planning
How Scalability Supports Availability
Scalability property
Availability benefit
Horizontal scaling (more instances)
Spikes are absorbed; losing one instance removes only a small share of capacity
Autoscaling
Capacity follows demand before saturation
No single bottleneck (sharding, partitioning)
One hot component does not cap the whole system
Stateless services
Instances can be added, removed or replaced quickly
Headroom (N+1, N+2)
Survives losing an instance or a zone at peak
Asynchronous processing
Queues absorb bursts instead of failing requests
How Poor Scalability Hurts Availability
Saturation: CPU, memory, connection pools or threads run out; latency climbs and requests time out.
Cascading failure: a slow database makes app threads wait; app servers exhaust threads; the load balancer marks them unhealthy; remaining servers get more traffic and fail.
Retry storms: clients retry timeouts and multiply load on an already overloaded system.
Hot spots: one popular key, tenant or partition overloads a single shard.
Slow scaling: autoscaling that takes several minutes to add capacity cannot handle sudden spikes.
Example: Zone Failure at Peak
A service needs 12 instances at peak. It runs 12 across three zones (4 each). Zone A fails: 8 instances remain and must take 150 percent of their normal load, so they overload.
With N+1 at the zone level, it runs 18 instances (6 per zone); after losing a zone, 12 remain and handle the peak.
import math
def instances_needed(peak_instances, zones, zones_lost=1):
per_zone = math.ceil(peak_instances / (zones - zones_lost))
return per_zone * zones
print(instances_needed(12, 3)) # 18 (6 per zone, survives one zone loss)
Scaling Without Hurting Availability
Load test to know real limits and bottlenecks; plan capacity from forecasts.
Autoscale on leading indicators (requests per instance, queue depth) rather than only CPU; keep a minimum warm capacity.
Rate limiting and load shedding to protect the system when demand exceeds even the scaled capacity.
Backpressure in queues and streams so producers slow down.
Caching and CDNs to cut origin load.
Partitioning and sharding for data, with care for hot keys.
Graceful degradation: disable expensive features under load.
Isolation: cells or bulkheads so one tenant’s spike does not take down everyone.
New Risks From Scaling
Risk
Mitigation
More components, more failures
Automation, self-healing, health checks
Distributed coordination
Avoid global locks; use partitioned designs
Autoscaling bugs or limits
Test scaling paths; set sensible minimum and maximum; watch cloud quotas
Dependency limits
Databases, third-party APIs and connection limits may not scale with your app
Interview Tip
Explain that capacity failures are a major cause of outages, show how horizontal scaling, headroom and autoscaling protect availability, and use the zone-failure capacity example. Mentioning cascading failures, retry storms and load shedding demonstrates real-world understanding.
20
Which strategies let a system scale while keeping or improving reliability?
KnownRevise
To scale without losing reliability, grow capacity in ways that do not add single points of failure or shared bottlenecks, and protect the system when demand outpaces capacity. Key strategies: stateless horizontal scaling with autoscaling, partitioning and sharding data, caching and CDNs, asynchronous processing with queues and backpressure, isolation by cells or bulkheads, rate limiting and load shedding, safe progressive deployments, capacity planning with load tests, and strong observability and automation.
Read full answerHide full answer
Applies togrowing web platforms, SaaS products, APIs, data pipelines and microservice architectures
Strategies and Their Reliability Effect
Strategy
How it scales
Reliability effect
Stateless services plus autoscaling
Add instances on demand
Instances are replaceable; failures affect a small share
Read replicas and caching
Offload reads from the primary database
Protects the primary from overload; cache can serve during brief outages
Sharding / partitioning
Split data and traffic by key
One shard failure affects only its users; no single hot database
Asynchronous queues
Decouple producers and consumers
Absorbs spikes; consumers retry safely with idempotency
Backpressure
Producers slow down when consumers lag
Prevents memory blow-ups and cascading overload
Cell-based architecture
Many independent copies of the stack, each serving a subset of customers
Limits blast radius of bugs, bad deploys and noisy tenants
Bulkheads
Separate resource pools per dependency or tenant
One slow dependency cannot exhaust all threads
Rate limiting and quotas
Cap usage per client or tenant
Protects shared capacity from abuse or runaway clients
Load shedding and prioritisation
Drop low-priority work under overload
Keeps critical traffic (checkout, login) working
CDN and edge caching
Serve static and cacheable content near users
Large share of traffic never reaches origin
Progressive delivery
Canary and staged rollouts per cell or region
Bad releases affect few users and roll back automatically
Capacity planning and load testing
Measure limits before users hit them
Fewer surprise saturation incidents
Example: Scaling an Order Service
Stateless API in three zones with autoscaling on requests per instance.
Orders written to a queue; workers process them idempotently (idempotency keys) and scale with queue depth.
Database sharded by customer ID; read replicas for order history.
Cache for product and price lookups with short TTLs.
Cells: customers split into cells of roughly equal load, each with its own API, workers and shard.
Load shedding: under overload, defer email notifications and analytics events, keep order placement working.
Canary per cell: deploy to one cell first and watch SLOs before continuing.
Guardrails While Scaling
Avoid shared global resources: one global lock, sequence generator or config database becomes a bottleneck and SPOF.
Know your dependencies’ limits: database connections, third-party API quotas, cloud service limits.
Connection management: connection poolers (PgBouncer) prevent thousands of instances from exhausting database connections.
Retry discipline: exponential backoff with jitter and retry budgets; more instances retrying naively can overload a recovering service.
Test failure at scale: chaos experiments, zone failure drills, load tests with failures injected.
Automate operations: provisioning, scaling and recovery should not depend on humans.
Measure the Effect
SLOs per cell and per region, not only global.
Saturation metrics: queue lag, connection pool usage, shard hot spots.
Change failure rate and MTTR, which should stay stable or improve as the system grows.
Interview Tip
Present scaling strategies alongside the reliability effect of each, give a concrete example combining queues, sharding, cells and load shedding, and warn about shared bottlenecks and retry storms. Mentioning cell-based architecture and progressive rollouts shows awareness of how large systems limit blast radius.
21
How does caching affect system reliability, and what are the trade-offs?
KnownRevise
Caching improves reliability by reducing load on databases and services, lowering latency, and allowing the system to serve responses even when the origin is slow or down (for example stale-if-error). But it introduces risks: stale or inconsistent data, cache stampedes when popular keys expire, dependence on the cache so that its failure overloads the origin, cold starts after restarts, and bugs from caching wrong or private data. Reliable designs treat the cache as an optimisation that can fail, with capacity to survive cache loss or controlled degradation.
Read full answerHide full answer
Applies toCDN and edge caches, Redis and Memcached, in-process caches, HTTP caching, and database query caches
How Caching Helps Reliability
Benefit
Explanation
Lower origin load
A 95 percent hit ratio means only 5 percent of reads reach the database
Burst absorption
Traffic spikes for popular content are served from cache
Lower latency
Fewer timeouts, more headroom
Resilience to origin failure
Serve stale content when the origin errors (stale-if-error, stale-while-revalidate)
Reduced dependency on slow services
Cache third-party API responses
Risks and Mitigations
Risk
What happens
Mitigation
Stale data
Users see old prices or profile data
Appropriate TTLs; invalidation on write; versioned keys; do not cache data needing strong consistency
Cache stampede (thundering herd)
A hot key expires; thousands of requests hit the database at once
Request coalescing, locks per key, probabilistic early refresh, TTL jitter
Cache as a hidden dependency
Cache cluster fails; database receives 20 times its normal load and collapses
Size the origin for some cache loss; rate limit misses; replicated cache clusters; local fallback caches
Cold cache
After restart or deploy, hit ratio is low; origin overloads
Warm-up, gradual traffic shift, persistent caches
Caching errors
A failed response is cached and served to everyone
Do not cache 5xx; short negative-cache TTLs
Caching private data
One user’s data shown to another
Correct cache keys and Cache-Control: private for per-user responses
Inconsistent invalidation
Some nodes have old data
Event-driven invalidation, short TTLs as a safety net
import threading, time
cache, locks = {}, {}
db_calls = [0]
def load_from_db(key):
db_calls[0] += 1
time.sleep(0.05)
return f"value-for-{key}"
def get(key, ttl=60):
entry = cache.get(key)
if entry and entry[1] > time.monotonic():
return entry[0]
lock = locks.setdefault(key, threading.Lock())
with lock: # only one thread reloads this key
entry = cache.get(key)
if entry and entry[1] > time.monotonic():
return entry[0]
value = load_from_db(key)
cache[key] = (value, time.monotonic() + ttl)
return value
threads = [threading.Thread(target=get, args=("home_page",)) for _ in range(50)]
for t in threads: t.start()
for t in threads: t.join()
print("database calls:", db_calls[0]) # database calls: 1
dict.setdefault is atomic in CPython for simple keys, so all threads share one lock per key. Distributed systems use the same idea with a short-lived lock in Redis or request coalescing in the CDN.
The CDN serves fresh content for 60 seconds, refreshes in the background for 30 more, and can serve stale content for up to a day if the origin returns errors.
Design Guidelines
Decide which data may be stale and for how long; keep balances, stock and permissions on strongly consistent paths.
Plan for cache failure: can the origin survive a cold or missing cache? If not, add protections.
Monitor hit ratio, latency, evictions, memory and origin load.
Use TTL jitter to avoid synchronised expiry.
Interview Tip
Explain both sides: caching protects origins and enables stale-if-error, but adds staleness, stampedes and a hidden dependency. Describing how you would survive a total cache loss and prevent stampedes shows reliability-focused thinking.
22
How does rate limiting help preserve system availability?
KnownRevise
Rate limiting preserves availability by capping how many requests a client, tenant or route can make in a time window, so that no single user, bug, bot or traffic spike can consume all capacity. Excess requests get a fast 429 Too Many Requests (with Retry-After) instead of slowing everyone down. It protects shared resources (databases, third-party quotas), ensures fairness between tenants, and stops retry storms and abuse from turning into outages. It works alongside load shedding, which drops work when the system as a whole is overloaded.
Read full answerHide full answer
Applies topublic APIs, API gateways (Kong, AWS API Gateway, Envoy, NGINX), login and signup endpoints, multi-tenant SaaS, and calls to third-party APIs
What It Protects Against
Threat
Example
Abusive or buggy clients
A misconfigured script calling an API 1,000 times per second
Noisy neighbours
One tenant’s batch job slowing down every other tenant
Brute force and credential stuffing
Rapid login attempts
Scrapers and bots
Aggressive crawling of search pages
Retry storms
Clients retrying failed requests without backoff
Cost and quota exhaustion
Overusing a paid third-party API
Common Algorithms
Algorithm
Behaviour
Notes
Fixed window
Count per minute; reset each minute
Simple; allows bursts at window edges
Sliding window log or counter
Smooths window edges
More accurate; moderately more state
Token bucket
Tokens refill at a steady rate; each request spends one; bucket size allows bursts
Most common for APIs
Leaky bucket
Requests flow out at a constant rate
Smooths traffic to a fixed rate
Concurrency limit
Caps in-flight requests rather than rate
Good for protecting slow back ends
Token Bucket Example
class TokenBucket:
def __init__(self, rate_per_s, capacity):
self.rate, self.capacity = rate_per_s, capacity
self.tokens, self.last = capacity, 0.0
def allow(self, now):
self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
self.last = now
if self.tokens >= 1:
self.tokens -= 1
return True
return False
bucket = TokenBucket(rate_per_s=5, capacity=10)
burst = [bucket.allow(0.0) for _ in range(12)]
print(burst.count(True), "allowed in burst") # 10 allowed in burst
print(bucket.allow(0.2), bucket.allow(0.2)) # True False (one token refilled in 0.2 s)
Where to Apply
Edge / API gateway: per IP, API key or user, before requests reach services.
Per tenant: quotas aligned with plans; prevents noisy neighbours.
Per endpoint: stricter limits for expensive operations (search, exports, login).
Outbound: limit calls to databases or third-party APIs to stay within their capacity and quotas.
Distributed enforcement: shared counters in Redis or gateway-level limits so all instances agree; local limits as a fallback if the shared store fails.
Good Practices
Return 429 with Retry-After and rate-limit headers so well-behaved clients back off.
Fail open or closed deliberately if the rate limiter’s store is unavailable (often fail open with conservative local limits).
Prioritise traffic: keep limits looser for critical operations and authenticated customers.
Monitor rejected requests by client to spot abuse and misconfigured limits.
Combine with load shedding: rate limits protect against individual clients; load shedding protects against total overload even when every client is within its limit.
Trade-offs
Limits that are too strict block legitimate users; too loose and they do not protect.
Distributed counting adds latency and its own dependency.
IP-based limits can affect many users behind one NAT or proxy.
Interview Tip
Explain how rate limiting stops any single client from exhausting shared capacity, describe token bucket with a short example, and cover where to enforce limits and how clients should respond to 429s. Distinguishing rate limiting from load shedding shows a precise understanding.
23
How do eventual and strong consistency differ, and how do they affect reliability?
KnownRevise
Strong consistency means every read returns the latest committed write, as if there were one copy of the data. Eventual consistency means replicas may temporarily return older values, but they converge once updates stop. For reliability, the difference is a trade-off between correctness and availability or latency: strong consistency prevents stale or conflicting data but requires coordination, so requests can fail or slow down when replicas are unreachable; eventual consistency keeps serving through failures and is faster, but the application must tolerate staleness and resolve conflicts, or users see wrong results. The partition case is the CAP trade-off.
Cassandra at ONE, DNS, CDN caches, async read replicas
Reliability Implications
Strong consistency
Correctness reliability is high: balances, inventory and permissions cannot be wrong due to replication lag.
Availability risk: if a majority of replicas or the leader is unreachable, writes (and strong reads) fail.
Latency risk: synchronous replication across regions adds round trips, increasing tail latency and timeouts.
Failover care: leader elections cause brief unavailability.
Eventual consistency
Availability is high: any replica can serve, so the system rides out replica and network failures.
Correctness risk: stale reads (a user does not see their own update), lost updates under last-writer-wins, oversold stock if used for invariants.
Hidden failures: replication lag can grow silently; monitor it.
Recovery work: repairs and reconciliation after failures.
Example: Read Replica Lag
A user updates their profile (written to the primary) and the next page reads from an asynchronous replica 2 seconds behind. They see their old name and think the update failed, then submit again.
Fixes: read-your-writes (read from the primary for a short time after a write, or pass a version token), monitoring replication lag, and routing critical reads to the primary.
Choosing Per Data Type
Data
Recommended
Account balances, payments, stock reservations
Strong
Authentication, permissions changes
Strong (or short, bounded staleness)
Product catalogue, reviews, feeds
Eventual, with reasonable TTLs or lag
Analytics, counters, likes
Eventual (CRDT counters if multi-writer)
User’s own recent edits
Session guarantees (read-your-writes) on top of eventual
Middle Ground
Bounded staleness: reads at most N seconds old (Azure Cosmos DB offers this level).
Causal consistency and session guarantees remove most confusing anomalies without full coordination.
Per-request choice: tunable consistency in Cassandra, DynamoDB and MongoDB.
Interview Tip
Define both models, then focus on the reliability trade-off: strong consistency protects correctness but can reduce availability and raise latency; eventual consistency stays up but risks stale or conflicting data. Choosing per data type and mentioning read-your-writes for replica lag shows practical judgement.
24
What is the CAP theorem, and how does it relate to system availability?
KnownRevise
The CAP theorem says that when a network partition occurs, a replicated system must choose between consistency (every read sees the latest write) and availability (every non-failed node responds without error). For availability engineering, the key lesson is that a strongly consistent design will reject some requests during partitions (usually on the minority side), while an availability-first design keeps serving but may return stale or conflicting data. CAP describes only partition behaviour; practical availability also depends on redundancy, failover speed, capacity and operations, which is why CP systems can still reach very high uptime.
Read full answerHide full answer
Applies toreplicated databases, multi-zone and multi-region designs, and choosing data stores for availability targets
CAP in One Scenario
Three replicas across zones A, B and C; zone C is cut off from the others.
Design
Zone C during the partition
Zones A and B
Consistency-first (CP), e.g. etcd, CockroachDB, MongoDB with majority concerns
Rejects writes and strong reads
Continue normally (they hold a majority)
Availability-first (AP), e.g. Cassandra at ONE or LOCAL_QUORUM per DC
Continues serving, possibly stale
Continue; replicas reconcile after healing
Partitions cannot be ruled out in a distributed system, so the choice is about behaviour during them, not about “picking two of three”.
Why It Matters for Availability Targets
CP systems lose availability only for clients that cannot reach the majority. With three zones and clients routed to healthy zones, users rarely notice, so CP systems commonly meet 99.99 percent targets in practice.
AP systems keep every side writable, which helps when users are pinned to an isolated region (multi-region active-active, mobile and offline clients).
PACELC adds the normal-case trade-off: strong consistency across regions costs latency on every write, which affects timeouts and perceived reliability.
Choose the partition behaviour per class: refuse (CP) or serve-and-reconcile (AP).
Place replicas across three failure domains so a single failure leaves a majority for CP data.
Route users to the healthy side with load balancers and health-checked DNS.
Degrade gracefully for CP data on the minority side: read-only mode, queued operations, clear error messages.
Plan reconciliation for AP data: merges, CRDTs, last-writer-wins only where acceptable.
CAP Is Not the Whole Availability Story
Most real outages come from bad deploys, configuration errors, overload and dependency failures, not partitions. Availability engineering also needs:
Redundancy and automated failover.
Safe deployments and rollback.
Capacity headroom, rate limiting and load shedding.
Timeouts, retries with backoff and circuit breakers.
Observability, SLOs and incident response.
Interview Tip
State CAP precisely (consistency vs availability during a partition), show the three-zone scenario, and explain that CP systems still achieve high practical availability because they serve the majority side. Adding that most outages are not partitions keeps the answer grounded in real operations.
25
How does quorum-based decision making affect reliability in distributed systems?
KnownRevise
Quorum-based decision making requires a minimum number of nodes (usually a majority) to agree before an action, such as committing a write or electing a leader, takes effect. It improves reliability by tolerating failures without losing committed data (a cluster of 2f + 1 nodes survives f failures), preventing split brain (two disjoint majorities cannot exist), and keeping replicas consistent. The cost is that progress stops when a quorum is unreachable, writes need extra round trips, and clusters must be sized and placed carefully across failure domains.
Read full answerHide full answer
Applies toconsensus systems (Raft in etcd, Consul, CockroachDB; ZAB in ZooKeeper; Paxos in Spanner), replicated databases with quorum reads and writes (Cassandra, Riak), Kafka's in-sync replicas, and failover managers
A write acknowledged by a majority survives the loss of any minority
Split-brain prevention
Only one side of a partition can hold a majority, so only one leader can commit
Consistent leader election
A new leader must win a majority vote and has the latest committed entries (Raft’s election restriction)
Read/write overlap
With R + W > N, a read quorum overlaps the latest write quorum
Cluster Sizing
def tolerated_failures(n):
return (n - 1) // 2
for n in (2, 3, 4, 5, 7):
print(f"{n} nodes: quorum {n // 2 + 1}, tolerates {tolerated_failures(n)} failure(s)")
# 2 nodes: quorum 2, tolerates 0 failure(s)
# 3 nodes: quorum 2, tolerates 1 failure(s)
# 4 nodes: quorum 3, tolerates 1 failure(s)
# 5 nodes: quorum 3, tolerates 2 failure(s)
# 7 nodes: quorum 4, tolerates 3 failure(s)
A 4-node cluster tolerates no more failures than 3 but needs a larger quorum, so odd sizes are standard. Two nodes tolerate zero failures for writes, which is why two-node “HA” pairs need a witness or arbiter.
Placement Matters
Three zones, one node each: losing any zone keeps a majority.
Two zones (2 + 1 nodes): losing the zone with 2 nodes stops the cluster. Add a third site or a lightweight witness.
Multi-region: five nodes across three regions (2 + 2 + 1) survive a full region loss, at the cost of cross-region latency on writes.
Effects on Availability and Latency
Unavailability when quorum is lost: if a majority fails or is partitioned away, writes stop. This is deliberate: refusing writes is better than risking split brain or lost data (the consistency-first choice during partitions, as in CAP).
Latency: every commit waits for the slowest member of the fastest majority; in multi-region clusters this adds tens to over a hundred milliseconds.
Leader dependence: in Raft, a leader failure triggers an election that pauses writes briefly (often under a few seconds with default timeouts).
Reliability Pitfalls
Pitfall
Mitigation
Even-sized or badly placed clusters
Odd sizes across independent failure domains
Too many voters (slow commits)
Keep voters at 3 or 5; use non-voting learners or read replicas for scale
Correlated failures (same rack, same bad config)
Diverse placement; staged config rollouts
Disk fsync latency
Fast, dedicated disks for consensus logs (etcd is very sensitive to this)
Clock- or timeout-tuning errors
Election timeouts matched to network latency
Operating without quorum awareness
Monitoring for leader changes, quorum health and member lag
Sloppy quorums (Dynamo-style)
Understand that they trade consistency for availability during failures
Kafka Example
With replication.factor=3, min.insync.replicas=2 and producers using acks=all, a write is acknowledged only when at least two in-sync replicas have it. Losing one broker loses no acknowledged data; losing two makes the partition reject writes rather than risk loss.
Interview Tip
Explain the 2f + 1 rule with a sizing table, show how majorities prevent split brain, and discuss placement across three failure domains. Being clear that losing quorum stops writes by design, and that quorums add latency, gives a balanced answer.
26
What role do distributed transactions play in reliability, and what are the challenges?
KnownRevise
Distributed transactions keep data consistent across several databases, partitions or services: either all participants commit or none do, so a failure halfway through does not leave money debited but not credited. They support reliability by preserving invariants despite crashes and network failures. The challenges are significant: protocols like two-phase commit (2PC) add latency, hold locks while waiting, can block if the coordinator fails, and reduce availability because every participant must be up. Many modern systems therefore use sagas, the transactional outbox and idempotent operations across services, and keep true distributed transactions inside databases designed for them (Spanner, CockroachDB, YugabyteDB).
Read full answerHide full answer
Applies tomicroservices, sharded databases, payment and order processing, message brokers with transactional guarantees
Approaches
Approach
How it works
Reliability profile
Two-phase commit (2PC, XA)
Coordinator asks all participants to prepare, then commit
Atomic, but blocking if the coordinator fails after prepare; every participant must be available
2PC whose coordinator and participants are each replicated with Paxos or Raft
Survives node failures without blocking; costs latency
Saga
Sequence of local transactions; failures trigger compensating actions
Highly available; eventual consistency; intermediate states are visible
Transactional outbox
Write the state change and an event in one local transaction; a relay publishes the event
Reliable event publishing without 2PC between database and broker
Kafka transactions
Atomic writes to multiple partitions plus consumer offset commits
Exactly-once processing within Kafka
Two-Phase Commit in Brief
Prepare: coordinator asks each participant to promise it can commit; participants write prepare records and hold locks.
Commit: if all vote yes, coordinator records the decision and tells everyone to commit; otherwise abort.
Problem: if the coordinator crashes after participants have prepared, they cannot decide alone and keep holding locks until it recovers. Replicating the coordinator (as consensus-based databases do) removes this blocking.
Saga Example: Order Placement
1. Order service: create order (PENDING)
2. Payment service: charge card -> on failure: cancel order
3. Inventory service: reserve stock -> on failure: refund payment, cancel order
4. Order service: mark order CONFIRMED
Each step is a local transaction. Failures run compensations in reverse. Sagas can be orchestrated (a coordinator such as Temporal, AWS Step Functions or Camunda drives the steps) or choreographed (services react to events).
Challenges
Challenge
Detail
Mitigation
Latency
Several network round trips per transaction
Keep transactions small; co-locate related data
Availability
2PC needs every participant up
Sagas across services; consensus-replicated participants
Some actions cannot be undone (sent email, shipped parcel)
Order steps so irreversible ones come last; compensating business actions (refunds)
Heterogeneous systems
Not every system supports XA
Outbox, sagas, idempotent consumers
Observability
Hard to follow multi-service flows
Distributed tracing with correlation IDs; saga state dashboards
Choosing
Within one database (even sharded): use its transactions; distributed SQL handles cross-shard atomicity.
Across microservices: prefer sagas plus outbox plus idempotency; avoid 2PC across service boundaries.
Database plus message broker: transactional outbox or change data capture (Debezium).
Interview Tip
Explain why atomicity across participants protects invariants, describe 2PC and its blocking problem, and present sagas and the outbox as the reliable alternatives across services. Mentioning that consensus-based databases make 2PC non-blocking shows depth.
27
What is a disaster recovery plan, and how does it relate to reliability?
KnownRevise
A disaster recovery (DR) plan is a documented, tested strategy for restoring systems and data after a major failure, such as a region outage, data centre fire, ransomware attack, accidental mass deletion or corrupted database. It defines recovery objectives (RPO: how much data loss is acceptable; RTO: how long recovery may take), the architecture that meets them (backups, replicas, standby regions), step-by-step procedures, roles and communication, and a testing schedule. It complements everyday high availability: HA handles routine failures automatically, while DR handles rare, large-scale events that HA cannot absorb.
Read full answerHide full answer
Applies toproduction systems, databases, cloud infrastructure, regulated industries (finance, healthcare) and business continuity planning
Key Objectives
Term
Meaning
Example
RPO (recovery point objective)
Maximum acceptable data loss, measured in time
5 minutes: you may lose up to 5 minutes of writes
RTO (recovery time objective)
Maximum acceptable time to restore service
1 hour: service must be back within an hour
MTD (maximum tolerable downtime)
Business limit before serious harm
4 hours for an online store
DR Strategies (Cost vs Speed)
Strategy
Description
Typical RPO
Typical RTO
Cost
Backup and restore
Backups stored in another region; rebuild on disaster
Hours (or less with continuous log shipping)
Hours to days
Low
Pilot light
Core data replicated; minimal infrastructure ready to scale
Minutes
Tens of minutes to hours
Medium
Warm standby
Scaled-down full environment running in another region
Seconds to minutes
Minutes
Higher
Multi-site active-active
Full capacity in several regions serving traffic
Near zero
Near zero to minutes
Highest
These categories follow AWS’s widely used DR guidance; exact numbers depend on implementation.
What a DR Plan Contains
Scope and priorities: which systems are critical, their RPO and RTO, and dependencies.
Disaster scenarios: region loss, data corruption, ransomware, provider account compromise, key personnel unavailable.
Architecture: backups (with immutable or offline copies), cross-region replication, infrastructure as code for rebuilding.
Runbooks: precise steps for failover and restore, including DNS changes, database promotion, secrets and configuration.
Roles and decision rights: who declares a disaster and authorises failover.
Communication: internal escalation, customer notifications, regulators if required.
Testing schedule: restore tests, failover drills, game days.
Failback plan: how to return to the primary site safely.
Relationship to Reliability
High availability
Disaster recovery
Handles common failures (instance, disk, zone)
Handles rare, catastrophic events
Automatic, seconds to minutes
Often partly manual, minutes to hours
Redundancy within a region
Copies in another region, account or offline
Protects uptime
Protects business survival and data
Replication alone is not DR for data corruption: a bad migration or ransomware encryption replicates to every replica. Point-in-time backups and immutable copies are required.
Testing Is Essential
Restore tests: regularly restore backups to a clean environment and verify data integrity and timing.
Failover drills: switch to the DR region in a controlled exercise; measure actual RTO and RPO.
Tabletop exercises: walk through scenarios with the team.
Untested plans often fail due to missing permissions, outdated runbooks, expired credentials or forgotten dependencies.
Interview Tip
Define DR with RPO and RTO, describe the strategy spectrum from backup-and-restore to active-active, and explain how DR complements HA. Pointing out that replication does not protect against corruption or ransomware, and that only tested plans count, shows real experience.
28
How do backup and restore operations affect system availability?
KnownRevise
Backups affect availability in two directions. During normal operation, taking backups can reduce performance or require downtime if done poorly (locking tables, saturating disk I/O or network). During a disaster or data corruption, restore speed determines how long the system stays down, so RTO is often limited by how quickly you can restore, and RPO by how often backups or logs are captured. Well-designed backup strategies use online, incremental and snapshot-based backups, run from replicas, keep point-in-time recovery, and are regularly restore-tested, so they protect data without harming uptime.
Read full answerHide full answer
Applies todatabases (PostgreSQL, MySQL, MongoDB), file and object storage, virtual machine and volume snapshots, Kubernetes state, and SaaS data
Back up from a replica; throttle bandwidth; schedule during low traffic
Large full backups
Long windows; heavy network usage
Incremental or differential backups; continuous WAL or binlog archiving
Snapshot consistency
Crash-inconsistent copies of running databases
Use database-aware snapshot procedures or freeze briefly
Backup storage failure
Backups silently fail
Monitoring, alerts on failed or missing backups
Impact During Recovery
RTO: restoring a multi-terabyte database from object storage can take hours. Faster options include restoring from snapshots, keeping a delayed replica, or failing over to a standby.
RPO: daily backups mean up to a day of data loss; continuous log archiving enables point-in-time recovery to within seconds or minutes.
Partial restores: restoring one table or tenant is often faster than a full restore; design for it.
Dependencies: restored data must match application versions, schemas and encryption keys.
def restore_hours(size_tb, throughput_mb_s):
seconds = size_tb * 1024 * 1024 / throughput_mb_s
return seconds / 3600
for mb_s in (200, 1000):
print(f"5 TB at {mb_s} MB/s: {restore_hours(5, mb_s):.1f} h")
# 5 TB at 200 MB/s: 7.3 h
# 5 TB at 1000 MB/s: 1.5 h
Estimates like this, checked against real restore tests, show whether backups alone can meet the RTO or a standby is required.
Good Backup Practices
3-2-1 rule: at least three copies, on two different media or services, with one off-site (another region or account).
Immutable or air-gapped copies (object lock, separate accounts) to survive ransomware and accidental deletion.
Encryption at rest and in transit, with keys stored and backed up separately.
Retention policies that meet business and legal requirements.
Point-in-time recovery via WAL or binlog archiving.
Automate and monitor: alerts when a backup fails or is older than expected.
Test restores regularly: measure real restore time and verify data integrity; a backup that has never been restored is unproven.
Backups vs Replication
Aspect
Replication
Backups
Protects against hardware or zone failure
Yes, with fast failover
Yes, slowly
Protects against corruption, bad migrations, deletion, ransomware
No: errors replicate
Yes: restore to before the error
Recovery speed
Seconds to minutes
Minutes to hours
Both are needed: replicas for availability, backups for recoverability.
Interview Tip
Explain how backups can hurt performance (and how to avoid it with online backups from replicas) and how restore speed sets RTO while backup frequency sets RPO. Stressing restore testing, immutable copies and the difference between backups and replication shows operational maturity.
29
Why is data replication important for high availability, and what are its challenges?
KnownRevise
Data replication keeps copies of data on several nodes, often across zones or regions, so that when one node or location fails, another copy can serve reads and take over writes. It is central to high availability because stateless services can be replaced instantly, but data must already exist elsewhere for the system to continue. The challenges are replication lag and stale reads, data loss on failover with asynchronous replication, write latency with synchronous replication, split brain and conflicting writes, failover complexity, and the fact that replication copies mistakes too.
Point-in-time backups and delayed replicas to recover from bad deletes or migrations
Cross-region bandwidth and cost
Compress, filter, replicate only what is needed
Schema changes
Backward-compatible migrations applied in a safe order
Monitoring blind spots
Alerts on lag, replica health, failed replication slots, disk usage on primaries
Example: PostgreSQL HA Setup
Primary in zone A, synchronous standby in zone B (synchronous_commit = on, quorum of one standby), asynchronous replica in zone C for reads and extra safety.
Patroni with etcd manages leader election and fencing.
Clients connect through a proxy or DNS name that follows the leader.
WAL archiving to object storage for point-in-time recovery.
A zone A failure leads to automatic promotion of the zone B standby with no acknowledged data loss; zone C keeps serving reads.
Interview Tip
Explain why stateful data needs replication for HA, compare synchronous and asynchronous trade-offs (latency vs RPO), and cover lag, split brain, conflicts and the “replication copies mistakes” problem. A concrete setup with automated failover and backups shows practical depth.
30
How would you plan a failover strategy for a multi-region deployment?
KnownRevise
A multi-region failover plan starts from business objectives (RTO, RPO, which features must survive) and then defines the topology (active-passive warm standby or active-active), how data is replicated to the other region, how failure is detected and who decides to fail over, how traffic is moved (DNS, global load balancer, anycast), how dependencies and capacity are prepared in the target region, and how to fail back. Above all, the plan must be automated where safe, documented, and regularly tested, because untested failover usually fails when needed.
Read full answerHide full answer
Applies tocloud deployments on AWS, GCP and Azure, global SaaS platforms, payment and e-commerce systems with regional disaster requirements
Step 1: Set Objectives
Item
Example
RTO
Service restored within 15 minutes of a regional outage
RPO
At most 1 minute of data loss for orders; zero for payments
Scope
Checkout, login and search must survive; reporting can wait
Traffic
Survive full peak traffic in the remaining region
Step 2: Choose a Topology
Topology
Description
Trade-offs
Active-passive (warm standby)
Region A serves; Region B runs scaled-down copies with replicated data
Lower cost and simpler data model; failover takes minutes; standby may be under-tested
Active-active
Both regions serve users
Fast failover and continuous testing; data must handle multi-region writes or have home regions per record; higher cost
Pilot light
Region B has data replicas and minimal infrastructure
Cheapest standby; slower RTO
Step 3: Data Strategy
Databases: cross-region replicas (Aurora Global Database, Cloud SQL cross-region replicas, distributed SQL across regions). Asynchronous replication means RPO equals lag; monitor it against the target.
Critical zero-loss data: synchronous or consensus-based replication across regions (higher write latency), or write payments to a ledger designed for multi-region durability.
Active-active data: assign each user or tenant a home region for writes, or use conflict-tolerant designs (CRDTs, last-writer-wins only where acceptable); the consistency choices during inter-region partitions follow the CAP trade-off.
Object storage, queues, search indexes, caches: decide which to replicate and which to rebuild.
Secrets, configuration and keys must exist in both regions.
Step 4: Detection and Decision
Health checks and SLO burn rates measured from outside the failing region.
Decide between automatic failover (fast but risky with false positives) and human-approved failover with a clear decision owner. Many teams automate stateless traffic shifts and require approval for database promotion.
Guard against split brain: fence the old primary (revoke its write access) before promoting the new one.
Step 5: Traffic Shifting
Mechanism
Notes
DNS failover (Route 53, Cloud DNS, Cloudflare)
Simple; limited by TTLs and resolver caching; use 30 to 60 second TTLs
Global load balancers / anycast (Google Cloud LB, AWS Global Accelerator, Cloudflare)
Faster, health-checked shifting
Client-side region selection
Mobile apps or SDKs with fallback endpoints
Step 6: Capacity and Dependencies
The surviving region must handle full traffic: pre-provision, or ensure autoscaling and cloud quotas allow rapid growth.
Avoid hidden cross-region dependencies (for example a global service whose control plane lives in the failed region).
Step 7: Runbook
1. Confirm regional impact (external probes, provider status, SLO dashboards)
2. Incident commander approves failover
3. Freeze deploys; fence primary database in Region A
4. Promote replica in Region B; verify replication position vs RPO
5. Scale Region B to full capacity
6. Shift traffic (global LB weights or DNS)
7. Verify SLIs; communicate status
8. Reconcile any writes accepted in Region A after the last replicated point
9. Plan failback once Region A is healthy and resynchronised
Step 8: Test
Scheduled regional failover drills (quarterly or more often), ideally in production with controlled traffic.
Chaos experiments: block inter-region traffic, fail a region’s database.
Measure actual RTO and RPO and fix gaps.
Interview Tip
Structure the answer as objectives, topology, data, detection, traffic, capacity, runbook and testing. Highlighting RPO and replication lag, split-brain fencing, full-capacity planning and regular drills shows you understand what makes multi-region failover actually work.