Skip to content
System Design

Availability & Reliability interview questions and answers

30 questions, 22 with code. About 2 h 2 min to read them all.

By TestLab · Updated

Plan for an interview
30 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is the difference between availability and reliability in software systems?

Availability is the share of time (or requests) in which a system is up and able to serve users, for example 99.9 percent of requests succeeding over a month. Reliability is the probability that the system performs correctly, without failure, over a period of time under stated conditions, often expressed through mean time between failures (MTBF) and error rates. A system can be highly available but unreliable (it is always up but often returns wrong results or fails briefly many times), or reliable but less available (it rarely fails, but each outage lasts a long time). Durability, a third related property, is whether stored data survives failures.

Read full answer

Applies toweb services, APIs, databases, storage systems and the SLOs used to manage them

Side-by-Side Comparison

Aspect Availability Reliability Durability
Question it answers Can I use it right now? Does it keep working correctly over time? Will my data still be there?
Typical measure Uptime percentage, successful-request ratio MTBF, failure rate, error rate, correctness Probability of data loss per year
Improved by Redundancy, fast failover, low MTTR Testing, fault prevention, quality, redundancy Replication, backups, checksums, erasure coding
Example target 99.95 percent of requests succeed Fewer than 1 failure per month 99.999999999 percent annual object durability (Amazon S3’s design target)

Relationship Through MTBF and MTTR

A common steady-state formula links them:

Availability = MTBF / (MTBF + MTTR)

  • MTBF (reliability): how long the system usually runs between failures.

  • MTTR (repair speed): how long it takes to restore service.

So availability depends on both how often you fail and how quickly you recover.

def availability(mtbf_hours, mttr_hours):
    return mtbf_hours / (mtbf_hours + mttr_hours)

# Fails rarely but takes long to fix
print(f"{availability(1000, 10):.4%}")    # 99.0099%
# Fails often but recovers in minutes
print(f"{availability(100, 0.05):.4%}")   # 99.9500%

The second system is less reliable (fails ten times as often) but more available, because it recovers quickly.

Examples

  • Available but unreliable: an API that is always reachable but returns intermittent 500 errors or occasionally wrong data. Users can reach it, but cannot trust it.

  • Reliable but less available: a batch system that rarely fails but is down for a planned four-hour maintenance window every week.

  • Available but not durable: a cache that is always up but loses data on restart (acceptable for a cache, not for a database).

Why the Distinction Matters

  • Different engineering levers: availability improves most with redundancy and fast recovery; reliability improves with correctness, testing, safe deployments and fault prevention.

  • Different SLOs: an availability SLO (successful requests ratio) and a correctness or latency SLO may both be needed.

  • User perception: users care about both “is it up?” and “does it work correctly when it is up?”.

Measuring Each

  • Availability: successful requests divided by total valid requests, or minutes in which health checks pass.

  • Reliability: incident frequency, error rates, failed transactions, data correctness checks, MTBF.

  • Durability: replication and backup verification, restore tests, checksum failures.

Interview Tip

Define availability as “up and serving” and reliability as “working correctly over time”, link them through MTBF and MTTR, and give an example of a system that has one but not the other. Mentioning durability as a separate property makes the answer complete.

2

How do you define system availability, and what do you measure for it?

System availability is the proportion of time, or of requests, in which the system successfully serves users. The modern, user-focused definition is request-based: availability = successful valid requests / total valid requests over a window such as 30 days. The key components are SLIs (what you measure, such as the success ratio of an endpoint), SLOs (the target, such as 99.9 percent over 30 days), the error budget (the allowed failure, 0.1 percent), and the SLA (a contractual promise, usually looser than the SLO). Measurement should reflect what users experience, from load balancer logs or synthetic probes, not just whether servers are running.

Read full answer

Applies toweb services, APIs, internal platforms, databases and anything with uptime commitments

Two Ways to Define It

Approach Formula Strengths Weaknesses
Time-based Uptime / (uptime + downtime) Simple, good for whole-system outages Misses partial outages; what counts as “down”?
Request-based Successful requests / valid requests Reflects user experience; captures partial failures Low traffic windows give noisy numbers

Key Components

Component Meaning Example
SLI (indicator) A measured ratio of good events to valid events Share of GET /checkout requests returning non-5xx within 1 second
SLO (objective) Internal target for the SLI 99.9 percent over a rolling 30 days
Error budget 100 percent minus SLO 0.1 percent of requests, about 43 minutes of full outage per 30 days
SLA (agreement) External, contractual commitment with penalties 99.5 percent monthly, with service credits if missed

What Counts as “Good”

Define precisely:

  • Success: HTTP status not 5xx (and some 429s if caused by your own overload), correct response body.

  • Latency: a request slower than a threshold (for example 1 second) is often counted as failed for availability purposes.

  • Valid requests: exclude client errors (4xx due to bad input) and health-check traffic.

Where to Measure

Source What it shows
Load balancer or API gateway logs Real user requests, status codes and latency; closest to the user
Application metrics Per-endpoint success, error types
Synthetic probes from several regions Detects outages even with no traffic, including DNS and network problems
Real user monitoring (browser or app) End-to-end experience including CDN and client network

Example Calculation

total_requests = 12_000_000
server_errors = 9_600            # 5xx or timeouts
slo = 0.999

availability = 1 - server_errors / total_requests
budget = (1 - slo) * total_requests
print(f"availability={availability:.4%}")                       # availability=99.9200%
print(f"budget used={server_errors / budget:.0%} of {budget:,.0f} allowed errors")
# budget used=80% of 12,000 allowed errors

Measuring Composite Systems

  • Components in series multiply: an API that depends on a 99.9 percent database and a 99.9 percent auth service can be at most about 99.8 percent available (0.999 x 0.999).

  • Redundant components in parallel improve it: two independent 99 percent replicas give 1 - (1 - 0.99)^2 = 99.99 percent, if failover works and failures are independent.

Common Mistakes

  • Measuring only “process is running” instead of successful user requests.

  • Ignoring partial outages (one region, one endpoint, one customer segment).

  • Averaging across endpoints so a broken checkout is hidden by healthy static pages.

  • Setting SLOs at 100 percent, which leaves no room for change and is unachievable.

Interview Tip

Give the request-based definition, explain SLI, SLO, error budget and SLA with numbers, and say where you would measure (load balancer logs plus synthetic probes). Showing the series and parallel availability maths demonstrates depth.

3

What does five nines mean, and how does it relate to availability?

“Five nines” means 99.999 percent availability: the system may be unavailable for only about 5.26 minutes per year, or roughly 26 seconds per month. Each additional nine cuts allowed downtime by a factor of ten and usually raises cost and complexity sharply, because it requires redundancy across failure domains, automated failover in seconds, zero-downtime deployments, and highly reliable dependencies. Most services do not need five nines; the right target comes from user needs and business impact, and it must account for every dependency in series.

Read full answer

Applies tosetting SLOs and SLAs for telecom, payments, cloud infrastructure, healthcare and other critical services

Downtime Allowed by Each Level

Availability Per year Per 30-day month Per week
99% (“two nines”) 3.65 days 7.2 hours 1.68 hours
99.9% (“three nines”) 8.76 hours 43.2 minutes 10.1 minutes
99.95% 4.38 hours 21.6 minutes 5.04 minutes
99.99% (“four nines”) 52.6 minutes 4.32 minutes 1.01 minutes
99.999% (“five nines”) 5.26 minutes 25.9 seconds 6.05 seconds
for nines in ["0.99", "0.999", "0.9999", "0.99999"]:
    a = float(nines)
    year_min = (1 - a) * 365 * 24 * 60
    month_s = (1 - a) * 30 * 24 * 3600
    print(f"{a:.3%}: {year_min:8.1f} min/year, {month_s:8.0f} s/month")
# 99.000%:   5256.0 min/year,    25920 s/month
# 99.900%:    525.6 min/year,     2592 s/month
# 99.990%:     52.6 min/year,      259 s/month
# 99.999%:      5.3 min/year,       26 s/month

Why Five Nines Is Hard

  • Detection alone takes time: if alerting needs a minute to fire, a single incident consumes the monthly budget.

  • Humans are too slow: recovery must be automatic (failover, rollback) within seconds.

  • Dependencies multiply: a service calling five components each at 99.999 percent can be at most about 99.995 percent (0.99999^5) unless calls are redundant.

  • Deployments and maintenance must cause no downtime: rolling, blue-green or canary releases, online schema migrations.

  • Correlated failures (bad configuration pushed everywhere, regional outages, DNS issues) must be contained.

What It Typically Requires

Area Practice
Redundancy Active-active across zones and usually regions
Failover Automated, tested, seconds not minutes
Releases Progressive rollout with automatic rollback
Dependencies Each critical dependency at least as available, or redundant and degradable
Data Synchronous replication or carefully designed conflict handling
Operations Chaos testing, game days, strong on-call and runbooks

Choosing the Right Target

  • User expectations: users often cannot tell the difference between 99.99 and 99.999 percent because their own networks and devices are less reliable.

  • Cost vs benefit: each extra nine can multiply infrastructure and engineering costs.

  • Error budgets: a target below 100 percent leaves room for releases and experiments. Google’s SRE practice treats the error budget as a resource to spend on change.

  • Different targets per feature: login and payments may need higher availability than reports or recommendations.

Interview Tip

Quote the downtime numbers (about 5 minutes a year, 26 seconds a month), explain why each nine is ten times harder, and show how dependencies in series reduce availability. Arguing that the target should follow user needs rather than prestige shows good engineering judgement.

4

How does redundancy improve the reliability of a system?

Redundancy means having more than one component able to do the same job, so that when one fails, another takes over and users keep being served. It improves reliability and availability by removing single points of failure: with independent redundant components, the system fails only if all of them fail at once. Redundancy comes in several forms (active-active, active-passive, N+1 capacity, data replication, geographic spread), and it only helps if failures are independent and failover actually works, which must be tested.

Read full answer

Applies toservers and instances, databases, load balancers, network paths, power and cooling, availability zones and regions, and data storage

The Maths

For independent components with availability a:

  • In series (all required): A = a1 x a2 x … (each extra dependency lowers availability).

  • In parallel (any one is enough): A = 1 - (1 - a)^n.

def parallel(a, n):
    return 1 - (1 - a) ** n

for n in (1, 2, 3):
    print(n, f"{parallel(0.99, n):.4%}")
# 1 99.0000%
# 2 99.9900%
# 3 99.9999%

Two independent 99 percent replicas give 99.99 percent in theory. In practice, correlated failures (shared power, the same bad deploy, a common dependency) and failover delays reduce the benefit, which is why redundancy is spread across failure domains.

Types of Redundancy

Type How it works Example
Active-active All copies serve traffic; failures just shrink capacity Several app instances behind a load balancer; multi-region active-active
Active-passive (standby) A standby takes over when the primary fails Database primary with a hot standby and automatic failover
N+1 or N+2 capacity Extra capacity so losing one or two units still meets demand Running 4 instances when 3 handle peak load
Data redundancy Multiple copies of data Database replicas, RAID, erasure coding, cross-region backups
Geographic redundancy Copies in different zones or regions Three availability zones; secondary region for disaster recovery
Network and power redundancy Dual links, redundant switches, multiple power feeds Two network providers, dual power supplies
Diversity Different implementations or vendors Multi-CDN or multi-DNS providers to avoid a single vendor outage

What Makes Redundancy Effective

  1. Independence: place copies in separate failure domains (hosts, racks, zones, regions).

  2. Detection: health checks and failure detectors notice failures quickly.

  3. Automatic failover: traffic moves without human action; DNS TTLs, connection draining and leader election tuned for speed.

  4. Enough capacity: surviving copies must handle the full load (N+1), or failover causes overload and a cascade.

  5. Consistent state: replicas must have up-to-date data, or failover loses writes (see RPO).

  6. Regular testing: failover drills and chaos experiments prove it works.

Pitfalls

  • False redundancy: two instances on the same host or in the same zone, or both depending on one database.

  • Split brain: two primaries accepting writes after a faulty failover; prevented with quorums and fencing.

  • Complexity: more components and failover logic mean new failure modes.

  • Correlated software bugs: identical replicas share the same bug; a bad configuration pushed everywhere defeats redundancy, so roll out changes progressively.

  • Cost: idle standby capacity and duplicated infrastructure.

Interview Tip

Explain redundancy as removing single points of failure, show the parallel availability formula, and stress independence of failure domains, automatic failover, spare capacity and testing. Mentioning correlated failures such as a bad deploy shows you understand its limits.

5

What is a single point of failure, and how can it be removed?

A single point of failure (SPOF) is any component whose failure alone makes the whole system, or a critical function, unavailable. It can be hardware (one database server, one load balancer), software (one cron host, one leader without failover), infrastructure (one availability zone, one DNS provider), data (one copy of a database), or even people and processes (one engineer who knows how to restore backups). You remove SPOFs by finding them through dependency mapping and failure testing, then adding redundancy, automatic failover, isolation and graceful degradation.

Read full answer

Applies toarchitecture reviews, reliability planning, cloud deployments and incident prevention

Common SPOFs and Mitigations

SPOF Mitigation
Single application server Several instances behind a load balancer, autoscaling
Single load balancer Managed or redundant load balancers (active-active or with floating IP failover)
Single database primary Replicas with automatic failover (Patroni, RDS Multi-AZ), or distributed databases
One availability zone Spread instances and replicas across three zones
One region Multi-region active-active or warm standby for critical systems
Single DNS or CDN provider Multiple providers or a provider with strong redundancy, sensible TTLs
Shared configuration or secrets service Caching of config, highly available clusters, safe defaults
Cron or batch job on one host Distributed scheduler with leader election, or idempotent jobs on several nodes with locking
Message broker with one node Clustered brokers with replicated queues or partitions
Certificate or credential expiry Automated renewal and expiry monitoring
One person with critical knowledge Runbooks, shared on-call, documentation, access for several people
Third-party API Timeouts, fallbacks, caching, an alternative provider where justified

How to Find SPOFs

  1. Draw the request path: from DNS and CDN through load balancers, services, databases, queues and third parties.

  2. Ask “what if this fails?” for every box and arrow, including control-plane services (identity, config, service discovery).

  3. Check shared dependencies: a “redundant” pair that uses one disk, one network switch, one zone or one database is not redundant.

  4. Run failure tests: terminate instances, block a zone, fail a dependency (chaos engineering, game days).

  5. Review incidents: past outages often reveal hidden SPOFs.

Example

A web app runs three instances across zones, but all of them read session data from one Redis node.

  • Problem: if Redis fails, every user is logged out or every request errors. Redis is a SPOF.

  • Fixes: Redis with replicas and automatic failover (Sentinel, Redis Cluster or a managed service across zones); or signed cookies or tokens so sessions do not need Redis; and a fallback that degrades gracefully instead of failing every request.

Beyond Redundancy

  • Graceful degradation: if recommendations fail, show the page without them.

  • Bulkheads: isolate resources so one failing dependency cannot consume all threads or connections.

  • Timeouts and circuit breakers to stop a slow dependency from becoming a SPOF through resource exhaustion.

  • Progressive rollouts: a bad configuration pushed everywhere at once acts as a SPOF in time; staged rollouts limit the damage.

Trade-offs

Removing every SPOF costs money and adds complexity. Prioritise by impact and likelihood: eliminate SPOFs on critical paths (checkout, login), and accept some on internal or low-impact tools with good recovery plans.

Interview Tip

Define a SPOF, give examples across infrastructure, data, software and people, explain how to find them (dependency mapping, failure testing) and how to remove them (redundancy, failover, degradation, bulkheads). Pointing out hidden shared dependencies shows real-world experience.

6

Why is mean time between failures (MTBF) important in reliability engineering?

Mean Time Between Failures (MTBF) is the average operating time between one failure and the next for a repairable system. It is a measure of reliability: a higher MTBF means failures happen less often. MTBF matters because it helps predict failure frequency, plan redundancy, maintenance and spare capacity, compare designs or vendors, and, together with MTTR, calculate availability = MTBF / (MTBF + MTTR). It is an average, so it says nothing precise about when a particular component will fail, and it is only meaningful for consistent definitions of “failure”.

Read full answer

Applies tohardware (disks, servers, network gear), software services, infrastructure planning and SRE metrics

Calculation

MTBF = total operating time / number of failures

uptime_hours = 720          # a 30-day month
failures = 3
mttr_hours = 0.5            # average time to restore

mtbf = uptime_hours / failures
availability = mtbf / (mtbf + mttr_hours)
print(f"MTBF={mtbf:.0f} h, availability={availability:.3%}")
# MTBF=240 h, availability=99.792%

Why It Matters

Use How MTBF helps
Availability planning Combined with MTTR it predicts expected uptime
Redundancy design Knowing how often components fail shows how many spares or replicas you need
Capacity planning In large fleets, failures are routine: 10,000 disks with an MTBF of 1 million hours means a disk failure roughly every 100 hours
Maintenance scheduling Replace or patch components before failure rates rise
Vendor and design comparison Compare hardware or architectures on failure frequency
Tracking improvement A rising MTBF shows that reliability work (testing, safer deploys) is paying off

Fleet Example

disks = 10_000
disk_mtbf_hours = 1_000_000
hours_between_failures = disk_mtbf_hours / disks
print(f"one disk failure about every {hours_between_failures:.0f} hours")
# one disk failure about every 100 hours

At scale, “rare” failures become constant, so systems must handle them automatically.

Related Metrics

Metric Meaning
MTTF (mean time to failure) For non-repairable components, average time until failure
MTTR (mean time to repair or restore) Average time to restore service
MTTD (mean time to detect) Time from failure to detection
Failure rate Approximately 1 / MTBF for a constant-rate model

Limitations

  • Averages hide patterns: hardware failure rates follow a “bathtub curve” (early failures, a stable period, wear-out), so failures are not evenly spread.

  • Definition matters: is a brief error spike a failure? A partial outage? Inconsistent definitions make MTBF meaningless.

  • Vendor figures are lab estimates: field failure rates can be noticeably higher (Google’s and Backblaze’s large disk studies found real-world rates differ from datasheets).

  • Software is different: failures often come from changes (deploys, configuration) rather than wear; tracking change failure rate may be more useful.

  • Small samples: with few incidents, MTBF fluctuates widely.

Improving MTBF

  • Better testing, code review and static analysis.

  • Safer deployments (canaries, feature flags, automatic rollback).

  • Capacity headroom, rate limiting and load shedding to avoid overload failures.

  • Eliminating known failure modes found in post-mortems.

  • Reliable hardware and proactive replacement.

Interview Tip

Define MTBF, show how it combines with MTTR to give availability, and use a fleet example to show that failures become routine at scale. Pointing out its limits (averages, definitions, the bathtub curve) and that in software, improving MTTR is often cheaper, shows mature thinking.

7

What role does mean time to repair (MTTR) play in system availability?

Mean Time to Repair (or Restore), MTTR is the average time from a failure until service is restored. It is one half of the availability equation, availability = MTBF / (MTBF + MTTR), so cutting recovery time directly raises availability, often more cheaply than preventing every failure. MTTR includes detection, diagnosis, mitigation and recovery, and modern reliability practice focuses heavily on reducing it through fast alerting, automated failover and rollback, good observability, runbooks and practised incident response.

Read full answer

Applies toSRE and operations teams, incident management, SLOs and error budgets, and system design for fast recovery

Components of MTTR

Phase What happens How to shorten it
Detect Notice something is wrong SLO-based alerts, synthetic probes, anomaly detection
Triage and diagnose Find the cause or affected component Dashboards, distributed tracing, structured logs, recent-change view
Mitigate Stop user impact (often before fixing the root cause) Rollback, feature flags, failover, traffic shifting, scaling
Recover Return to full normal operation Automated recovery, replay of queued work, data repair
Verify Confirm the fix Health checks, SLI dashboards

Why It Has a Big Impact

def availability(mtbf_h, mttr_h):
    return mtbf_h / (mtbf_h + mttr_h)

mtbf = 168                       # one failure per week
for mttr_min in (60, 15, 2):
    print(f"MTTR {mttr_min:>2} min -> {availability(mtbf, mttr_min / 60):.3%}")
# MTTR 60 min -> 99.408%
# MTTR 15 min -> 99.851%
# MTTR  2 min -> 99.980%

With the same failure frequency, cutting MTTR from an hour to two minutes moves availability from about 99.4 percent to about 99.98 percent.

Techniques to Reduce MTTR

  • Automated failover: database replicas promoted automatically; load balancers removing unhealthy instances.

  • Fast rollback: keep the previous version deployable in one step; canary releases that roll back automatically on SLO regressions.

  • Feature flags: switch off a faulty feature without deploying.

  • Self-healing: orchestrators (Kubernetes) restart crashed containers and replace failed nodes.

  • Good observability: metrics, logs and traces linked by request IDs, plus a timeline of recent deployments and config changes.

  • Runbooks and playbooks: documented steps for common failures.

  • Clear incident roles: incident commander, communications lead, subject-matter experts.

  • Practice: game days and chaos experiments so teams and tools are ready.

  • Small, frequent changes: easier to identify and revert the change that caused a problem.

MTTR vs MTBF Investments

Approach Example Effect
Increase MTBF More testing, safer infrastructure Fewer incidents
Decrease MTTR Rollback automation, failover, better alerts Shorter incidents

Complex systems always fail eventually, so many teams find that reducing MTTR gives faster gains, while MTBF work removes repeat causes found in post-mortems.

Caveats

  • MTTR is an average; a single long outage can dominate user impact, so also track the distribution and the worst incidents.

  • Definitions vary (repair, restore, recover, respond); agree on one and measure consistently.

  • Mitigation first: restore users quickly, then investigate the root cause.

Interview Tip

Explain MTTR’s phases, show numerically how lowering it raises availability, and list concrete ways to cut it (automated failover, rollback, feature flags, observability, runbooks). Stressing “mitigate first, then find the root cause” shows incident-response maturity.

8

What is the difference between high availability and fault tolerance?

High availability (HA) means the system is designed to minimise downtime: when a component fails, it recovers quickly, often with a brief interruption (seconds to minutes) while failover happens. Fault tolerance (FT) means the system continues operating with no interruption and no lost work when a component fails, because redundant components are running in lockstep or fully active and take over instantly. FT is stricter, more expensive and more complex; most web systems aim for HA, while FT is used where even a brief interruption is unacceptable.

Read full answer

Applies todatabases, servers and virtual machines, network equipment, aircraft and industrial control systems, payment and telecom infrastructure

Comparison

Aspect High availability Fault tolerance
Goal Minimise downtime Zero interruption on component failure
Failure effect Short disruption during failover No visible disruption
In-flight requests May fail and need a retry Continue, or are transparently completed
Typical design Active-passive or active-active with failover Fully redundant, synchronized components, lockstep execution or replicated state machines
Cost Moderate High (duplicate hardware, synchronization overhead)
Complexity Moderate High
Examples Database with automatic standby failover; app servers behind a load balancer Aircraft flight control computers, VMware vSphere Fault Tolerance, RAID 1 mirrored disks, redundant power supplies

Example: Database Failover vs Fault Tolerance

  • HA: a PostgreSQL primary with a synchronous standby and automatic failover (Patroni or a managed Multi-AZ service). When the primary fails, failover usually takes from several seconds to a couple of minutes; open connections drop and clients reconnect and retry.

  • FT-like behaviour: a consensus-based database (CockroachDB, Spanner) or a storage layer that writes to several replicas synchronously, so a single node failure does not lose committed data, and the remaining nodes keep serving. Even there, requests in flight on the failed node may need retrying, so it is strictly “highly available with no data loss” rather than perfect fault tolerance.

Fault Tolerance Techniques

  • Hardware redundancy: dual power supplies, mirrored disks, ECC memory, redundant network interfaces.

  • Lockstep or triple modular redundancy: several processors execute the same instructions and vote on results (used in aerospace and some safety systems).

  • Replicated state machines: every replica applies the same ordered log (Raft, Paxos).

  • Transparent retries and idempotency: the system hides individual failures from callers.

High Availability Techniques

  • Redundant instances with health checks and load balancing.

  • Automatic failover of databases, caches and brokers.

  • Multi-zone deployment and fast DNS or anycast routing changes.

  • Graceful degradation when parts of the system are down.

How They Relate

  • FT is a stronger form of availability engineering; a fault-tolerant component contributes to a highly available system.

  • Many systems combine both: fault-tolerant storage (replicated disks) underneath highly available services (stateless instances that can be replaced).

  • Neither protects against correlated failures such as a bad deployment or configuration pushed to all replicas; that needs progressive rollouts and isolation.

Choosing

Requirement Choice
A few seconds of disruption are acceptable HA
Interruption could endanger lives or cause major financial loss FT for those components
Cost-sensitive web applications HA with retries and idempotent operations

Interview Tip

Say that HA minimises downtime with quick recovery, while FT continues with no interruption, then compare cost, complexity and examples. Noting that most distributed systems achieve “HA plus client retries” rather than true FT shows a realistic view.

9

How would you architect a system for high availability?

To architect for high availability, remove single points of failure at every layer, spread redundant components across independent failure domains (zones, then regions if needed), make failure detected and handled automatically, and keep the system useful while degraded. Concretely: stateless services behind load balancers, replicated data stores with automatic failover, health checks, timeouts, retries with backoff and circuit breakers, safe deployments, capacity headroom and load shedding, and observability with SLOs, all verified by failure testing.

Read full answer

Applies toweb applications, APIs, microservices platforms and data services on cloud or on-premises infrastructure

Reference Architecture (Single Region, Three Zones)

Users -> DNS (health-checked) -> CDN / edge
      -> Load balancer (spans zones A, B, C)
      -> Stateless app instances in A, B, C (autoscaled, N+1 capacity)
      -> Cache cluster with replicas across zones
      -> Database: primary in A, synchronous standby in B, replica in C, automatic failover
      -> Message queue cluster replicated across zones
      -> Object storage (replicated by the provider)

Layer-by-Layer Decisions

Layer HA technique
DNS and edge Managed DNS with health checks, CDN serving static content and cached pages
Load balancing Managed or redundant load balancers across zones; health checks remove bad instances
Application Stateless instances in each zone; sessions in signed tokens or a replicated store; autoscaling
Data Replication with automatic failover (RDS Multi-AZ, Patroni, distributed SQL); regular backups
Asynchronous work Durable queues; idempotent consumers; retries with dead-letter queues
Dependencies Timeouts, retries with exponential backoff and jitter, circuit breakers, fallbacks
Capacity N+1 per zone (survive one zone loss at peak), rate limiting, load shedding
Releases Rolling, blue-green or canary deployments with automatic rollback; feature flags
Operations SLOs, alerts on error budget burn, runbooks, on-call

Availability Maths to Guide the Design

  • Dependencies in series multiply: an app (99.95 percent) that needs a database (99.95 percent) and a cache (99.9 percent) is at most about 99.8 percent available, unless the cache is optional.

  • Parallel redundancy helps: three independent zones each at 99.5 percent give 1 - 0.005^3, well above 99.99 percent, if failover is automatic and capacity suffices.

  • Make non-critical dependencies optional (degrade instead of failing) so they leave the series chain.

Multi-Region (When Needed)

  • Active-passive (warm standby): second region with replicated data; fail over on regional outage. Lower cost; RTO of minutes.

  • Active-active: users served from several regions; needs data designed for multi-region writes or a home-region per record. Highest availability and complexity. The consistency choices during inter-region partitions follow the CAP trade-off.

Resilience Patterns

  • Timeouts on every network call.

  • Retries only for idempotent operations, with exponential backoff and jitter and a retry budget, to avoid retry storms.

  • Circuit breakers to stop calling a failing dependency.

  • Bulkheads to isolate resource pools per dependency.

  • Graceful degradation: hide recommendations or show cached data rather than failing the page.

Verify It Works

  • Chaos experiments: terminate instances, fail a zone, inject latency.

  • Regular failover drills for databases and regions.

  • Restore tests for backups.

Interview Tip

Walk layer by layer from DNS to data, emphasise multi-zone redundancy, automatic failover, resilience patterns and safe deployments, and use the series and parallel availability maths to justify choices. Ending with failure testing shows you know designs must be proven.

10

Which design patterns are commonly used to improve system availability?

Common availability patterns fall into four groups: redundancy and failover (active-active, active-passive, leader election, replication), failure isolation (bulkheads, cell-based architecture, circuit breakers, timeouts), controlled recovery (retries with exponential backoff and jitter, idempotency, queue-based load levelling, health checks with self-healing), and graceful degradation under load (rate limiting, load shedding, fallbacks, caching, feature flags). Combined with safe deployment patterns (canary, blue-green) they reduce both how often users are affected and for how long.

Read full answer

Applies todistributed systems, microservices, cloud platforms and client libraries

Pattern Catalogue

Pattern What it does Example
Active-active redundancy Several instances or regions serve traffic App servers in three zones behind a load balancer
Active-passive failover Standby takes over on primary failure Database primary with hot standby
Leader election Exactly one node coordinates, with automatic replacement etcd or ZooKeeper electing a scheduler leader
Health check and self-healing Detect and replace unhealthy instances Kubernetes liveness and readiness probes
Timeout Bound waiting on a dependency 300 ms timeout on a pricing call
Retry with exponential backoff and jitter Recover from transient errors without stampedes 100 ms, 200 ms, 400 ms plus random jitter
Circuit breaker Stop calling a failing dependency, fail fast Open after 50 percent errors, probe after 30 seconds
Bulkhead Separate resource pools so one failure cannot exhaust all Separate connection pools per downstream service
Fallback Alternative response when a dependency fails Default recommendations, cached prices
Queue-based load levelling Buffer bursts; process at a steady rate Orders written to a queue, workers process asynchronously
Rate limiting and load shedding Protect the system under overload Return 429 or 503 for low-priority traffic first
Caching Serve reads without the origin CDN with stale-if-error
Cell-based architecture Split users into independent cells to limit blast radius Each cell has its own stack serving a subset of customers
Canary and blue-green deployment Reduce risk of bad releases 1 percent of traffic to the new version, automatic rollback
Feature flags (kill switches) Disable faulty features instantly Turn off a new search ranker

Retry With Backoff and Jitter

import random

def backoff_delays(base=0.1, cap=2.0, attempts=5, seed=7):
    rng = random.Random(seed)
    return [round(rng.uniform(0, min(cap, base * 2 ** i)), 3) for i in range(attempts)]

print(backoff_delays())   # "full jitter": random delay up to an exponentially growing cap
# [0.032, 0.03, 0.26, 0.058, 0.857]

Retries without backoff or idempotency can turn a small outage into a retry storm: every client hammers the recovering service at once. Limit attempts, use jitter, retry only idempotent operations (or use idempotency keys), and prefer a retry budget (for example at most 10 percent extra load).

How Patterns Work Together

A product page calls pricing, inventory and recommendations:

  1. Each call has a timeout.

  2. Transient failures get one or two jittered retries.

  3. A circuit breaker around recommendations opens when it is failing.

  4. The fallback shows popular items instead.

  5. Bulkheads ensure slow recommendations cannot use up threads needed for pricing.

  6. Under overload, load shedding drops low-priority requests (bots, prefetches) first.

Trade-offs

  • More moving parts and configuration to tune.

  • Fallbacks can show stale or generic data.

  • Active-active and cells cost more infrastructure.

  • Patterns must be tested; an untested failover path often fails when needed.

Interview Tip

Group patterns into redundancy, isolation, recovery and degradation, give a concrete example of several working together, and highlight the retry storm risk with the backoff-and-jitter fix. That shows you understand how patterns interact, not just their names.

11

How does load balancing improve availability, and what are its pitfalls?

Load balancing improves availability by spreading traffic across several healthy instances, removing failed instances automatically through health checks, and allowing maintenance and deployments without downtime (drain one instance at a time). With instances in several zones, losing one server or even one zone only reduces capacity. The pitfalls are that the load balancer itself can become a single point of failure, poorly designed health checks can remove all instances or keep broken ones, uneven distribution and sticky sessions can overload some nodes, and failover can overload the survivors if there is not enough spare capacity.

Read full answer

Applies toL4 and L7 load balancers (AWS ALB and NLB, Google Cloud Load Balancing, NGINX, HAProxy, Envoy), Kubernetes Services and Ingress, DNS and global load balancing

How It Helps Availability

Mechanism Effect
Distribution across instances No single server handles all traffic
Health checks Unhealthy instances stop receiving requests
Multi-zone targets A zone failure only reduces capacity
Connection draining Instances finish in-flight requests before removal during deploys
Autoscaling integration New instances join automatically under load
Global load balancing Route users to another region on regional failure

Pitfalls and Mitigations

Pitfall What goes wrong Mitigation
Load balancer as SPOF A single NGINX or HAProxy box fails Managed load balancers, redundant pairs with floating IPs, DNS across several LB nodes
Shallow health checks /health returns 200 while the app cannot reach its database Readiness checks that verify critical local dependencies
Over-deep health checks Health check depends on a shared database; when it slows, every instance is marked unhealthy and removed Keep checks local; do not fail health on shared dependency outages; use circuit breakers instead
Insufficient capacity After a zone fails, remaining instances overload and fail too (cascade) N+1 capacity per zone, load shedding, autoscaling headroom
Sticky sessions Users pinned to one instance lose sessions when it fails; uneven load Stateless services with external or token-based sessions
Uneven distribution Round robin ignores slow or busy instances; long-lived connections (gRPC, WebSockets) stick to old instances Least-outstanding-requests or least-connections algorithms, connection recycling, client-side balancing
Slow start New instances with cold caches or JIT get full traffic immediately Slow-start ramp-up
Retry amplification LB retries plus client retries multiply load during an incident Single retry layer, retry budgets, idempotency
DNS caching Clients keep sending traffic to a failed endpoint Short TTLs, anycast, health-checked DNS
Misconfigured timeouts LB timeout shorter than the app’s, causing 502s or duplicate work Align timeouts across layers

Example: Health Check Design

# Kubernetes probes
livenessProbe:          # restart if the process is stuck
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:         # stop routing traffic while not ready
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2

/livez checks only that the process responds. /readyz checks that the instance has finished starting and can serve (for example its local connection pool is initialised), but it should not fail just because a shared dependency is slow, or every pod would be removed at once.

Algorithms

Algorithm Best for
Round robin Similar instances, short requests
Least connections or least outstanding requests Varying request durations
Weighted Mixed instance sizes, canary releases
Consistent hashing Cache affinity or sharded state
Power of two random choices Large fleets; avoids herding onto one “least loaded” node

Interview Tip

Explain how health checks, distribution and draining improve availability, then cover the pitfalls: LB as SPOF, health check design, capacity after failover, sticky sessions and retry amplification. The “deep health check removes every instance” example shows real operational experience.

12

What role do health checks play in keeping a system available?

Health checks are periodic probes that tell load balancers, orchestrators and monitoring systems whether an instance or service can do its job. They keep a system available by removing broken instances from traffic, restarting stuck processes, holding back new instances until they are ready, and triggering failover of databases or regions. Their value depends on design: a check that is too shallow keeps broken instances in service, while one that is too deep (depending on shared databases) can remove every instance at once during a dependency problem.

Read full answer

Applies toload balancers, Kubernetes probes, service discovery (Consul), DNS failover, database failover managers, and synthetic monitoring

Types of Health Checks

Type Question Action on failure Example
Liveness Is the process alive and not deadlocked? Restart the container or process Kubernetes livenessProbe on /livez
Readiness Can this instance serve traffic now? Stop routing traffic to it (do not restart) Kubernetes readinessProbe, LB target health
Startup Has slow initialisation finished? Delay liveness checks until it has Kubernetes startupProbe for apps with long warm-up
Dependency or deep check Can it reach its critical dependencies? Report degraded status; alert /health/deep used by monitoring, not by the LB
Synthetic (external) Can users complete key journeys? Alert, or trigger DNS or regional failover Probes from several regions logging in and searching

Example Endpoint Design

from http.server import BaseHTTPRequestHandler

STATE = {"started": True, "draining": False, "db_pool_ready": True}

def health_status(path):
    if path == "/livez":
        return 200                                    # process responds: alive
    if path == "/readyz":
        ok = STATE["started"] and not STATE["draining"] and STATE["db_pool_ready"]
        return 200 if ok else 503                     # route traffic only when ready
    return 404

class Health(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(health_status(self.path))
        self.end_headers()

print(health_status("/livez"), health_status("/readyz"))   # 200 200
STATE["draining"] = True                                    # e.g. during shutdown
print(health_status("/livez"), health_status("/readyz"))   # 200 503

During graceful shutdown, readiness fails first so the load balancer stops sending new requests, then the process finishes in-flight work and exits.

Design Guidelines

  • Liveness should be cheap and local: do not include database checks, or a database outage restarts every pod and makes recovery slower.

  • Readiness reflects ability to serve: initialisation done, not draining, local resources ready.

  • Avoid cascading removal: if a shared dependency fails, all instances fail readiness together; prefer circuit breakers and degraded responses over marking everything unhealthy. Some load balancers “fail open” when all targets are unhealthy for this reason.

  • Use thresholds: require several consecutive failures before removal and several successes before return, to avoid flapping.

  • Set sensible timeouts and intervals: fast enough to detect failures within seconds, not so frequent that checks add load.

  • Secure and lightweight: health endpoints should not expose sensitive details publicly or do expensive work.

  • Measure from outside too: internal checks can all pass while DNS, TLS certificates or the CDN are broken; synthetic probes catch that.

Health Checks in Failover

  • Database failover managers (Patroni, Orchestrator, cloud Multi-AZ) use health checks plus a quorum to decide when to promote a replica, with care to avoid split brain.

  • DNS failover (Route 53, Cloudflare) switches traffic to a standby region when health checks fail.

  • Service discovery removes unhealthy instances from the registry.

Interview Tip

Distinguish liveness, readiness and startup checks, explain what each triggers, and stress the danger of deep checks causing cascading removal or restart loops. Mentioning graceful draining and external synthetic probes rounds out a strong answer.

13

What is the purpose of the circuit breaker pattern in distributed systems?

A circuit breaker wraps calls to a remote dependency and stops sending requests when that dependency is failing, returning an error or fallback immediately instead. It protects availability in three ways: callers fail fast instead of piling up threads and connections waiting on timeouts, the failing service gets time to recover instead of being flooded, and failures do not cascade through the call chain. After a cool-down period, the breaker lets a few trial requests through and closes again if they succeed.

Read full answer

Applies tomicroservices, API clients, database and cache clients, service meshes (Envoy outlier detection), and libraries such as Resilience4j, Polly (.NET), opossum (Node.js) and pybreaker (Python)

The Three States

State Behaviour Transition
Closed Requests pass through; failures are counted Opens when the failure rate or count crosses a threshold
Open Requests fail immediately (or use a fallback) After a wait period, moves to half-open
Half-open A limited number of trial requests are allowed Closes if they succeed; reopens if they fail

Example Implementation

import time

class CircuitBreaker:
    def __init__(self, failure_threshold=3, recovery_seconds=2.0, clock=time.monotonic):
        self.threshold, self.recovery, self.clock = failure_threshold, recovery_seconds, clock
        self.state, self.failures, self.opened_at = "closed", 0, 0.0

    def call(self, fn, fallback):
        if self.state == "open":
            if self.clock() - self.opened_at < self.recovery:
                return fallback()
            self.state = "half_open"                 # allow a trial request
        try:
            result = fn()
        except Exception:
            self._record_failure()
            return fallback()
        self.state, self.failures = "closed", 0
        return result

    def _record_failure(self):
        self.failures += 1
        if self.state == "half_open" or self.failures >= self.threshold:
            self.state, self.opened_at = "open", self.clock()

now = [0.0]
cb = CircuitBreaker(clock=lambda: now[0])
healthy = [False]

def pricing():
    if not healthy[0]:
        raise TimeoutError("pricing timed out")
    return "price: 499"

for _ in range(4):
    print(cb.state, cb.call(pricing, lambda: "price unavailable"))
now[0] = 3.0; healthy[0] = True
print(cb.call(pricing, lambda: "price unavailable"), cb.state)
# closed price unavailable
# closed price unavailable
# closed price unavailable
# open price unavailable
# price: 499 closed

The fourth call never reaches the pricing service: the breaker is open. After the recovery period, a trial call succeeds and the breaker closes.

Why It Matters: Cascading Failures

Without a breaker, a slow dependency with a 2-second timeout causes every request thread to wait 2 seconds. Thread pools fill, the caller becomes slow too, and its own callers start timing out. The failure spreads upward. The breaker cuts this chain by failing fast.

Configuration Choices

Setting Consideration
Failure threshold Rate-based (for example 50 percent of the last 20 calls) is more robust than a fixed count
What counts as failure Timeouts and 5xx errors, not 4xx client errors
Slow-call threshold Treat calls slower than a limit as failures (Resilience4j supports this)
Open duration Long enough for recovery, short enough to resume quickly
Half-open trial count A few requests, to avoid flooding a recovering service
Scope One breaker per dependency (and sometimes per endpoint or host)

Use With Other Patterns

  • Timeouts: the breaker needs failures to detect; timeouts produce them quickly.

  • Retries: retry only when the breaker is closed, with backoff and jitter; never retry against an open breaker.

  • Fallbacks: cached data, defaults, or a clear degraded message.

  • Bulkheads: limit concurrency per dependency.

  • Monitoring: alert on breaker state changes; open breakers reveal unhealthy dependencies.

Interview Tip

Explain the closed, open and half-open states, the cascading-failure problem it solves, and how it combines with timeouts, retries and fallbacks. Mentioning that failure thresholds should be rate-based and scoped per dependency shows practical experience.

14

Which key indicators would you monitor to ensure system reliability?

To ensure reliability, monitor user-facing service level indicators first: the four golden signals of latency, traffic, errors and saturation, expressed as SLIs with SLOs and error budget burn rates. Then add dependency health (databases, caches, queues, third-party APIs), infrastructure resources, data correctness and durability signals (replication lag, backup success, data freshness), change-related signals (deploys, configuration changes, change failure rate), and operational metrics (incident count, MTTD, MTTR).

Read full answer

Applies toweb services, APIs, data pipelines, databases and platform infrastructure

User-Facing Indicators (Most Important)

Indicator What to measure Example SLO
Availability / success rate Share of valid requests not failing (5xx, timeouts) 99.9 percent over 30 days
Latency p50, p95 and p99 per endpoint (not averages) 99 percent of checkout requests under 800 ms
Traffic Requests per second, active users, messages per second Used for capacity and anomaly detection
Errors Error rate by type and endpoint Alert on burn rate, not single errors
Saturation How “full” the system is: CPU, memory, connections, thread pools, queue depth Alert before limits, for example 80 percent of the connection pool

Error Budget Burn Rate

Alerting on how fast the error budget is being consumed catches serious problems quickly without noisy alerts:

slo = 0.999
budget = 1 - slo                 # 0.1 percent of requests may fail
error_rate_last_hour = 0.0144    # 1.44 percent failing right now

burn_rate = error_rate_last_hour / budget
hours_to_exhaust = 30 * 24 / burn_rate
print(f"burn rate {burn_rate:.1f}x, 30-day budget gone in {hours_to_exhaust:.0f} h")
# burn rate 14.4x, 30-day budget gone in 50 h

A burn rate of 14.4 sustained for one hour consumes 2 percent of a 30-day budget; Google’s SRE guidance uses this as a typical fast-burn paging threshold, paired with a slower-burn ticket alert.

Dependency and Data Indicators

Area Indicators
Databases Query latency, error rate, connections used, replication lag, deadlocks, disk space
Caches Hit ratio, evictions, latency, memory usage
Queues and streams Queue depth, consumer lag, age of oldest message, dead-letter count
Third-party APIs Latency, error rate, rate-limit responses, circuit breaker state
Data correctness Reconciliation mismatches, failed validations, data freshness for pipelines
Durability Backup success and age, restore test results, replication health

Infrastructure Indicators

  • CPU, memory, disk I/O and space, network errors.

  • Container restarts and out-of-memory kills.

  • Node health, autoscaling events, instance failures.

  • Certificate expiry dates and DNS resolution health.

Change and Operational Indicators

Indicator Why
Deploy and configuration change events Most incidents follow a change; overlay them on dashboards
Change failure rate Share of deployments causing incidents or rollbacks
Incident frequency and severity Reliability trend over time
MTTD and MTTR Detection and recovery speed
Alert volume and noise Too many alerts lead to missed real ones

Good Practices

  • Measure at the edge (load balancer, API gateway) to see what users experience, plus synthetic probes for critical journeys.

  • Use percentiles and histograms, not averages.

  • Break down by endpoint, region, customer tier and version to find partial outages.

  • Correlate metrics, logs and traces via request IDs.

  • Alert on symptoms (SLO burn) and use cause-level metrics for diagnosis.

Interview Tip

Lead with the golden signals expressed as SLIs and SLOs, explain burn-rate alerting with numbers, then add dependency, data, infrastructure and change indicators. Saying you would page on user-facing symptoms rather than every CPU spike shows mature monitoring practice.

15

How do you build monitoring that accurately reflects system availability?

Monitoring that accurately reflects availability measures what users experience: it defines SLIs as good events / valid events for key user journeys, collects them from the edge (load balancer or API gateway logs) and from outside (synthetic probes and real-user monitoring), excludes invalid traffic, breaks results down by region, endpoint and customer segment to catch partial outages, and compares them with SLOs and error budgets using burn-rate alerts. Internal host metrics remain useful for diagnosis, but they do not define availability.

Read full answer

Applies toSRE and platform teams, SLO programmes, status pages and SLA reporting

Step 1: Define User Journeys and SLIs

Journey SLI (good / valid)
Log in Login requests returning success within 1 s / all valid login attempts
Search Search requests returning non-5xx within 500 ms / all search requests
Checkout Orders completed without server error / checkout attempts
API Requests without 5xx or timeout / requests excluding 4xx client errors

Decide what “good” means precisely: status codes, latency thresholds, correctness checks.

Step 2: Measure at the Right Places

Source Strength Gap
Load balancer or gateway logs Real traffic, closest to users inside your infrastructure Misses DNS, CDN and client network failures
Synthetic probes from several regions Works with low traffic; detects total outages and DNS or TLS issues Tests only scripted paths
Real user monitoring (browser or mobile) True end-to-end view Noisy, affected by users’ own networks
Application metrics Detailed per-endpoint and per-error data Cannot see requests that never reached the app

Using at least two (edge logs plus synthetic probes) avoids blind spots.

Step 3: Compute SLIs and SLOs

# Availability SLI over 30 days from load balancer metrics (Prometheus)
sum(rate(http_requests_total{job="edge", code!~"5.."}[30d]))
/
sum(rate(http_requests_total{job="edge"}[30d]))

Exclude health checks and known synthetic traffic; decide how to treat 429s caused by your own overload (usually as failures).

Step 4: Alert on Burn Rate

Use multi-window, multi-burn-rate alerts:

Alert Condition (99.9 percent SLO) Action
Fast burn Error rate above 14.4x budget over 1 hour and 5 minutes Page on-call
Slow burn Error rate above 1x budget over 3 days (often with a 6-hour short window) Ticket

The short window confirms the problem is still happening, which reduces stale alerts.

Step 5: Make Partial Outages Visible

  • Break SLIs down by region, zone, endpoint, client version, tenant.

  • Track per-customer availability for large tenants; an overall 99.95 percent can hide one customer at 90 percent.

  • Use multiple synthetic locations to see regional or ISP issues.

Step 6: Avoid Common Distortions

Distortion Fix
Measuring “servers up” Measure successful requests instead
Averages across endpoints Separate SLIs for critical journeys
Low-traffic periods Combine with synthetic probes; use time-based SLIs for rarely used features
Retries hiding failures Count first-attempt failures too, or measure from the client
Monitoring in the same failure domain Run probes and alerting outside the production region or provider
Clock and aggregation errors Consistent time windows; avoid averaging percentiles

Step 7: Report and Act

  • SLO dashboards with remaining error budget.

  • Weekly or monthly reliability reviews; error budget policies (slow releases when budget is spent).

  • A status page driven by the same SLIs for consistency.

Interview Tip

Describe the pipeline: define user-journey SLIs, measure at the edge and with synthetic probes, compute SLOs, alert on burn rates, and break down to catch partial outages. Pointing out that monitoring must run outside the failing system, and that averages hide outages, shows real experience.

16

Why do alerting and on-call rotations matter for system reliability?

Alerting turns monitoring into action: it tells the right person quickly that users are affected or soon will be, which shortens time to detect and therefore time to restore. On-call rotations make sure someone qualified is always available to respond, while sharing the load fairly so engineers stay effective. Together they directly reduce MTTR and protect SLOs, but only if alerts are actionable, symptom-based and low-noise, and on-call is sustainable, with runbooks, escalation paths and follow-up work to remove recurring causes.

Read full answer

Applies toSRE and DevOps teams, incident management, and tools such as PagerDuty, Opsgenie, Grafana OnCall, Alertmanager and incident.io

Why Alerting Matters

Benefit Explanation
Faster detection Incidents are found by monitoring, not by angry customers
Lower MTTR Detection is the first phase of recovery
Protects error budgets Burn-rate alerts trigger before SLOs are breached
Prevents escalation Saturation alerts (disk filling, certificate expiring) allow fixes before outages

What Good Alerts Look Like

  • Symptom-based: page on user impact (SLO burn rate, failed checkouts), not on every CPU spike.

  • Actionable: the responder can do something; if not, it should be a ticket or dashboard.

  • Urgent only when urgent: severity levels: page for user impact now, ticket for slow burn or capacity trends.

  • With context: links to dashboards, runbooks, recent deploys and affected regions.

  • Deduplicated and grouped: one incident, one page, not 200.

  • Reviewed regularly: remove or fix alerts that fire without needing action.

# Prometheus alert on fast SLO burn (99.9 percent SLO)
- alert: CheckoutErrorBudgetFastBurn
  expr: |
    (sum(rate(http_requests_total{route="/checkout", code=~"5.."}[1h]))
      / sum(rate(http_requests_total{route="/checkout"}[1h]))) > (14.4 * 0.001)
    and
    (sum(rate(http_requests_total{route="/checkout", code=~"5.."}[5m]))
      / sum(rate(http_requests_total{route="/checkout"}[5m]))) > (14.4 * 0.001)
  labels:
    severity: page
  annotations:
    summary: "Checkout is burning its error budget about 14x faster than allowed"
    runbook: "https://runbooks.internal/checkout-errors"

Why On-Call Rotations Matter

Aspect Value
Coverage Someone responds 24/7, including nights and weekends
Ownership Teams that build a service also operate it (“you build it, you run it”), which motivates reliability improvements
Knowledge sharing Rotations spread operational knowledge and avoid single human points of failure
Escalation Primary and secondary on-call, plus clear escalation to experts and managers

Making On-Call Sustainable

  • Reasonable load: Google’s SRE book suggests no more than about two incidents per 12-hour shift so each one can be handled and followed up properly.

  • Rotation size: enough people that each person is on call only every few weeks; follow-the-sun across time zones where possible.

  • Compensation and recovery time after nights with pages.

  • Handoffs: summaries of ongoing issues at each shift change.

  • Training: shadowing, game days and incident simulations before taking primary on-call.

  • Runbooks for known failure modes, kept up to date.

Closing the Loop

  • Post-incident reviews produce action items that remove causes and noisy alerts.

  • Track metrics: pages per week, false-positive rate, time to acknowledge, MTTR.

  • Automate recurring manual fixes so they no longer need a human.

Anti-Patterns

  • Alert fatigue from noisy, non-actionable alerts, which leads to real alerts being missed.

  • Paging on causes (high CPU) rather than symptoms (users failing).

  • One “hero” who knows everything and is always called.

  • No follow-up: the same incident pages every week.

Interview Tip

Explain that alerting shortens detection and on-call ensures response, then describe good alert design (symptom-based, actionable, burn-rate) and sustainable rotations with runbooks and escalation. Mentioning alert fatigue and the post-incident loop shows operational maturity.

17

What steps would you take to respond to an incident that reduces availability?

A good incident response follows a clear sequence: detect and declare the incident, assign roles (an incident commander coordinates), assess impact, mitigate first to restore users quickly (roll back, fail over, shed load, disable a feature), communicate regularly with stakeholders and customers, verify recovery, and then run a blameless post-incident review that produces concrete follow-up actions. The priority is stopping user impact, not finding the root cause during the outage.

Read full answer

Applies toSRE, DevOps and on-call teams; production outages, partial degradations and severe latency incidents

Step-by-Step Response

1. Detect and declare

  • An SLO burn-rate alert, synthetic probe or customer report triggers the response.

  • Declare an incident early with a severity level; it is cheaper to downgrade than to start late.

  • Open a dedicated channel and an incident document with a timeline.

2. Assign roles

Role Responsibility
Incident commander (IC) Coordinates, decides priorities, keeps the team focused; usually does not debug
Operations / subject-matter experts Investigate and apply mitigations
Communications lead Status page, customer support, executives
Scribe Records timeline, decisions and actions

3. Assess impact

  • What is broken: which journeys, regions, customers, and what percentage of requests?

  • When did it start? What changed around then (deploys, config, traffic, dependency incidents)?

4. Mitigate first

Mitigation When to use
Roll back the latest deployment or config change Impact started after a change
Disable a feature flag A new feature is failing
Fail over to another zone or region Infrastructure or regional failure
Scale out or shed load Overload or traffic spike
Block abusive traffic, tighten rate limits Bots, DDoS, a runaway client
Open circuit breakers, serve cached or degraded responses Failing dependency
Restart or replace unhealthy instances Stuck processes, memory leaks

Prefer reversible, low-risk actions; announce each action in the channel before taking it.

5. Communicate

  • Post status updates at a regular interval (for example every 30 minutes for a major incident), even if there is no new information.

  • Be factual: impact, what is being done, next update time.

  • Keep internal teams (support, sales, leadership) informed.

6. Verify recovery

  • Confirm SLIs have returned to normal, not just that the fix was applied.

  • Watch for recurrence; check queues, backlogs and data consistency.

  • Replay or reconcile work that failed during the incident if needed.

7. Close and review

  • Declare the incident resolved and record the end time.

  • Hold a blameless post-incident review within days: timeline, contributing factors, what went well, what was hard.

  • Create owned, prioritised action items: fix the cause, improve detection, add automation, update runbooks.

Example Timeline

10:02  Burn-rate alert: checkout 5xx at 6 percent
10:04  Incident declared SEV-2; IC assigned; channel opened
10:08  Deploy of payment-service v412 at 09:58 identified as likely trigger
10:10  Rollback to v411 started (announced in channel)
10:16  Error rate back to 0.05 percent; monitoring for recurrence
10:30  Resolved; status page updated; review scheduled

Common Mistakes

  • Debugging the root cause for an hour instead of rolling back in five minutes.

  • Everyone trying fixes at once without coordination.

  • No communication, so support and customers are left guessing.

  • Skipping the review, so the same incident happens again.

Interview Tip

Walk through detect, declare, roles, impact, mitigation, communication, verification and review, and stress “mitigate first, root-cause later”. Giving a short example timeline with a rollback shows practical incident experience.

18

How can post-mortem analysis improve future reliability and availability?

A post-mortem (post-incident review) improves reliability by turning an incident into lasting changes: it reconstructs what happened and why, identifies contributing factors (technical, process and organisational), and produces owned, prioritised action items that prevent recurrence, detect problems sooner, and recover faster. It works only when it is blameless, so people share honest details, and when actions are tracked to completion. Over time, post-mortems reveal patterns across incidents that guide investment in the biggest reliability risks.

Read full answer

Applies toSRE and engineering teams after outages, near misses and significant degradations

What a Good Post-Mortem Contains

Section Content
Summary What happened, impact, duration in plain language
Impact Users, requests, revenue, SLO and error budget consumed
Timeline Detection, key decisions, mitigation and resolution, with times
Contributing factors Technical triggers plus conditions that allowed impact (missing tests, weak alerts, unclear runbooks)
What went well Practices worth keeping
What was difficult Gaps in tooling, knowledge or communication
Action items Specific, owned, prioritised, with due dates
Lessons learned Insights for other teams

Blameless Culture

  • Focus on systems and conditions, not individual mistakes: “the deploy pipeline allowed an unreviewed config change to reach all regions at once” rather than “Sam pushed bad config”.

  • People who fear blame hide information, so root causes stay hidden.

  • Ask “how did the system make this easy to get wrong?”

Finding Contributing Factors

Incidents rarely have a single cause. Look at:

  • Trigger: the change or event (deploy, traffic spike, dependency failure).

  • Detection: how long until we knew, and why?

  • Mitigation: what slowed recovery (missing rollback, unclear ownership)?

  • Blast radius: why did it affect everyone instead of a subset?

Techniques include “five whys” (used carefully, since it can narrow to one cause), timeline analysis, and fault trees.

Turning Findings Into Improvements

Category Example action
Prevent Add validation for configuration changes; add tests for the bug class
Detect Burn-rate alert for the affected journey; synthetic probe for login
Mitigate faster One-click rollback; feature flag; automated failover
Limit blast radius Progressive rollout by region; cell-based isolation; rate limits per tenant
Process Update runbooks; improve on-call training; clarify incident roles

Make actions SMART-like: specific, owned, with dates. Track them in the normal backlog and review completion.

Example

Incident: a configuration change set a cache TTL to 0, causing 40 minutes of database overload.

Findings: no validation of config values; change rolled out to all regions at once; alert fired only after 15 minutes; no quick config rollback.

Actions: schema validation and safe ranges for config; staged config rollout with automatic halt on SLO regression; burn-rate alert with a shorter window; config rollback command added to the runbook.

Learning Across Incidents

  • Tag incidents by cause category (change, capacity, dependency, data, human process).

  • Review trends quarterly: if 60 percent of incidents follow deploys, invest in safer deployments.

  • Share post-mortems widely so other teams learn without repeating the incident.

  • Review near misses too; they are cheaper lessons.

Pitfalls

  • Blame, which drives information underground.

  • Writing the document but never completing action items.

  • Action items that are too vague (“be more careful”).

  • Only reviewing large incidents.

Interview Tip

Describe the structure (timeline, impact, contributing factors, action items), stress blamelessness and tracking actions to completion, and give an example where the actions cover prevention, detection and mitigation. Mentioning trend analysis across incidents shows a strategic view.

19

How does system scalability affect availability?

Scalability affects availability because most outages under load are capacity failures: when demand exceeds what the system can handle, latency rises, requests time out, queues grow and components fail, often cascading to others. A system that scales smoothly (adds capacity automatically, has no single bottleneck, and degrades gracefully when it cannot keep up) stays available through traffic spikes and component failures. Scaling also adds more components and coordination, which brings new failure modes, so it must be paired with good isolation and automation.

Read full answer

Applies toweb services under traffic spikes, databases, queues and stream processors, autoscaling platforms and capacity planning

How Scalability Supports Availability

Scalability property Availability benefit
Horizontal scaling (more instances) Spikes are absorbed; losing one instance removes only a small share of capacity
Autoscaling Capacity follows demand before saturation
No single bottleneck (sharding, partitioning) One hot component does not cap the whole system
Stateless services Instances can be added, removed or replaced quickly
Headroom (N+1, N+2) Survives losing an instance or a zone at peak
Asynchronous processing Queues absorb bursts instead of failing requests

How Poor Scalability Hurts Availability

  • Saturation: CPU, memory, connection pools or threads run out; latency climbs and requests time out.

  • Cascading failure: a slow database makes app threads wait; app servers exhaust threads; the load balancer marks them unhealthy; remaining servers get more traffic and fail.

  • Retry storms: clients retry timeouts and multiply load on an already overloaded system.

  • Hot spots: one popular key, tenant or partition overloads a single shard.

  • Slow scaling: autoscaling that takes several minutes to add capacity cannot handle sudden spikes.

Example: Zone Failure at Peak

A service needs 12 instances at peak. It runs 12 across three zones (4 each). Zone A fails: 8 instances remain and must take 150 percent of their normal load, so they overload.

With N+1 at the zone level, it runs 18 instances (6 per zone); after losing a zone, 12 remain and handle the peak.

import math

def instances_needed(peak_instances, zones, zones_lost=1):
    per_zone = math.ceil(peak_instances / (zones - zones_lost))
    return per_zone * zones

print(instances_needed(12, 3))   # 18 (6 per zone, survives one zone loss)

Scaling Without Hurting Availability

  • Load test to know real limits and bottlenecks; plan capacity from forecasts.

  • Autoscale on leading indicators (requests per instance, queue depth) rather than only CPU; keep a minimum warm capacity.

  • Rate limiting and load shedding to protect the system when demand exceeds even the scaled capacity.

  • Backpressure in queues and streams so producers slow down.

  • Caching and CDNs to cut origin load.

  • Partitioning and sharding for data, with care for hot keys.

  • Graceful degradation: disable expensive features under load.

  • Isolation: cells or bulkheads so one tenant’s spike does not take down everyone.

New Risks From Scaling

Risk Mitigation
More components, more failures Automation, self-healing, health checks
Distributed coordination Avoid global locks; use partitioned designs
Autoscaling bugs or limits Test scaling paths; set sensible minimum and maximum; watch cloud quotas
Dependency limits Databases, third-party APIs and connection limits may not scale with your app

Interview Tip

Explain that capacity failures are a major cause of outages, show how horizontal scaling, headroom and autoscaling protect availability, and use the zone-failure capacity example. Mentioning cascading failures, retry storms and load shedding demonstrates real-world understanding.

20

Which strategies let a system scale while keeping or improving reliability?

To scale without losing reliability, grow capacity in ways that do not add single points of failure or shared bottlenecks, and protect the system when demand outpaces capacity. Key strategies: stateless horizontal scaling with autoscaling, partitioning and sharding data, caching and CDNs, asynchronous processing with queues and backpressure, isolation by cells or bulkheads, rate limiting and load shedding, safe progressive deployments, capacity planning with load tests, and strong observability and automation.

Read full answer

Applies togrowing web platforms, SaaS products, APIs, data pipelines and microservice architectures

Strategies and Their Reliability Effect

Strategy How it scales Reliability effect
Stateless services plus autoscaling Add instances on demand Instances are replaceable; failures affect a small share
Read replicas and caching Offload reads from the primary database Protects the primary from overload; cache can serve during brief outages
Sharding / partitioning Split data and traffic by key One shard failure affects only its users; no single hot database
Asynchronous queues Decouple producers and consumers Absorbs spikes; consumers retry safely with idempotency
Backpressure Producers slow down when consumers lag Prevents memory blow-ups and cascading overload
Cell-based architecture Many independent copies of the stack, each serving a subset of customers Limits blast radius of bugs, bad deploys and noisy tenants
Bulkheads Separate resource pools per dependency or tenant One slow dependency cannot exhaust all threads
Rate limiting and quotas Cap usage per client or tenant Protects shared capacity from abuse or runaway clients
Load shedding and prioritisation Drop low-priority work under overload Keeps critical traffic (checkout, login) working
CDN and edge caching Serve static and cacheable content near users Large share of traffic never reaches origin
Progressive delivery Canary and staged rollouts per cell or region Bad releases affect few users and roll back automatically
Capacity planning and load testing Measure limits before users hit them Fewer surprise saturation incidents

Example: Scaling an Order Service

  1. Stateless API in three zones with autoscaling on requests per instance.

  2. Orders written to a queue; workers process them idempotently (idempotency keys) and scale with queue depth.

  3. Database sharded by customer ID; read replicas for order history.

  4. Cache for product and price lookups with short TTLs.

  5. Cells: customers split into cells of roughly equal load, each with its own API, workers and shard.

  6. Load shedding: under overload, defer email notifications and analytics events, keep order placement working.

  7. Canary per cell: deploy to one cell first and watch SLOs before continuing.

Guardrails While Scaling

  • Avoid shared global resources: one global lock, sequence generator or config database becomes a bottleneck and SPOF.

  • Know your dependencies’ limits: database connections, third-party API quotas, cloud service limits.

  • Connection management: connection poolers (PgBouncer) prevent thousands of instances from exhausting database connections.

  • Retry discipline: exponential backoff with jitter and retry budgets; more instances retrying naively can overload a recovering service.

  • Test failure at scale: chaos experiments, zone failure drills, load tests with failures injected.

  • Automate operations: provisioning, scaling and recovery should not depend on humans.

Measure the Effect

  • SLOs per cell and per region, not only global.

  • Saturation metrics: queue lag, connection pool usage, shard hot spots.

  • Change failure rate and MTTR, which should stay stable or improve as the system grows.

Interview Tip

Present scaling strategies alongside the reliability effect of each, give a concrete example combining queues, sharding, cells and load shedding, and warn about shared bottlenecks and retry storms. Mentioning cell-based architecture and progressive rollouts shows awareness of how large systems limit blast radius.

21

How does caching affect system reliability, and what are the trade-offs?

Caching improves reliability by reducing load on databases and services, lowering latency, and allowing the system to serve responses even when the origin is slow or down (for example stale-if-error). But it introduces risks: stale or inconsistent data, cache stampedes when popular keys expire, dependence on the cache so that its failure overloads the origin, cold starts after restarts, and bugs from caching wrong or private data. Reliable designs treat the cache as an optimisation that can fail, with capacity to survive cache loss or controlled degradation.

Read full answer

Applies toCDN and edge caches, Redis and Memcached, in-process caches, HTTP caching, and database query caches

How Caching Helps Reliability

Benefit Explanation
Lower origin load A 95 percent hit ratio means only 5 percent of reads reach the database
Burst absorption Traffic spikes for popular content are served from cache
Lower latency Fewer timeouts, more headroom
Resilience to origin failure Serve stale content when the origin errors (stale-if-error, stale-while-revalidate)
Reduced dependency on slow services Cache third-party API responses

Risks and Mitigations

Risk What happens Mitigation
Stale data Users see old prices or profile data Appropriate TTLs; invalidation on write; versioned keys; do not cache data needing strong consistency
Cache stampede (thundering herd) A hot key expires; thousands of requests hit the database at once Request coalescing, locks per key, probabilistic early refresh, TTL jitter
Cache as a hidden dependency Cache cluster fails; database receives 20 times its normal load and collapses Size the origin for some cache loss; rate limit misses; replicated cache clusters; local fallback caches
Cold cache After restart or deploy, hit ratio is low; origin overloads Warm-up, gradual traffic shift, persistent caches
Caching errors A failed response is cached and served to everyone Do not cache 5xx; short negative-cache TTLs
Caching private data One user’s data shown to another Correct cache keys and Cache-Control: private for per-user responses
Inconsistent invalidation Some nodes have old data Event-driven invalidation, short TTLs as a safety net
Memory pressure and evictions Hit ratio drops suddenly Monitor evictions, size properly, choose eviction policy

Example: Preventing a Stampede

import threading, time

cache, locks = {}, {}
db_calls = [0]

def load_from_db(key):
    db_calls[0] += 1
    time.sleep(0.05)
    return f"value-for-{key}"

def get(key, ttl=60):
    entry = cache.get(key)
    if entry and entry[1] > time.monotonic():
        return entry[0]
    lock = locks.setdefault(key, threading.Lock())
    with lock:                                   # only one thread reloads this key
        entry = cache.get(key)
        if entry and entry[1] > time.monotonic():
            return entry[0]
        value = load_from_db(key)
        cache[key] = (value, time.monotonic() + ttl)
        return value

threads = [threading.Thread(target=get, args=("home_page",)) for _ in range(50)]
for t in threads: t.start()
for t in threads: t.join()
print("database calls:", db_calls[0])   # database calls: 1

dict.setdefault is atomic in CPython for simple keys, so all threads share one lock per key. Distributed systems use the same idea with a short-lived lock in Redis or request coalescing in the CDN.

HTTP Caching for Resilience

Cache-Control: public, max-age=60, stale-while-revalidate=30, stale-if-error=86400

The CDN serves fresh content for 60 seconds, refreshes in the background for 30 more, and can serve stale content for up to a day if the origin returns errors.

Design Guidelines

  • Decide which data may be stale and for how long; keep balances, stock and permissions on strongly consistent paths.

  • Plan for cache failure: can the origin survive a cold or missing cache? If not, add protections.

  • Monitor hit ratio, latency, evictions, memory and origin load.

  • Use TTL jitter to avoid synchronised expiry.

Interview Tip

Explain both sides: caching protects origins and enables stale-if-error, but adds staleness, stampedes and a hidden dependency. Describing how you would survive a total cache loss and prevent stampedes shows reliability-focused thinking.

22

How does rate limiting help preserve system availability?

Rate limiting preserves availability by capping how many requests a client, tenant or route can make in a time window, so that no single user, bug, bot or traffic spike can consume all capacity. Excess requests get a fast 429 Too Many Requests (with Retry-After) instead of slowing everyone down. It protects shared resources (databases, third-party quotas), ensures fairness between tenants, and stops retry storms and abuse from turning into outages. It works alongside load shedding, which drops work when the system as a whole is overloaded.

Read full answer

Applies topublic APIs, API gateways (Kong, AWS API Gateway, Envoy, NGINX), login and signup endpoints, multi-tenant SaaS, and calls to third-party APIs

What It Protects Against

Threat Example
Abusive or buggy clients A misconfigured script calling an API 1,000 times per second
Noisy neighbours One tenant’s batch job slowing down every other tenant
Brute force and credential stuffing Rapid login attempts
Scrapers and bots Aggressive crawling of search pages
Retry storms Clients retrying failed requests without backoff
Cost and quota exhaustion Overusing a paid third-party API

Common Algorithms

Algorithm Behaviour Notes
Fixed window Count per minute; reset each minute Simple; allows bursts at window edges
Sliding window log or counter Smooths window edges More accurate; moderately more state
Token bucket Tokens refill at a steady rate; each request spends one; bucket size allows bursts Most common for APIs
Leaky bucket Requests flow out at a constant rate Smooths traffic to a fixed rate
Concurrency limit Caps in-flight requests rather than rate Good for protecting slow back ends

Token Bucket Example

class TokenBucket:
    def __init__(self, rate_per_s, capacity):
        self.rate, self.capacity = rate_per_s, capacity
        self.tokens, self.last = capacity, 0.0

    def allow(self, now):
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

bucket = TokenBucket(rate_per_s=5, capacity=10)
burst = [bucket.allow(0.0) for _ in range(12)]
print(burst.count(True), "allowed in burst")       # 10 allowed in burst
print(bucket.allow(0.2), bucket.allow(0.2))        # True False  (one token refilled in 0.2 s)

Where to Apply

  • Edge / API gateway: per IP, API key or user, before requests reach services.

  • Per tenant: quotas aligned with plans; prevents noisy neighbours.

  • Per endpoint: stricter limits for expensive operations (search, exports, login).

  • Outbound: limit calls to databases or third-party APIs to stay within their capacity and quotas.

  • Distributed enforcement: shared counters in Redis or gateway-level limits so all instances agree; local limits as a fallback if the shared store fails.

Good Practices

  • Return 429 with Retry-After and rate-limit headers so well-behaved clients back off.

  • Fail open or closed deliberately if the rate limiter’s store is unavailable (often fail open with conservative local limits).

  • Prioritise traffic: keep limits looser for critical operations and authenticated customers.

  • Monitor rejected requests by client to spot abuse and misconfigured limits.

  • Combine with load shedding: rate limits protect against individual clients; load shedding protects against total overload even when every client is within its limit.

Trade-offs

  • Limits that are too strict block legitimate users; too loose and they do not protect.

  • Distributed counting adds latency and its own dependency.

  • IP-based limits can affect many users behind one NAT or proxy.

Interview Tip

Explain how rate limiting stops any single client from exhausting shared capacity, describe token bucket with a short example, and cover where to enforce limits and how clients should respond to 429s. Distinguishing rate limiting from load shedding shows a precise understanding.

23

How do eventual and strong consistency differ, and how do they affect reliability?

Strong consistency means every read returns the latest committed write, as if there were one copy of the data. Eventual consistency means replicas may temporarily return older values, but they converge once updates stop. For reliability, the difference is a trade-off between correctness and availability or latency: strong consistency prevents stale or conflicting data but requires coordination, so requests can fail or slow down when replicas are unreachable; eventual consistency keeps serving through failures and is faster, but the application must tolerate staleness and resolve conflicts, or users see wrong results. The partition case is the CAP trade-off.

Read full answer

Applies toreplicated databases, caches, multi-region systems, read replicas and event-driven architectures

Comparison

Aspect Strong consistency Eventual consistency
Read guarantee Latest committed value Possibly stale; converges later
Coordination Synchronous with a leader or majority Asynchronous replication
Latency Higher, especially across regions Lower; local reads and writes
Behaviour under replica or network failure Requests may fail or wait on the minority side Keeps serving, possibly stale
Conflicts Prevented Possible; need resolution (last-writer-wins, merges, CRDTs)
Application complexity Lower for correctness Higher: handle staleness, idempotency, reconciliation
Examples Single-leader SQL reads from primary, etcd, Spanner, DynamoDB strongly consistent reads Cassandra at ONE, DNS, CDN caches, async read replicas

Reliability Implications

Strong consistency

  • Correctness reliability is high: balances, inventory and permissions cannot be wrong due to replication lag.

  • Availability risk: if a majority of replicas or the leader is unreachable, writes (and strong reads) fail.

  • Latency risk: synchronous replication across regions adds round trips, increasing tail latency and timeouts.

  • Failover care: leader elections cause brief unavailability.

Eventual consistency

  • Availability is high: any replica can serve, so the system rides out replica and network failures.

  • Correctness risk: stale reads (a user does not see their own update), lost updates under last-writer-wins, oversold stock if used for invariants.

  • Hidden failures: replication lag can grow silently; monitor it.

  • Recovery work: repairs and reconciliation after failures.

Example: Read Replica Lag

A user updates their profile (written to the primary) and the next page reads from an asynchronous replica 2 seconds behind. They see their old name and think the update failed, then submit again.

Fixes: read-your-writes (read from the primary for a short time after a write, or pass a version token), monitoring replication lag, and routing critical reads to the primary.

Choosing Per Data Type

Data Recommended
Account balances, payments, stock reservations Strong
Authentication, permissions changes Strong (or short, bounded staleness)
Product catalogue, reviews, feeds Eventual, with reasonable TTLs or lag
Analytics, counters, likes Eventual (CRDT counters if multi-writer)
User’s own recent edits Session guarantees (read-your-writes) on top of eventual

Middle Ground

  • Bounded staleness: reads at most N seconds old (Azure Cosmos DB offers this level).

  • Causal consistency and session guarantees remove most confusing anomalies without full coordination.

  • Per-request choice: tunable consistency in Cassandra, DynamoDB and MongoDB.

Interview Tip

Define both models, then focus on the reliability trade-off: strong consistency protects correctness but can reduce availability and raise latency; eventual consistency stays up but risks stale or conflicting data. Choosing per data type and mentioning read-your-writes for replica lag shows practical judgement.

24

What is the CAP theorem, and how does it relate to system availability?

The CAP theorem says that when a network partition occurs, a replicated system must choose between consistency (every read sees the latest write) and availability (every non-failed node responds without error). For availability engineering, the key lesson is that a strongly consistent design will reject some requests during partitions (usually on the minority side), while an availability-first design keeps serving but may return stale or conflicting data. CAP describes only partition behaviour; practical availability also depends on redundancy, failover speed, capacity and operations, which is why CP systems can still reach very high uptime.

Read full answer

Applies toreplicated databases, multi-zone and multi-region designs, and choosing data stores for availability targets

CAP in One Scenario

Three replicas across zones A, B and C; zone C is cut off from the others.

Design Zone C during the partition Zones A and B
Consistency-first (CP), e.g. etcd, CockroachDB, MongoDB with majority concerns Rejects writes and strong reads Continue normally (they hold a majority)
Availability-first (AP), e.g. Cassandra at ONE or LOCAL_QUORUM per DC Continues serving, possibly stale Continue; replicas reconcile after healing

Partitions cannot be ruled out in a distributed system, so the choice is about behaviour during them, not about “picking two of three”.

Why It Matters for Availability Targets

  • CP systems lose availability only for clients that cannot reach the majority. With three zones and clients routed to healthy zones, users rarely notice, so CP systems commonly meet 99.99 percent targets in practice.

  • AP systems keep every side writable, which helps when users are pinned to an isolated region (multi-region active-active, mobile and offline clients).

  • PACELC adds the normal-case trade-off: strong consistency across regions costs latency on every write, which affects timeouts and perceived reliability.

How to Use CAP in Availability Design

  1. Classify data: correctness-critical (payments, stock, identity) vs staleness-tolerant (feeds, catalogues, sessions).

  2. Choose the partition behaviour per class: refuse (CP) or serve-and-reconcile (AP).

  3. Place replicas across three failure domains so a single failure leaves a majority for CP data.

  4. Route users to the healthy side with load balancers and health-checked DNS.

  5. Degrade gracefully for CP data on the minority side: read-only mode, queued operations, clear error messages.

  6. Plan reconciliation for AP data: merges, CRDTs, last-writer-wins only where acceptable.

CAP Is Not the Whole Availability Story

Most real outages come from bad deploys, configuration errors, overload and dependency failures, not partitions. Availability engineering also needs:

  • Redundancy and automated failover.

  • Safe deployments and rollback.

  • Capacity headroom, rate limiting and load shedding.

  • Timeouts, retries with backoff and circuit breakers.

  • Observability, SLOs and incident response.

Interview Tip

State CAP precisely (consistency vs availability during a partition), show the three-zone scenario, and explain that CP systems still achieve high practical availability because they serve the majority side. Adding that most outages are not partitions keeps the answer grounded in real operations.

25

How does quorum-based decision making affect reliability in distributed systems?

Quorum-based decision making requires a minimum number of nodes (usually a majority) to agree before an action, such as committing a write or electing a leader, takes effect. It improves reliability by tolerating failures without losing committed data (a cluster of 2f + 1 nodes survives f failures), preventing split brain (two disjoint majorities cannot exist), and keeping replicas consistent. The cost is that progress stops when a quorum is unreachable, writes need extra round trips, and clusters must be sized and placed carefully across failure domains.

Read full answer

Applies toconsensus systems (Raft in etcd, Consul, CockroachDB; ZAB in ZooKeeper; Paxos in Spanner), replicated databases with quorum reads and writes (Cassandra, Riak), Kafka's in-sync replicas, and failover managers

How Quorums Improve Reliability

Benefit Explanation
Fault tolerance 3 nodes tolerate 1 failure; 5 tolerate 2; 7 tolerate 3
Durability of committed data A write acknowledged by a majority survives the loss of any minority
Split-brain prevention Only one side of a partition can hold a majority, so only one leader can commit
Consistent leader election A new leader must win a majority vote and has the latest committed entries (Raft’s election restriction)
Read/write overlap With R + W > N, a read quorum overlaps the latest write quorum

Cluster Sizing

def tolerated_failures(n):
    return (n - 1) // 2

for n in (2, 3, 4, 5, 7):
    print(f"{n} nodes: quorum {n // 2 + 1}, tolerates {tolerated_failures(n)} failure(s)")
# 2 nodes: quorum 2, tolerates 0 failure(s)
# 3 nodes: quorum 2, tolerates 1 failure(s)
# 4 nodes: quorum 3, tolerates 1 failure(s)
# 5 nodes: quorum 3, tolerates 2 failure(s)
# 7 nodes: quorum 4, tolerates 3 failure(s)

A 4-node cluster tolerates no more failures than 3 but needs a larger quorum, so odd sizes are standard. Two nodes tolerate zero failures for writes, which is why two-node “HA” pairs need a witness or arbiter.

Placement Matters

  • Three zones, one node each: losing any zone keeps a majority.

  • Two zones (2 + 1 nodes): losing the zone with 2 nodes stops the cluster. Add a third site or a lightweight witness.

  • Multi-region: five nodes across three regions (2 + 2 + 1) survive a full region loss, at the cost of cross-region latency on writes.

Effects on Availability and Latency

  • Unavailability when quorum is lost: if a majority fails or is partitioned away, writes stop. This is deliberate: refusing writes is better than risking split brain or lost data (the consistency-first choice during partitions, as in CAP).

  • Latency: every commit waits for the slowest member of the fastest majority; in multi-region clusters this adds tens to over a hundred milliseconds.

  • Leader dependence: in Raft, a leader failure triggers an election that pauses writes briefly (often under a few seconds with default timeouts).

Reliability Pitfalls

Pitfall Mitigation
Even-sized or badly placed clusters Odd sizes across independent failure domains
Too many voters (slow commits) Keep voters at 3 or 5; use non-voting learners or read replicas for scale
Correlated failures (same rack, same bad config) Diverse placement; staged config rollouts
Disk fsync latency Fast, dedicated disks for consensus logs (etcd is very sensitive to this)
Clock- or timeout-tuning errors Election timeouts matched to network latency
Operating without quorum awareness Monitoring for leader changes, quorum health and member lag
Sloppy quorums (Dynamo-style) Understand that they trade consistency for availability during failures

Kafka Example

With replication.factor=3, min.insync.replicas=2 and producers using acks=all, a write is acknowledged only when at least two in-sync replicas have it. Losing one broker loses no acknowledged data; losing two makes the partition reject writes rather than risk loss.

Interview Tip

Explain the 2f + 1 rule with a sizing table, show how majorities prevent split brain, and discuss placement across three failure domains. Being clear that losing quorum stops writes by design, and that quorums add latency, gives a balanced answer.

26

What role do distributed transactions play in reliability, and what are the challenges?

Distributed transactions keep data consistent across several databases, partitions or services: either all participants commit or none do, so a failure halfway through does not leave money debited but not credited. They support reliability by preserving invariants despite crashes and network failures. The challenges are significant: protocols like two-phase commit (2PC) add latency, hold locks while waiting, can block if the coordinator fails, and reduce availability because every participant must be up. Many modern systems therefore use sagas, the transactional outbox and idempotent operations across services, and keep true distributed transactions inside databases designed for them (Spanner, CockroachDB, YugabyteDB).

Read full answer

Applies tomicroservices, sharded databases, payment and order processing, message brokers with transactional guarantees

Approaches

Approach How it works Reliability profile
Two-phase commit (2PC, XA) Coordinator asks all participants to prepare, then commit Atomic, but blocking if the coordinator fails after prepare; every participant must be available
Consensus-backed transactions (Spanner, CockroachDB) 2PC whose coordinator and participants are each replicated with Paxos or Raft Survives node failures without blocking; costs latency
Saga Sequence of local transactions; failures trigger compensating actions Highly available; eventual consistency; intermediate states are visible
Transactional outbox Write the state change and an event in one local transaction; a relay publishes the event Reliable event publishing without 2PC between database and broker
Kafka transactions Atomic writes to multiple partitions plus consumer offset commits Exactly-once processing within Kafka

Two-Phase Commit in Brief

  1. Prepare: coordinator asks each participant to promise it can commit; participants write prepare records and hold locks.

  2. Commit: if all vote yes, coordinator records the decision and tells everyone to commit; otherwise abort.

Problem: if the coordinator crashes after participants have prepared, they cannot decide alone and keep holding locks until it recovers. Replicating the coordinator (as consensus-based databases do) removes this blocking.

Saga Example: Order Placement

1. Order service:     create order (PENDING)
2. Payment service:   charge card          -> on failure: cancel order
3. Inventory service: reserve stock        -> on failure: refund payment, cancel order
4. Order service:     mark order CONFIRMED

Each step is a local transaction. Failures run compensations in reverse. Sagas can be orchestrated (a coordinator such as Temporal, AWS Step Functions or Camunda drives the steps) or choreographed (services react to events).

Challenges

Challenge Detail Mitigation
Latency Several network round trips per transaction Keep transactions small; co-locate related data
Availability 2PC needs every participant up Sagas across services; consensus-replicated participants
Lock contention Prepared transactions hold locks Short transactions; optimistic concurrency
Coordinator failure Blocking or uncertain outcomes Replicated coordinators; durable decision logs; recovery processes
Partial failures and retries Timeouts leave outcome unknown Idempotency keys; query status before retrying
Isolation in sagas Other users see intermediate states Semantic locks (PENDING states), commutative updates, careful ordering
Compensation complexity Some actions cannot be undone (sent email, shipped parcel) Order steps so irreversible ones come last; compensating business actions (refunds)
Heterogeneous systems Not every system supports XA Outbox, sagas, idempotent consumers
Observability Hard to follow multi-service flows Distributed tracing with correlation IDs; saga state dashboards

Choosing

  • Within one database (even sharded): use its transactions; distributed SQL handles cross-shard atomicity.

  • Across microservices: prefer sagas plus outbox plus idempotency; avoid 2PC across service boundaries.

  • Database plus message broker: transactional outbox or change data capture (Debezium).

Interview Tip

Explain why atomicity across participants protects invariants, describe 2PC and its blocking problem, and present sagas and the outbox as the reliable alternatives across services. Mentioning that consensus-based databases make 2PC non-blocking shows depth.

27

What is a disaster recovery plan, and how does it relate to reliability?

A disaster recovery (DR) plan is a documented, tested strategy for restoring systems and data after a major failure, such as a region outage, data centre fire, ransomware attack, accidental mass deletion or corrupted database. It defines recovery objectives (RPO: how much data loss is acceptable; RTO: how long recovery may take), the architecture that meets them (backups, replicas, standby regions), step-by-step procedures, roles and communication, and a testing schedule. It complements everyday high availability: HA handles routine failures automatically, while DR handles rare, large-scale events that HA cannot absorb.

Read full answer

Applies toproduction systems, databases, cloud infrastructure, regulated industries (finance, healthcare) and business continuity planning

Key Objectives

Term Meaning Example
RPO (recovery point objective) Maximum acceptable data loss, measured in time 5 minutes: you may lose up to 5 minutes of writes
RTO (recovery time objective) Maximum acceptable time to restore service 1 hour: service must be back within an hour
MTD (maximum tolerable downtime) Business limit before serious harm 4 hours for an online store

DR Strategies (Cost vs Speed)

Strategy Description Typical RPO Typical RTO Cost
Backup and restore Backups stored in another region; rebuild on disaster Hours (or less with continuous log shipping) Hours to days Low
Pilot light Core data replicated; minimal infrastructure ready to scale Minutes Tens of minutes to hours Medium
Warm standby Scaled-down full environment running in another region Seconds to minutes Minutes Higher
Multi-site active-active Full capacity in several regions serving traffic Near zero Near zero to minutes Highest

These categories follow AWS’s widely used DR guidance; exact numbers depend on implementation.

What a DR Plan Contains

  1. Scope and priorities: which systems are critical, their RPO and RTO, and dependencies.

  2. Disaster scenarios: region loss, data corruption, ransomware, provider account compromise, key personnel unavailable.

  3. Architecture: backups (with immutable or offline copies), cross-region replication, infrastructure as code for rebuilding.

  4. Runbooks: precise steps for failover and restore, including DNS changes, database promotion, secrets and configuration.

  5. Roles and decision rights: who declares a disaster and authorises failover.

  6. Communication: internal escalation, customer notifications, regulators if required.

  7. Testing schedule: restore tests, failover drills, game days.

  8. Failback plan: how to return to the primary site safely.

Relationship to Reliability

High availability Disaster recovery
Handles common failures (instance, disk, zone) Handles rare, catastrophic events
Automatic, seconds to minutes Often partly manual, minutes to hours
Redundancy within a region Copies in another region, account or offline
Protects uptime Protects business survival and data

Replication alone is not DR for data corruption: a bad migration or ransomware encryption replicates to every replica. Point-in-time backups and immutable copies are required.

Testing Is Essential

  • Restore tests: regularly restore backups to a clean environment and verify data integrity and timing.

  • Failover drills: switch to the DR region in a controlled exercise; measure actual RTO and RPO.

  • Tabletop exercises: walk through scenarios with the team.

  • Untested plans often fail due to missing permissions, outdated runbooks, expired credentials or forgotten dependencies.

Interview Tip

Define DR with RPO and RTO, describe the strategy spectrum from backup-and-restore to active-active, and explain how DR complements HA. Pointing out that replication does not protect against corruption or ransomware, and that only tested plans count, shows real experience.

28

How do backup and restore operations affect system availability?

Backups affect availability in two directions. During normal operation, taking backups can reduce performance or require downtime if done poorly (locking tables, saturating disk I/O or network). During a disaster or data corruption, restore speed determines how long the system stays down, so RTO is often limited by how quickly you can restore, and RPO by how often backups or logs are captured. Well-designed backup strategies use online, incremental and snapshot-based backups, run from replicas, keep point-in-time recovery, and are regularly restore-tested, so they protect data without harming uptime.

Read full answer

Applies todatabases (PostgreSQL, MySQL, MongoDB), file and object storage, virtual machine and volume snapshots, Kubernetes state, and SaaS data

Impact During Normal Operation

Issue Effect Mitigation
Locking backups Writes blocked while dumping tables Online backup tools (pg_basebackup, Percona XtraBackup, MongoDB oplog-based backups)
I/O and CPU load Higher latency for users Back up from a replica; throttle bandwidth; schedule during low traffic
Large full backups Long windows; heavy network usage Incremental or differential backups; continuous WAL or binlog archiving
Snapshot consistency Crash-inconsistent copies of running databases Use database-aware snapshot procedures or freeze briefly
Backup storage failure Backups silently fail Monitoring, alerts on failed or missing backups

Impact During Recovery

  • RTO: restoring a multi-terabyte database from object storage can take hours. Faster options include restoring from snapshots, keeping a delayed replica, or failing over to a standby.

  • RPO: daily backups mean up to a day of data loss; continuous log archiving enables point-in-time recovery to within seconds or minutes.

  • Partial restores: restoring one table or tenant is often faster than a full restore; design for it.

  • Dependencies: restored data must match application versions, schemas and encryption keys.

def restore_hours(size_tb, throughput_mb_s):
    seconds = size_tb * 1024 * 1024 / throughput_mb_s
    return seconds / 3600

for mb_s in (200, 1000):
    print(f"5 TB at {mb_s} MB/s: {restore_hours(5, mb_s):.1f} h")
# 5 TB at 200 MB/s: 7.3 h
# 5 TB at 1000 MB/s: 1.5 h

Estimates like this, checked against real restore tests, show whether backups alone can meet the RTO or a standby is required.

Good Backup Practices

  • 3-2-1 rule: at least three copies, on two different media or services, with one off-site (another region or account).

  • Immutable or air-gapped copies (object lock, separate accounts) to survive ransomware and accidental deletion.

  • Encryption at rest and in transit, with keys stored and backed up separately.

  • Retention policies that meet business and legal requirements.

  • Point-in-time recovery via WAL or binlog archiving.

  • Automate and monitor: alerts when a backup fails or is older than expected.

  • Test restores regularly: measure real restore time and verify data integrity; a backup that has never been restored is unproven.

Backups vs Replication

Aspect Replication Backups
Protects against hardware or zone failure Yes, with fast failover Yes, slowly
Protects against corruption, bad migrations, deletion, ransomware No: errors replicate Yes: restore to before the error
Recovery speed Seconds to minutes Minutes to hours

Both are needed: replicas for availability, backups for recoverability.

Interview Tip

Explain how backups can hurt performance (and how to avoid it with online backups from replicas) and how restore speed sets RTO while backup frequency sets RPO. Stressing restore testing, immutable copies and the difference between backups and replication shows operational maturity.

29

Why is data replication important for high availability, and what are its challenges?

Data replication keeps copies of data on several nodes, often across zones or regions, so that when one node or location fails, another copy can serve reads and take over writes. It is central to high availability because stateless services can be replaced instantly, but data must already exist elsewhere for the system to continue. The challenges are replication lag and stale reads, data loss on failover with asynchronous replication, write latency with synchronous replication, split brain and conflicting writes, failover complexity, and the fact that replication copies mistakes too.

Read full answer

Applies torelational databases (PostgreSQL, MySQL), NoSQL stores (Cassandra, MongoDB, DynamoDB), caches (Redis), message brokers (Kafka) and object storage

Why It Matters

Benefit Explanation
Failover A replica becomes primary when the primary fails
Read scaling Replicas serve read traffic
Geographic resilience Copies in other zones or regions survive local disasters
Low-latency reads Users read from nearby replicas
Maintenance Upgrade or repair nodes while others serve

Replication Models

Model Description Examples
Single leader One primary accepts writes; replicas follow PostgreSQL streaming replication, MySQL, MongoDB replica sets
Multi-leader Several nodes accept writes; changes exchanged Multi-region MySQL setups, CouchDB, DynamoDB global tables
Leaderless Any replica accepts writes; quorums decide Cassandra, Riak
Consensus-based Majority must agree on each write etcd, CockroachDB, Spanner, Kafka KRaft metadata

Synchronous vs Asynchronous

Aspect Synchronous Asynchronous
Write acknowledged after Replica(s) confirm Primary writes locally
Data loss on primary failure None for acknowledged writes (RPO about 0) Recent writes may be lost (RPO = lag)
Write latency Higher (extra round trip) Lower
Availability impact Writes block if the synchronous replica is unreachable Writes continue
Common compromise Semi-synchronous, or synchronous to one nearby replica plus asynchronous to others

Challenges and Mitigations

Challenge Mitigation
Replication lag causes stale reads Read-your-writes routing; monitor lag; read critical data from primary
Data loss on asynchronous failover Synchronous or quorum replication for critical data; measure lag against RPO
Split brain (two primaries) Quorum-based failover, fencing tokens, STONITH, leases
Conflicting writes (multi-leader, leaderless) Single home region per record, CRDTs, conflict resolution rules
Failover complexity Automated, tested failover (Patroni, managed Multi-AZ); clear client reconnection
Replicating mistakes Point-in-time backups and delayed replicas to recover from bad deletes or migrations
Cross-region bandwidth and cost Compress, filter, replicate only what is needed
Schema changes Backward-compatible migrations applied in a safe order
Monitoring blind spots Alerts on lag, replica health, failed replication slots, disk usage on primaries

Example: PostgreSQL HA Setup

  • Primary in zone A, synchronous standby in zone B (synchronous_commit = on, quorum of one standby), asynchronous replica in zone C for reads and extra safety.

  • Patroni with etcd manages leader election and fencing.

  • Clients connect through a proxy or DNS name that follows the leader.

  • WAL archiving to object storage for point-in-time recovery.

A zone A failure leads to automatic promotion of the zone B standby with no acknowledged data loss; zone C keeps serving reads.

Interview Tip

Explain why stateful data needs replication for HA, compare synchronous and asynchronous trade-offs (latency vs RPO), and cover lag, split brain, conflicts and the “replication copies mistakes” problem. A concrete setup with automated failover and backups shows practical depth.

30

How would you plan a failover strategy for a multi-region deployment?

A multi-region failover plan starts from business objectives (RTO, RPO, which features must survive) and then defines the topology (active-passive warm standby or active-active), how data is replicated to the other region, how failure is detected and who decides to fail over, how traffic is moved (DNS, global load balancer, anycast), how dependencies and capacity are prepared in the target region, and how to fail back. Above all, the plan must be automated where safe, documented, and regularly tested, because untested failover usually fails when needed.

Read full answer

Applies tocloud deployments on AWS, GCP and Azure, global SaaS platforms, payment and e-commerce systems with regional disaster requirements

Step 1: Set Objectives

Item Example
RTO Service restored within 15 minutes of a regional outage
RPO At most 1 minute of data loss for orders; zero for payments
Scope Checkout, login and search must survive; reporting can wait
Traffic Survive full peak traffic in the remaining region

Step 2: Choose a Topology

Topology Description Trade-offs
Active-passive (warm standby) Region A serves; Region B runs scaled-down copies with replicated data Lower cost and simpler data model; failover takes minutes; standby may be under-tested
Active-active Both regions serve users Fast failover and continuous testing; data must handle multi-region writes or have home regions per record; higher cost
Pilot light Region B has data replicas and minimal infrastructure Cheapest standby; slower RTO

Step 3: Data Strategy

  • Databases: cross-region replicas (Aurora Global Database, Cloud SQL cross-region replicas, distributed SQL across regions). Asynchronous replication means RPO equals lag; monitor it against the target.

  • Critical zero-loss data: synchronous or consensus-based replication across regions (higher write latency), or write payments to a ledger designed for multi-region durability.

  • Active-active data: assign each user or tenant a home region for writes, or use conflict-tolerant designs (CRDTs, last-writer-wins only where acceptable); the consistency choices during inter-region partitions follow the CAP trade-off.

  • Object storage, queues, search indexes, caches: decide which to replicate and which to rebuild.

  • Secrets, configuration and keys must exist in both regions.

Step 4: Detection and Decision

  • Health checks and SLO burn rates measured from outside the failing region.

  • Decide between automatic failover (fast but risky with false positives) and human-approved failover with a clear decision owner. Many teams automate stateless traffic shifts and require approval for database promotion.

  • Guard against split brain: fence the old primary (revoke its write access) before promoting the new one.

Step 5: Traffic Shifting

Mechanism Notes
DNS failover (Route 53, Cloud DNS, Cloudflare) Simple; limited by TTLs and resolver caching; use 30 to 60 second TTLs
Global load balancers / anycast (Google Cloud LB, AWS Global Accelerator, Cloudflare) Faster, health-checked shifting
Client-side region selection Mobile apps or SDKs with fallback endpoints

Step 6: Capacity and Dependencies

  • The surviving region must handle full traffic: pre-provision, or ensure autoscaling and cloud quotas allow rapid growth.

  • Check regional dependencies: identity providers, third-party APIs, control planes, container registries, CI/CD.

  • Avoid hidden cross-region dependencies (for example a global service whose control plane lives in the failed region).

Step 7: Runbook

1. Confirm regional impact (external probes, provider status, SLO dashboards)
2. Incident commander approves failover
3. Freeze deploys; fence primary database in Region A
4. Promote replica in Region B; verify replication position vs RPO
5. Scale Region B to full capacity
6. Shift traffic (global LB weights or DNS)
7. Verify SLIs; communicate status
8. Reconcile any writes accepted in Region A after the last replicated point
9. Plan failback once Region A is healthy and resynchronised

Step 8: Test

  • Scheduled regional failover drills (quarterly or more often), ideally in production with controlled traffic.

  • Chaos experiments: block inter-region traffic, fail a region’s database.

  • Measure actual RTO and RPO and fix gaps.

Interview Tip

Structure the answer as objectives, topology, data, detection, traffic, capacity, runbook and testing. Highlighting RPO and replication lag, split-brain fencing, full-capacity planning and regular drills shows you understand what makes multi-region failover actually work.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 30 Availability & Reliability answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All System Design subjects