Skip to content
System Design

Load Balancing interview questions and answers

50 questions, 50 with code. About 3 h 52 min to read them all.

By TestLab · Updated

Plan for an interview
50 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

Define load balancing in the context of modern web services.

Load balancing is the practice of distributing incoming requests or connections across a pool of servers (instances, containers, pods) so that no single server is overwhelmed, the service stays available when servers fail, and it can scale horizontally by adding more servers behind a single entry point. A load balancer sits between clients and back ends, exposing one address (a virtual IP or DNS name). It: - chooses a back end for each request or connection using an algorithm (round robin, least connections, hashing); - health-checks back ends and stops sending traffic to unhealthy ones.

In modern web services, load balancing happens at several layers: - DNS and global load balancing across regions; - edge and CDN load balancing; - Layer 4 (TCP/UDP) load balancers; - Layer 7 (HTTP) load balancers and reverse proxies, which also handle TLS termination, routing by path or host, rate limiting and WAF rules; - inside clusters: Kubernetes Services, service meshes and client-side balancing in gRPC.

Read full answer

Applies toweb and cloud architectures (nginx, HAProxy, Envoy, cloud load balancers, Kubernetes)

Where Load Balancing Happens

users -> DNS / global LB (pick region) -> CDN / edge -> L7 load balancer (TLS, routing, WAF)
      -> service instances (pods/VMs) -> internal LB, service mesh or client-side LB -> other services

Layer 4 Versus Layer 7

Layer 4 (transport) Layer 7 (application)
Sees IPs, ports, TCP/UDP HTTP method, path, headers, cookies, gRPC methods
Balances Connections Individual requests
Features Very fast, protocol-agnostic, preserves end-to-end TLS TLS termination, content routing, retries, header rewriting, compression, WAF
Examples AWS NLB, Google TCP/UDP LB, IPVS, Maglev AWS ALB, nginx, HAProxy (HTTP mode), Envoy, Traefik

Example: nginx as an HTTP Load Balancer

upstream app {
    least_conn;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.13:8080 backup;
    keepalive 64;
}

server {
    listen 443 ssl;
    server_name shop.example;
    ssl_certificate     /etc/ssl/shop.crt;
    ssl_certificate_key /etc/ssl/shop.key;

    location / {
        proxy_pass http://app;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Open-source nginx marks servers down passively (after failed requests); active health checks need NGINX Plus, HAProxy, Envoy or a cloud load balancer.

What It Enables

Goal How load balancing helps
Scalability Add servers behind the same address
Availability Route around failed instances and zones
Performance Spread load; send users to nearer or less-busy servers
Zero-downtime deployments Drain old instances, shift traffic gradually (canary, blue-green)
Security Single hardened entry point: TLS, WAF, rate limits, DDoS absorption

Pitfalls

  • The load balancer as a single point of failure: run it redundantly (managed services, active-passive pairs, anycast).

  • Losing client information: forward X-Forwarded-For or use the PROXY protocol.

  • Stateful servers that break when requests move between instances.

  • Health checks that are too shallow or too deep.

Interview Tips

  • Define it as distributing traffic across servers for scalability, availability and performance.

  • Contrast L4 and L7, and name the layers (DNS/global, edge, cluster, mesh).

  • Mention health checks, algorithms and redundancy for the load balancer itself.

2

What are the primary objectives of implementing load balancing?

The primary objectives of load balancing are: 1. High availability and fault tolerance: detect failed or unhealthy servers with health checks and route around them, across instances, zones and even regions, so users see no outage. 2. Scalability: serve more traffic by adding servers horizontally behind one stable endpoint, often automatically with autoscaling. 3. Performance and efficient resource use: spread work so no server is overloaded while others sit idle, keeping latency (especially tail latency) low, and route users to nearby or faster back ends. 4. Maintainability and safe change: take servers out for maintenance or deployments without downtime (connection draining, rolling, blue-green and canary releases). 5. Security and traffic management: provide one controlled entry point for TLS termination, WAF and rate limiting, absorb DDoS traffic, and hide internal topology. 6. Flexibility: route by content (path, host, header), version or tenant, and manage traffic across clouds and data centres.

Read full answer

Applies toload balancing design goals (vendor-neutral)

Objectives and Mechanisms

Objective Mechanisms Measured by
Availability Health checks, failover, multi-zone and multi-region pools, redundant LBs Uptime, error rate during failures
Scalability Horizontal pools, autoscaling integration, consistent hashing for caches Throughput versus instances
Performance Least-connections, power of two choices, latency-aware and geo routing, keep-alive pools p50/p95/p99 latency, server utilisation balance
Safe deployments Connection draining, weighted traffic shifting, canaries Deploy-related incidents, rollback time
Security TLS termination, WAF, rate limiting, DDoS protection, IP allow lists Blocked attacks, certificate hygiene
Traffic control Path/host routing, header routing, A/B splits Correct routing; experiment control
Cost efficiency Higher utilisation, fewer idle servers, cheaper regions Cost per request

Example: Objectives in a Cloud Setup

Route 53 latency/failover records      -> region choice + regional failover (availability, performance)
ALB in 3 availability zones            -> zone failure tolerance (availability)
  target group: auto scaling group      -> add/remove instances with load (scalability, cost)
  health check /ready every 10 s        -> remove bad instances (availability)
  deregistration delay 30 s             -> drain during deploys (safe change)
  HTTPS listener + WAF + rate rules     -> single secured entry (security)
  weighted target groups 95/5           -> canary release (safe change)

Trade-Offs Between Objectives

Tension Example
Affinity versus even spread Sticky sessions help caches and stateful apps but unbalance load
Deep health checks versus stability Checking dependencies can remove every server when the database blips
Latency versus availability Nearest-region routing versus failing over to a distant region
Security inspection versus latency WAF and TLS termination add processing time

Pitfalls

  • Focusing only on spreading load and ignoring health checks and failover.

  • Making the load balancer a single point of failure.

  • Not planning capacity for failure: with N servers, losing one must not overload the rest (N+1 or N+2 sizing).

Interview Tips

  • List the objectives: availability, scalability, performance, maintainability, security and flexibility.

  • Map each one to concrete mechanisms.

  • Mention trade-offs such as stickiness and health-check depth.

3

Explain the difference between hardware and software load balancers.

Hardware load balancers are dedicated physical appliances from vendors such as F5 BIG-IP, Citrix NetScaler (now Cloud Software Group) and A10, with specialised hardware (ASICs, FPGAs, crypto accelerators) for very high throughput, connection rates and TLS offload. They come with vendor support, rich traffic-management features, and often certified security. They are expensive, scale vertically (buy a bigger box), need physical installation and capacity planning, and fit poorly with elastic, automated cloud environments. Software load balancers run on commodity servers, VMs or containers: nginx, HAProxy, Envoy, Traefik, the Linux kernel’s IPVS, Katran (XDP/eBPF), and the managed cloud load balancers, which are software-defined underneath. They are cheap or free, elastic, programmable and automatable (API and configuration-as-code), and they scale horizontally. Their performance depends on the host, though modern software (eBPF, DPDK, multi-core proxies) reaches very high rates. Today most new deployments use software or cloud load balancers. Hardware appliances remain in large on-premises data centres, at telco and financial edges, and where existing investment or specific features matter. Most hardware vendors also sell virtual editions of the same software.

Read full answer

Applies toon-premises and cloud load-balancing options

Comparison

Aspect Hardware appliance Software load balancer
Form Physical device with specialised chips Software on servers, VMs, containers; managed cloud services
Performance Very high and predictable per box High; scales out with more instances
Scaling Vertical (bigger or more appliances) Horizontal and elastic
Cost High purchase and support cost Free or open source, or pay-as-you-go
Deployment speed Weeks (procurement, racking) Minutes (automation, infrastructure as code)
Flexibility Vendor features and scripting (for example iRules) Highly configurable and programmable; integrates with CI/CD, Kubernetes, service discovery
High availability Active-standby pairs, vendor clustering Multiple instances, anycast, managed redundancy
Lock-in Vendor-specific Lower (open-source options)
Best for High-volume on-premises data centres, TLS-heavy legacy environments, regulated setups Cloud, containers, microservices, most modern applications

Software Load Balancer Examples

Tool Layer Notes
nginx L4/L7 Web server and reverse proxy; very common
HAProxy L4/L7 High performance, rich health checks and stick tables
Envoy L4/L7 Dynamic configuration (xDS); basis of Istio and many gateways
Traefik L7 Automatic discovery from Docker and Kubernetes
IPVS / LVS L4 Linux kernel; used by kube-proxy in IPVS mode
Katran, Cilium (eBPF/XDP) L4 Very fast packet-level load balancing on commodity hardware
Cloud LBs (ALB, NLB, Azure Load Balancer, Google Cloud Load Balancing) L4/L7 Managed, autoscaling, pay per use

Example: HAProxy (Software)

frontend web
    bind :443 ssl crt /etc/haproxy/certs/shop.pem
    default_backend app

backend app
    balance leastconn
    option httpchk GET /ready
    http-check expect status 200
    server app1 10.0.1.11:8080 check inter 5s fall 3 rise 2
    server app2 10.0.1.12:8080 check inter 5s fall 3 rise 2

Choosing

Situation Choice
Cloud-native application Managed cloud load balancer, plus Ingress or Gateway in Kubernetes
On-premises with automation HAProxy, nginx or Envoy on VMs; eBPF L4 for very high volumes
Existing large F5 or NetScaler estate Keep, or move to the vendors’ virtual and cloud editions gradually
Extreme L4 volume eBPF/XDP-based load balancers or hardware

Pitfalls

  • Overbuying hardware for workloads that need elasticity.

  • Under-provisioning software load balancers (CPU for TLS, file descriptor limits, kernel tuning).

  • Forgetting redundancy for whichever load balancer you choose.

Interview Tips

  • Contrast dedicated appliances (performance, cost, vertical scaling) with software (elastic, cheap, programmable).

  • Name examples: F5 and NetScaler versus nginx, HAProxy, Envoy and cloud LBs.

  • Note the trend: software and cloud dominate, and vendors offer virtual editions.

4

What are common load balancing algorithms and how do they work?

Load balancing algorithms decide which back end receives the next request or connection. They fall into two families: - Static algorithms ignore current load: round robin (take turns), weighted round robin (more turns for bigger servers), random, and hash-based methods (source IP hash, or a URL or header hash, often with consistent hashing) so the same key lands on the same server. - Dynamic algorithms use live state: least connections (fewest active connections), weighted least connections, least outstanding requests, least response time or latency-aware (EWMA) methods, resource-based methods (CPU or agent-reported load), and power of two random choices (sample two servers, pick the less loaded), which gets near-optimal balance cheaply.

Static methods suit uniform, short requests; dynamic methods handle uneven or long requests; hashing gives affinity for caches and sessions.

Read full answer

Applies tonginx, HAProxy, Envoy, cloud load balancers (Python 3.12 example)

Code Example: Several Algorithms

import bisect
import hashlib
import itertools
import random
from collections import Counter

SERVERS = ["a", "b", "c"]
WEIGHTS = {"a": 5, "b": 3, "c": 2}

# Round robin: take turns
rr = itertools.cycle(SERVERS)
print("round robin:", [next(rr) for _ in range(6)])           # ['a', 'b', 'c', 'a', 'b', 'c']


# Smooth weighted round robin (as in nginx): spreads picks instead of bursting
def weighted_rr(weights):
    current = dict.fromkeys(weights, 0)
    total = sum(weights.values())
    while True:
        for s in weights:
            current[s] += weights[s]
        best = max(current, key=current.get)
        current[best] -= total
        yield best


w = weighted_rr(WEIGHTS)
print("weighted rr:", "".join(next(w) for _ in range(10)))   # abcaabacba: a 5, b 3, c 2 per 10, interleaved

# Least connections: pick the server with the fewest active connections
active = {"a": 12, "b": 4, "c": 9}
print("least conn:", min(active, key=active.get))             # b

# Power of two choices: sample two at random, take the less loaded one (cheap, near-optimal)
rng = random.Random(7)
picks = Counter()
for _ in range(1000):
    x, y = rng.sample(SERVERS, 2)
    choice = x if active[x] <= active[y] else y
    picks[choice] += 1
print("p2c:", dict(sorted(picks.items())))                    # {'b': 649, 'c': 351}: never the busiest (a)


# IP hash / consistent hashing: the same client key maps to the same server
def h(key):
    return int(hashlib.md5(key.encode()).hexdigest(), 16)    # non-cryptographic use: placement only


ring = sorted((h(f"{s}#{v}"), s) for s in SERVERS for v in range(100))   # 100 virtual nodes each
points = [p for p, _ in ring]


def lookup(key):
    i = bisect.bisect(points, h(key)) % len(ring)
    return ring[i][1]


clients = [f"10.0.0.{i}" for i in range(1000)]
before = {c: lookup(c) for c in clients}
ring = sorted(ring + [(h(f"d#{v}"), "d") for v in range(100)])            # add a 4th server
points = [p for p, _ in ring]
moved = sum(before[c] != lookup(c) for c in clients)
print(f"consistent hash: {moved / len(clients):.0%} of clients moved after adding a server")
# 20%: close to the ideal 1/4; a plain "hash mod N" would move about 3/4 of clients

Smooth weighted round robin interleaves picks (a, b, c, a, a, …) instead of sending five requests in a row to a. Power of two choices never chooses the busiest server. Consistent hashing with virtual nodes moves only about 1/N of keys when a server is added, while plain hash mod N would move most of them.

Summary

Algorithm How it works Good for Weakness
Round robin Next server in order Equal servers, uniform requests Ignores load and request cost
Weighted round robin Servers get turns in proportion to weights Mixed server sizes, canaries Weights are static
Random Pick uniformly at random Simple, stateless distributed balancers Short-term imbalance
Least connections Fewest active connections Long or variable connections (databases, WebSockets) Connection count may not reflect real load
Least outstanding requests (Envoy, AWS ALB) Fewest in-flight requests HTTP/2 and gRPC, variable latency Needs per-request tracking
Least response time / EWMA Lowest observed latency (often combined with active requests) Heterogeneous back ends Can herd onto a server that just got fast
Power of two choices Sample two, choose the less loaded Large pools, many balancers with stale information Slightly worse than global least-loaded
Source IP hash Hash of the client IP selects the server Simple affinity Uneven with NAT (many users behind one IP); remaps on pool change
Consistent hashing (ring hash, Maglev, rendezvous) Keys map to points on a ring or table Caches, sharded state, sticky routing Hot keys overload one server

Configuration Examples

upstream api {
    least_conn;                                  # or: hash $request_uri consistent; or: ip_hash;
    server 10.0.1.11:8080 weight=5;
    server 10.0.1.12:8080 weight=3;
    server 10.0.1.13:8080 weight=2;
}
backend api
    balance leastconn        # roundrobin | static-rr | source | uri | hdr(X-Tenant) | random(2)
    hash-type consistent

random(2) in HAProxy is the power-of-two-choices algorithm.

Choosing

Workload Suggested algorithm
Uniform, short HTTP requests Round robin or random
Variable request costs Least outstanding requests, power of two choices
Long-lived connections (WebSockets, databases) Least connections
Cache servers, sharded data Consistent hashing
Servers of different sizes Weighted variants
Canary releases Weighted round robin (95/5)

Pitfalls

  • Round robin with very uneven request costs, which builds queues on unlucky servers.

  • IP hash behind corporate NAT or a CDN: many users share one IP.

  • Many balancers each running least connections with only their local view (power of two choices handles this better).

Interview Tips

  • Separate static (round robin, weighted, hash) from dynamic (least connections, latency, P2C).

  • Explain consistent hashing and power of two choices.

  • Match algorithms to workload characteristics.

5

Describe the term “sticky session” in load balancing.

A sticky session (session affinity or session persistence) means the load balancer sends all requests from the same client session to the same back-end server, instead of choosing a server independently for each request. It is used when servers keep per-user state in local memory, such as an in-memory login session, shopping cart, upload in progress or WebSocket context, so a request landing on a different server would not find it. The load balancer identifies the client by: - a cookie: either one the load balancer inserts (for example AWS ALB’s AWSALB, or HAProxy and nginx inserted cookies), or the application’s own session cookie; - the source IP address (IP hash); - a header or URL parameter, such as a tenant or session ID.

Stickiness is easy to enable but has costs: uneven load (heavy users pinned to one server), lost sessions when a server fails or is replaced, and harder scaling and deployments. The preferred design is stateless servers with sessions in a shared store (Redis or a database) or in signed tokens. Use stickiness as an optimisation (cache locality) or for legacy apps, not as a correctness requirement.

Read full answer

Applies toHTTP load balancers (HAProxy, nginx, Envoy, AWS ALB, Azure Application Gateway, Google Cloud LB)

How It Works (Cookie Insertion)

1. First request: no cookie -> LB picks server B (normal algorithm)
2. LB response adds: Set-Cookie: SERVERID=B; Path=/; HttpOnly; Secure
3. Next requests carry SERVERID=B -> LB sends them to B
4. If B is unhealthy -> LB picks another server; the session state on B is lost

Configuration Examples

backend app
    balance roundrobin
    cookie SERVERID insert indirect nocache httponly secure
    server app1 10.0.1.11:8080 check cookie app1
    server app2 10.0.1.12:8080 check cookie app2
upstream app {
    ip_hash;                       # open-source nginx: affinity by client IP
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

Cookie-based sticky directives are an NGINX Plus feature. In Kubernetes, Services support sessionAffinity: ClientIP, and Ingress controllers offer cookie affinity through annotations.

Methods Compared

Method Pros Cons
LB-inserted cookie Precise per browser; works behind NAT Needs L7; cookies must be accepted
Application cookie Reuses the existing session ID LB must parse it; the app controls the lifetime
Source IP hash Works at L4; no cookies Many users behind one NAT IP; mobile IPs change
Header or URL key Good for APIs (tenant ID) Clients must send it

Trade-Offs

Benefit Cost
Works with in-memory sessions Uneven load (“hot” servers)
Better local cache hit rates Session loss on failure, scale-in or deployment
Keeps WebSocket and long-poll context together Draining takes longer; autoscaling is less effective

Better Alternatives

  • Externalise session state to Redis, Memcached or a database.

  • Stateless tokens: signed cookies or JWTs with server-side revocation where needed.

  • Consistent hashing for cache locality, which degrades gracefully when servers change.

Pitfalls

  • Long stickiness durations that keep users on overloaded or old-version servers.

  • Assuming stickiness guarantees state survival: it does not survive server failure.

  • IP-based stickiness behind CDNs or carrier NAT.

  • Deployments that ignore stickiness and cut off pinned users abruptly (drain first).

Interview Tips

  • Define it as routing a session’s requests to the same server, usually via a cookie or IP hash.

  • Explain why it is used (local state) and its costs (imbalance, failover loss).

  • Recommend stateless servers with shared session stores as the better design.

6

How does load balancing improve application reliability?

Load balancing improves reliability by removing single points of failure among application servers and reacting automatically to failures: - Redundancy: several instances serve the same application, ideally across availability zones, so losing one machine or even one zone does not take the service down. - Health checking: the load balancer probes each back end (active checks) and watches real traffic for errors and timeouts (passive checks, outlier detection). It removes failing instances from rotation within seconds and adds them back when they recover. - Failover and retries: requests are routed to healthy servers. Some L7 balancers retry idempotent requests on another back end. - Overload protection: spreading load prevents any server from being pushed into failure; connection limits, queues, rate limiting and load shedding protect the back ends. - Safe changes: connection draining, rolling, blue-green and canary releases let you deploy and patch without downtime, and roll back quickly. - Geographic resilience: global load balancing fails traffic over to another region during a regional outage.

The load balancer itself must also be highly available: redundant pairs, managed multi-zone services, or anycast.

Read full answer

Applies toreliability engineering with load balancers (vendor-neutral)

Failure Scenarios

Failure Without load balancing With load balancing
One server crashes Users on it get errors until manual action Health check fails → traffic moves in seconds
Server overloaded or slow Requests time out Least-request routing avoids it; outlier detection ejects it
Bad deployment on one instance Partial outage Canary metrics catch it; traffic shifted back
Availability zone outage Full outage if single-zone Cross-zone pools keep serving
Region outage Full outage Global LB or DNS failover to another region
Maintenance or patching Downtime window Drain and patch instances one at a time

Example: Envoy Outlier Detection and Retries

clusters:
  - name: orders
    connect_timeout: 0.5s
    lb_policy: LEAST_REQUEST
    outlier_detection:
      consecutive_5xx: 5
      interval: 10s
      base_ejection_time: 30s
      max_ejection_percent: 50
    health_checks:
      - timeout: 1s
        interval: 5s
        unhealthy_threshold: 3
        healthy_threshold: 2
        http_health_check:
          path: /ready

In the route configuration, a retry policy such as retry_on: "5xx,reset,connect-failure" with num_retries: 2 retries failed idempotent requests on other hosts. max_ejection_percent stops the balancer ejecting too many hosts at once, which could overload the rest.

Capacity Planning for Failure

Peak load needs 6 servers. To survive losing one zone out of 3:
each zone must carry 1/2 of peak when one is down -> 3 servers per zone -> 9 servers total (N+3 by zone)

Load balancing only helps if the remaining servers can absorb the redistributed traffic.

Reliability of the Load Balancer Itself

Approach Example
Managed, multi-zone service AWS ALB/NLB, Azure Load Balancer, Google Cloud LB
Active-passive pair with a floating IP keepalived (VRRP) with HAProxy or nginx
Active-active with ECMP or anycast BGP-announced VIPs to several LB nodes
Multiple layers Global LB in front of regional LBs

Pitfalls

  • Health checks that only test the port, so hung applications still receive traffic.

  • Deep health checks that fail every server when a shared dependency blips.

  • Retries without limits, which amplify overload; use retry budgets.

  • No spare capacity, so failover overloads the survivors.

  • A single load balancer instance.

Interview Tips

  • Explain redundancy + health checks + automatic failover.

  • Mention outlier detection, retries, draining, canaries and global failover.

  • Stress capacity for failure and redundancy of the load balancer itself.

7

How do load balancers perform health checks on backend servers?

Load balancers decide which back ends may receive traffic by checking their health, using two complementary methods: - Active health checks: the load balancer sends probes to each back end on a schedule (for example every 5–10 seconds). A probe can be a TCP connect, an HTTP(S) request to a path such as /healthz or /ready expecting a status code (200) or response content, a gRPC health check, or a custom script. A back end is marked unhealthy after N consecutive failures (unhealthy threshold) and healthy again after M consecutive successes (healthy threshold). Each probe has a timeout. - Passive health checks (outlier detection): the load balancer watches real traffic: connection failures, timeouts and 5xx rates. It temporarily ejects back ends that misbehave, even if the probes still pass.

Good health endpoints report whether this instance can serve requests: started, warmed up, not shutting down, and with critical local resources available. They should not fail because of a shared downstream dependency. Unhealthy back ends stop receiving new requests; in-flight connections are usually allowed to finish (draining).

Read full answer

Applies toHAProxy, nginx, Envoy, AWS ELB, Azure, Google Cloud, Kubernetes probes

Active Check Parameters

Parameter Typical value Effect
Protocol and target HTTP GET /ready on port 8080 What is tested
Interval 5–30 s Detection speed versus probe load
Timeout 2–5 s How long a probe may take
Unhealthy threshold 2–3 failures Avoids flapping on one slow probe
Healthy threshold 2–5 successes Avoids re-adding a still-broken instance
Expected response Status 200–399, or a body match What “healthy” means

Detection time ≈ interval × unhealthy threshold (for example 5 s × 3 = 15 s).

Examples

backend app
    option httpchk GET /ready
    http-check expect status 200
    default-server inter 5s fall 3 rise 2
    server app1 10.0.1.11:8080 check
    server app2 10.0.1.12:8080 check
# AWS ALB target group health check
aws elbv2 modify-target-group --target-group-arn "$TG_ARN" \
  --health-check-protocol HTTP --health-check-path /ready \
  --health-check-interval-seconds 10 --health-check-timeout-seconds 5 \
  --healthy-threshold-count 2 --unhealthy-threshold-count 3 --matcher HttpCode=200
# Kubernetes: readiness decides whether the pod is in Service endpoints
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /live
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

What the Health Endpoint Should Check

Check Include?
Process is up and the HTTP server responds Yes
App finished start-up and warm-up (caches, connections) Yes (readiness)
Instance is draining or shutting down Yes: return 503 so the LB stops sending traffic
Local resources (disk space, thread pool not exhausted) Often
Shared database or downstream services Usually no: a shared outage would mark every instance unhealthy at once
Expensive queries No: health checks run constantly

Passive Checks / Outlier Detection

Product Feature
nginx (open source) max_fails and fail_timeout on upstream servers
HAProxy observe layer7, error-limit, on-error mark-down
Envoy and Istio outlier_detection (consecutive 5xx, success-rate ejection)
AWS ALB Automatic target weights (anomaly mitigation)

Pitfalls

  • TCP-only checks on HTTP apps: a hung app still accepts connections.

  • Health checks touching shared dependencies, causing all-instance failures.

  • Thresholds too aggressive (flapping) or too lax (slow detection).

  • Health endpoints exposed publicly with sensitive details; restrict them or keep responses minimal.

  • Fail-open behaviour unknown: some load balancers send traffic to all targets when all are unhealthy (AWS ALB does). Know your platform’s behaviour.

Interview Tips

  • Explain active probes (interval, timeout, thresholds) and passive outlier detection.

  • Describe a good /ready endpoint, and why to avoid shared-dependency checks.

  • Mention draining, Kubernetes readiness versus liveness and fail-open behaviour.

8

What are the advantages and disadvantages of round-robin load balancing?

Round robin sends each new request or connection to the next server in a fixed rotation: A, B, C, A, B, C, and so on. Its advantages: - it is simple and predictable; - it is fast, with almost no state or computation; - it is fair when servers are identical and requests are similar and short; - it needs no feedback from the servers; - it is easy to reason about and debug; - weighted round robin handles servers of different sizes.

Its disadvantages come from ignoring current load: - it treats all requests as equal, so with variable request costs (a heavy report next to tiny API calls) some servers build queues while others are idle, raising tail latency; - it ignores server speed, health degradation or warm-up (a slow or new server still gets its full share); - it handles long-lived connections poorly (WebSockets, HTTP/2, gRPC), because balancing per connection creates lasting imbalance; - it provides no affinity; - several independent balancers can end up synchronised.

Read full answer

For variable workloads, least outstanding requests or power of two choices usually perform better.

Applies to: load balancing algorithm selection (Python 3.12 simulation)

Code Example: Round Robin Versus Least-Loaded With Uneven Requests

import random


def simulate(policy, n_requests=20_000, servers=4, seed=1):
    rng = random.Random(seed)
    busy_until = [0.0] * servers                 # when each server finishes its queued work
    waits = []
    t = 0.0
    turn = 0
    for _ in range(n_requests):
        t += rng.expovariate(2.6)                # arrivals: 2.6 per time unit (about 80% utilisation)
        # 90% cheap requests (0.5), 10% expensive ones (8.0): uneven work per request
        cost = 8.0 if rng.random() < 0.1 else 0.5
        if policy == "round_robin":
            s = turn % servers
            turn += 1
        else:                                    # least loaded (least outstanding work)
            s = min(range(servers), key=lambda i: busy_until[i])
        start = max(t, busy_until[s])
        busy_until[s] = start + cost
        waits.append(start - t)
    waits.sort()
    return sum(waits) / len(waits), waits[int(len(waits) * 0.99)]


for policy in ("round_robin", "least_loaded"):
    avg, p99 = simulate(policy)
    print(f"{policy:13s} avg wait {avg:5.2f}  p99 wait {p99:6.2f}")
# round_robin   avg wait 11.06  p99 wait  64.57
# least_loaded  avg wait  2.58  p99 wait  14.66

Both policies serve the same traffic at about 80% utilisation, where 10% of requests are 16 times more expensive. Round robin ignores the queues, so expensive requests pile up on whichever server is next. Its average wait is about 4 times higher and its p99 wait about 4 times higher than a load-aware policy.

Advantages and Disadvantages

Advantages Disadvantages
Very simple; minimal state Ignores current load and queue length
Low CPU overhead at very high rates Poor with variable request costs (higher tail latency)
Even distribution of request counts Even counts ≠ even work
Works well for homogeneous, stateless, short requests Poor with long-lived connections
Weighted variant supports different server sizes Static weights do not adapt to slowdowns
Predictable for testing and capacity math No session affinity (needs separate stickiness)

When Round Robin Is a Good Choice

  • Identical, stateless servers handling short, uniform requests (static content, simple API calls).

  • DNS round robin for coarse distribution (with health-checked records).

  • Canary splits with weights (95/5).

  • Very high request rates where per-request load tracking is too costly.

Better Alternatives When It Struggles

Problem Alternative
Variable request durations Least outstanding requests, power of two choices
Long-lived connections Least connections; L7 per-request balancing for HTTP/2 and gRPC
Heterogeneous or degrading servers Latency-aware (EWMA) balancing, outlier detection
Cache locality Consistent hashing
New instances cold Slow start or warm-up (HAProxy slowstart, ALB slow start mode)

Pitfalls

  • Assuming equal request counts mean equal load.

  • Round robin over HTTP/2 or gRPC connections at L4: one connection can carry most of the traffic.

  • No health checks, so round robin keeps sending traffic to failed servers.

Interview Tips

  • Describe round robin as simple, stateless and fair for uniform work.

  • Explain the main weakness, ignoring load with uneven requests, using the queueing intuition.

  • Suggest weighted, least-requests or P2C alternatives depending on the workload.

9

In load balancing, what is the significance of the least connections method?

Least connections sends each new connection or request to the back end with the fewest active connections at that moment. Its significance is that it is the simplest dynamic, load-aware algorithm: it uses live state instead of a fixed rotation, so it adapts automatically when connections have very different durations or servers process at different speeds. A server stuck with slow or long-lived work accumulates connections and stops receiving new ones until it catches up. This makes it a strong default for: - long-lived connections: WebSockets, database and message-broker connections, streaming, gRPC streams; - variable request times: reports next to fast API calls; - mixed server performance.

Weighted least connections divides active connections by server capacity. Close relatives are least outstanding requests (per HTTP request rather than per TCP connection, used for HTTP/2 and gRPC in Envoy and AWS ALB) and power of two choices, which samples two back ends and picks the one with fewer connections, scaling better across many load balancers. Limitations: it counts connections, not actual work; it needs state shared across balancer instances, otherwise each sees only its own connections; and it can flood a newly added or recovered server, which has zero connections, unless slow start is used.

Read full answer

Applies tonginx `least_conn`, HAProxy `leastconn`, Envoy `LEAST_REQUEST`, IPVS `lc`/`wlc` (Python 3.12 simulation)

Code Example: Long-Lived Connections

import heapq
import random


def run(policy, servers=4, n=5_000, seed=3):
    rng = random.Random(seed)
    active = [0] * servers
    ends = []                                    # heap of (end_time, server)
    peak = [0] * servers
    t, turn = 0.0, 0
    for _ in range(n):
        t += rng.expovariate(1.0)
        while ends and ends[0][0] <= t:          # close finished connections
            _, s = heapq.heappop(ends)
            active[s] -= 1
        # Mixed connection lengths: most short API calls, some long-lived WebSocket sessions
        duration = rng.expovariate(1 / 300) if rng.random() < 0.05 else rng.expovariate(1 / 2)
        if policy == "round_robin":
            s = turn % servers
            turn += 1
        else:
            s = min(range(servers), key=lambda i: active[i])
        active[s] += 1
        peak[s] = max(peak[s], active[s])
        heapq.heappush(ends, (t + duration, s))
    return peak


print("round robin peak connections:      ", run("round_robin"))   # [8, 11, 9, 10]
print("least connections peak connections:", run("least_conn"))    # [7, 7, 7, 6]

With 5% long-lived sessions mixed into short calls, round robin piles long sessions onto some servers by chance. Least connections keeps the peak connection counts nearly equal.

Configuration

upstream ws_backend {
    least_conn;
    server 10.0.1.11:8080 weight=2;     # weighted least connections: bigger server
    server 10.0.1.12:8080;
    server 10.0.1.13:8080;
}
backend websockets
    balance leastconn
    timeout tunnel 1h                   # keep idle WebSocket tunnels open
    server ws1 10.0.1.11:8080 check slowstart 60s
    server ws2 10.0.1.12:8080 check slowstart 60s

slowstart ramps a recovered server’s weight up over 60 seconds, so it is not flooded the moment it returns with zero connections.

Least Connections Versus Alternatives

Algorithm Unit Strength Weakness
Round robin Request or connection Simple; good for uniform short requests Ignores load
Least connections Active connections Adapts to long or uneven connections Counts, not work; new-server flood
Weighted least connections Connections ÷ weight Mixed server sizes Static weights
Least outstanding requests In-flight HTTP requests Correct for HTTP/2 and gRPC multiplexing Needs L7
Power of two choices Two random samples Works with many independent balancers Slightly less precise
Least response time / EWMA Latency Accounts for slow servers Can herd and oscillate

When to Use It

Workload Least connections?
WebSockets, SSE, long polling Yes
Database or broker proxies (PgBouncer, ProxySQL in front of replicas) Yes
Mixed long and short HTTP requests Yes (or least outstanding requests)
Uniform, very short requests at huge rates Round robin is simpler and similar
Cache servers No: use consistent hashing for locality

Pitfalls

  • Many balancer instances each counting only their own connections: the global view is lost (P2C handles this better).

  • HTTP/2 or gRPC at L4: one connection carries many requests, so connection counts mislead.

  • No slow start: a recovered or newly scaled server receives a burst of connections.

  • Ignoring weights when servers differ in size.

Interview Tips

  • Define it as “send to the server with the fewest active connections”: a dynamic algorithm.

  • Explain why it suits long-lived and uneven connections.

  • Mention weighted variants, least outstanding requests, P2C and slow start.

10

Explain how a load balancer handles the failure of one of its servers.

When a back-end server fails, a load balancer typically goes through this sequence: 1. Detect: active health checks (missed probes beyond the unhealthy threshold) or passive detection (connection refused, timeouts, a burst of 5xx errors) mark the server down. With outlier detection this takes seconds. 2. Stop sending new traffic: the server is removed from the rotation, and the algorithm only chooses among healthy servers. With session persistence, affected sessions are reassigned to other servers; their in-memory state is lost unless it is stored externally. 3. Handle in-flight requests: requests already sent to the failed server fail or time out. An L7 load balancer can retry idempotent requests on another server (limited by retry budgets), so users often see nothing. 4. Protect the rest: the other servers absorb the load. Connection limits, queues and autoscaling prevent a cascade. If too many servers fail, some balancers fail open (send traffic to all servers) to avoid a total outage, or serve a fallback or maintenance response. 5. Recover: once health checks pass the healthy threshold, the server is gradually reintroduced, ideally with slow start. Orchestrators or autoscaling groups replace dead instances automatically. 6. Alert and record: metrics and logs show the event for investigation.

Read full answer

Applies toHAProxy, nginx, Envoy, cloud load balancers, Kubernetes Services

Timeline Example

t=0s    app2 crashes
t=0-1s  in-flight requests to app2 fail -> LB retries GETs on app1/app3 (POSTs return 502)
t=5s    probe 1 fails; t=10s probe 2 fails; t=15s probe 3 fails -> app2 marked DOWN
        (passive detection may eject it after the first few connection errors, sooner)
t=15s+  all new traffic goes to app1/app3; autoscaling group launches app4
t=90s   app4 passes 2 health checks -> added with slow start

Configuration Examples

upstream app {
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;   # passive: 3 failures -> skip for 10 s
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.13:8080 backup;                         # used only when the others are down
}
server {
    location / {
        proxy_pass http://app;
        proxy_next_upstream error timeout http_502 http_503;   # retry on another server
        proxy_next_upstream_tries 2;
        proxy_connect_timeout 1s;
    }
}

By default nginx retries only idempotent methods with proxy_next_upstream; non_idempotent must be added explicitly to retry POSTs, which is usually unsafe.

backend app
    option httpchk GET /ready
    option redispatch                  # move persistent sessions to a healthy server
    retries 2
    default-server inter 3s fall 3 rise 2 slowstart 30s
    server app1 10.0.1.11:8080 check
    server app2 10.0.1.12:8080 check

Handling Different Failure Types

Failure Detection Handling
Crash or host down Connection refused, failed probes Remove; retry elsewhere
Hung process (accepts, never replies) Timeouts; HTTP checks (TCP checks miss it) Remove; set short connect and read timeouts
Slow or degraded Latency-aware outlier detection Eject temporarily or reduce weight
Partial errors (some 500s) Passive 5xx tracking Eject when over threshold
Whole zone fails Many servers fail at once Cross-zone pools; global LB fails over
All back ends fail All checks fail Fail open, or serve a static error or maintenance page; alert

Pitfalls

  • Retrying non-idempotent requests, causing duplicate orders or payments.

  • Slow detection (long intervals, high thresholds), or flapping with thresholds too low.

  • No spare capacity, so the remaining servers overload.

  • Session loss with stickiness and in-memory state.

  • Re-adding recovered servers at full weight immediately.

Interview Tips

  • Walk through detect → remove → retry → absorb → recover → alert.

  • Distinguish active and passive detection and idempotent retries.

  • Mention slow start, fail-open behaviour, capacity headroom and automatic replacement.

11

How does a load balancer distribute traffic in a stateless vs stateful scenario?

In a stateless scenario, each request carries everything needed to process it (an auth token, parameters), and any shared state lives in external stores (databases, Redis, object storage). The load balancer can therefore send every request to any healthy server using a pure load-oriented algorithm: round robin, least requests or power of two choices. This gives the best balance, simplest failover (a failed server loses nothing), easy autoscaling and zero-downtime deployments. In a stateful scenario, a server keeps per-client or per-key state locally: in-memory sessions, a WebSocket connection, a game room, a cached shard, a partition of data. Requests for that client or key must reach the server that holds the state. The load balancer then routes by affinity: - sticky sessions via cookies or source IP for user sessions; - consistent hashing on a key (user ID, room ID, cache key) for sharded state; - connection-level routing for long-lived connections, where all messages on one connection stay on its server.

Stateful routing trades away even load and easy failover for locality. Designs aim to make app servers stateless and keep state in purpose-built stateful systems with their own replication.

Read full answer

Applies toapplication and load balancer architecture

Comparison

Aspect Stateless Stateful
Routing Any healthy server Same server per session or key
Algorithms Round robin, least requests, P2C, random Cookie stickiness, IP hash, consistent hashing, header-based routing
Load balance Even Can be skewed (hot users or keys)
Server failure Transparent (retry elsewhere) Sessions or shards lost or must be rebuilt; clients reconnect
Scaling out or in Instant Requires rebalancing or draining; keys move
Deployments Rolling, easy Need draining, state migration or reconnect handling
Examples REST APIs with JWTs, static content WebSocket chat, multiplayer games, in-memory sessions, cache clusters

Example Configurations

# Stateless API: any server
upstream api {
    least_conn;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

# Stateful by key: game rooms, consistent hashing on the room ID in the URL
upstream rooms {
    hash $arg_room consistent;
    server 10.0.2.11:9000;
    server 10.0.2.12:9000;
    server 10.0.2.13:9000;
}

hash ... consistent uses a ketama-style ring, so adding a server moves only about 1/N of the rooms.

Making Stateful Workloads Easier to Balance

Technique Effect
Externalise sessions (Redis, database) Turns web tiers stateless
Signed tokens (JWT, signed cookies) Clients carry state securely
Pub/sub for real-time fan-out (Redis, NATS, Kafka) WebSocket servers become interchangeable
Consistent hashing with virtual nodes Small, predictable movement on scaling
Replicated state (Raft, primary-replica) Failover without losing data
Graceful draining and reconnection logic Clients move smoothly during deployments

Kubernetes Note

Deployments behind a Service suit stateless pods: any ready pod receives traffic. StatefulSets give pods stable names (db-0, db-1), and clients or headless Services address specific replicas. Stateful systems (databases, Kafka) usually handle their own routing (leaders, partitions) rather than relying on a generic load balancer.

Pitfalls

  • Accidentally stateful apps (in-memory sessions or file uploads on local disk) behind round-robin balancers, causing random logouts or errors.

  • IP-hash stickiness with NAT or mobile clients.

  • Hot keys overloading one server under consistent hashing.

  • Scaling stateful tiers like stateless ones without moving their state.

Interview Tips

  • Stateless: any server, load-based algorithms, easy failover. Stateful: affinity, sticky sessions or consistent hashing.

  • Discuss trade-offs (balance, failover, scaling).

  • Recommend externalising state, and consistent hashing where locality is needed.

12

What is the concept of session persistence, and why is it important?

Session persistence (session affinity, “sticky sessions”) is the load balancer’s ability to keep sending a client’s requests to the same back-end server for the duration of a session. The client is identified by a cookie (inserted by the load balancer or set by the application), the source IP, or a header or URL token. It is important when correctness or performance depends on server-local state: - in-memory user sessions (login state, shopping carts) in applications that do not share session storage; - multi-step workflows that keep temporary data on one server (wizards, chunked uploads); - long-lived or stateful protocols (WebSockets, SignalR and Socket.IO fallbacks, some SSO and SAML flows); - performance: better local cache hit rates, warm per-user data, fewer repeated authentications; - legacy applications that were never designed for horizontal scaling.

Without persistence, such applications show random logouts, lost carts and failed uploads. Persistence is not a guarantee of state survival: if the server fails, its sessions are gone. It also skews load and complicates scaling and deployments. Modern designs therefore treat persistence as an optimisation and keep important session state in shared stores or tokens.

Read full answer

Applies toL7 and L4 load balancers, Kubernetes Ingress and Service affinity

Persistence Methods

Method How the client is recognised Notes
LB-inserted cookie Set-Cookie: SERVERID=app2 (or AWSALB, ApplicationGatewayAffinity) Most precise; needs HTTP and cookies
Application cookie Existing session cookie such as JSESSIONID The LB learns the mapping from the app’s cookie
Source IP affinity Client IP (hash or table) Works at L4; poor behind NAT, proxies or mobile networks
Header or URL parameter X-Session-ID, tenant ID, query parameter Good for APIs and non-browser clients
TLS session ID TLS session Rarely used today

Example: Kubernetes Ingress (NGINX Ingress Controller) Cookie Affinity

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "3600"
spec:
  ingressClassName: nginx
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: shop
                port:
                  number: 80

At L4, a Kubernetes Service can use sessionAffinity: ClientIP with a timeout.

Why It Matters, and the Costs

Benefit Cost
Correctness for apps with local session state Uneven load: heavy users pinned to one server
Better cache locality and performance Session loss when that server fails or is scaled in
Required by some protocols and legacy apps Slower draining and deployments
Simple to enable Masks architectural debt (stateful web tiers)

Best Practices

  • Keep stickiness duration short, and set it in line with application session timeouts.

  • Drain servers before removing them, so sticky users finish their work.

  • Allow redispatch when the pinned server is down (HAProxy option redispatch).

  • Prefer externalised sessions (Redis, a database) or signed tokens, using stickiness only for performance.

  • Monitor per-server load to spot imbalance caused by affinity.

Pitfalls

  • Treating stickiness as reliability: it is the opposite; failures lose sessions.

  • IP affinity for users behind CGNAT or corporate proxies, which pins thousands of users to one server.

  • Cookie affinity with CDNs or caches that strip or share cookies.

  • Very long affinity that keeps users on old versions after deployments.

Interview Tips

  • Define it as keeping a client’s requests on the same server.

  • Explain why: local session state, protocols, cache locality.

  • Cover methods (cookie, IP, header), costs, and the preferred stateless design with shared session stores.

13

Discuss the role of DNS in load balancing.

DNS is often the first layer of load balancing: it decides which IP address, and therefore which data centre, region, load balancer or CDN edge, a client connects to. Techniques: - DNS round robin: several A or AAAA records for one name, rotated in responses. - Weighted records: traffic split by proportion (canaries, migrations). - Geo-DNS / geolocation routing: answers based on the resolver’s or client’s location. - Latency-based routing: the region with the lowest measured latency. - Health-checked failover: remove records for unhealthy endpoints; primary/secondary failover. - Global server load balancing (GSLB): combines these with health and capacity data.

Managed DNS services provide them: Amazon Route 53, Azure Traffic Manager, Google Cloud DNS routing policies, Cloudflare, NS1.

Read full answer

DNS load balancing is cheap, global and scalable, but coarse and slow to react: - caching and TTLs mean clients keep old answers (often longer than the TTL, because some resolvers and apps ignore it); - DNS sees the resolver’s location, not the user’s (partly fixed by EDNS Client Subnet); - it knows nothing about per-request load; - clients may pick addresses in different orders.

In practice DNS distributes traffic between regions or sites, and regional load balancers (L4/L7) handle fine-grained, per-request balancing. Anycast is the complementary network-level technique.

Applies to: DNS-based traffic management (Route 53, Azure Traffic Manager, Cloudflare, NS1)

DNS Techniques

Technique How Use
Round robin Several A/AAAA records Simple spreading across equal endpoints
Weighted Records with weights (for example 90/10) Canary, blue-green, migrations between providers
Geolocation Answer by client or resolver region Data residency, regional content
Latency-based Answer with the lowest-latency region Global performance
Failover Health checks; primary/secondary Disaster recovery
Multi-value answer Several healthy records, randomised Basic health-aware spreading

Example: Route 53 Records

api.example.com  A  ALIAS  alb-eu-west-1   routing: latency (eu-west-1), health check: /healthz
api.example.com  A  ALIAS  alb-us-east-1   routing: latency (us-east-1), health check: /healthz
api.example.com  A  ALIAS  alb-ap-south-1  routing: latency (ap-south-1), health check: /healthz
TTL: 60 seconds

A client in Paris resolves to the eu-west-1 load balancer. If its health check fails, Route 53 answers with the next-best healthy region.

dig +short api.example.com             # see which addresses this resolver returns
dig api.example.com +ttlunits           # remaining TTL on the cached answer

Strengths and Limitations

Strengths Limitations
Global reach; no extra network hop Caching: changes take TTL + resolver/client caching to apply
Very scalable and cheap Coarse: per resolver, not per request or per user
Works across clouds and data centres No awareness of server load (unless integrated with GSLB data)
Simple failover between regions Clients may cache failed addresses; some retry poorly
Enables geo and latency steering Resolver location ≠ user location (EDNS Client Subnet helps)

DNS Combined With Other Layers

DNS / GSLB (region choice, failover) -> anycast edge or CDN -> regional L7 load balancer
-> Kubernetes Service / service mesh -> pods

Best Practices

  • Use short TTLs (30–60 s) for records that may fail over, balanced against query cost.

  • Point DNS at highly available load balancers, not at individual servers.

  • Use health checks that reflect real service health in each region.

  • Plan for clients that ignore TTLs: keep old endpoints serving or redirecting during migrations.

  • Protect DNS: DNSSEC, provider redundancy, registrar lock.

Pitfalls

  • DNS round robin to individual servers without health checks: failed servers keep receiving traffic.

  • Very long TTLs that make failover take hours.

  • Treating DNS as precise load balancing.

  • A single DNS provider as a single point of failure.

Interview Tips

  • Explain DNS as global, coarse-grained balancing (round robin, weighted, geo, latency, failover).

  • Discuss TTL and caching limits and resolver location.

  • Show the layered design: DNS for regions, L4/L7 load balancers within regions.

14

In what scenarios would you use weighted load balancing?

Weighted load balancing assigns each back end a weight, and traffic is split in proportion to the weights: weights 3:1 send about 75% and 25%. It is used when back ends should not receive equal shares: - heterogeneous capacity: servers or instance types of different sizes (8 cores versus 2), or mixed hardware generations; - canary releases and progressive delivery: send 1–5% of traffic to a new version, then ramp up (10%, 25%, 50%, 100%) while watching metrics; - blue-green cut-overs and migrations: shift traffic gradually between environments, clusters, clouds or data centres, including DNS-weighted moves between providers; - A/B testing: fixed percentages to experiment variants (often combined with stickiness so users stay in their variant); - warming up new or recovered instances (slow start ramps the weight up); - draining or de-prioritising back ends: lower weights for servers being phased out, under maintenance or in a costlier region; - cost or licence balancing: favour cheaper capacity (reserved instances, on-premises) and overflow to more expensive capacity.

Weighted versions exist for most algorithms: weighted round robin, weighted least connections, weighted DNS records, and weighted target groups or routes (AWS ALB, Istio, Gateway API, Argo Rollouts).

Read full answer

Applies tonginx, HAProxy, Envoy/Istio, Gateway API, AWS ALB and Route 53

Examples

# Heterogeneous servers
upstream app {
    server big1.internal:8080   weight=4;    # 8 vCPU
    server big2.internal:8080   weight=4;
    server small1.internal:8080 weight=1;    # 2 vCPU
}
# Canary with the Kubernetes Gateway API: 95% stable, 5% canary
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkout
spec:
  parentRefs:
    - name: public-gateway
  rules:
    - backendRefs:
        - name: checkout-stable
          port: 80
          weight: 95
        - name: checkout-canary
          port: 80
          weight: 5
# AWS ALB: weighted forward between two target groups (90/10)
aws elbv2 modify-listener --listener-arn "$LISTENER" --default-actions '[{"Type":"forward",
  "ForwardConfig":{"TargetGroups":[{"TargetGroupArn":"'"$BLUE"'","Weight":90},
                                   {"TargetGroupArn":"'"$GREEN"'","Weight":10}]}}]'

Scenario Guide

Scenario How weights are used
Different server sizes Weights proportional to capacity (cores, benchmarks)
Canary deployment 1–5% to the new version; automated promotion (Argo Rollouts, Flagger)
Blue-green migration 0 → 10 → 50 → 100% to green; keep blue for rollback
Cloud or data-centre migration Weighted DNS between old and new sites
A/B experiments Fixed split plus sticky assignment
New or recovered instance Slow start: weight rises over a warm-up period
Maintenance Weight 0 (drain) without removing the server
Cost optimisation Higher weight to cheaper capacity; overflow elsewhere

Choosing Weights

  • Base capacity weights on measured throughput, not just CPU counts.

  • Use small canary percentages together with error-rate and latency gates.

  • Remember stickiness interacts with weights: sticky sessions keep users where they started, so shifts take effect gradually.

  • With DNS weights, caching delays changes. With few requests, actual splits deviate from the weights in the short term.

Pitfalls

  • Static weights that no longer match reality after instance changes.

  • Weights on the wrong layer (DNS weights expected to give precise per-request splits).

  • Canaries without automated analysis, so bad releases ramp up anyway.

  • Forgetting dependencies: a canary at 5% may still write data or schema changes that affect 100% of users.

Interview Tips

  • Define it as traffic proportional to weights.

  • Give scenarios: heterogeneous servers, canaries, migrations, A/B tests, slow start, draining.

  • Show examples (nginx weights, Gateway API, ALB weighted target groups), and mention monitoring and stickiness interactions.

15

How can load balancers help mitigate DDoS attacks?

Load balancers help against distributed denial-of-service (DDoS) attacks in several ways: - Absorbing and spreading volume: large managed and anycast load balancers (cloud LBs, CDN edges) have huge capacity spread across many locations, so attack traffic is diluted. Back-end servers are hidden behind them. - Filtering at the edge: dropping malformed and non-application traffic. An L7 load balancer accepts only valid HTTP(S) on allowed ports, and its TCP stack absorbs SYN floods (SYN cookies, managed connection tracking) before anything reaches the servers. - Rate limiting and connection limits per IP, per client, per path or globally, plus request size limits and timeouts against slow attacks such as Slowloris and slow POSTs. - Integration with WAF and bot management: blocking known bad IPs, countries, signatures and HTTP floods; challenges such as CAPTCHAs or JavaScript checks. - Autoscaling the back ends behind the balancer, with caching offloading repeated requests. - Health-based isolation and graceful degradation: shed low-priority traffic and keep critical paths alive.

A load balancer alone cannot stop very large volumetric attacks. Those need upstream scrubbing: cloud DDoS protection such as AWS Shield, Azure DDoS Protection, Google Cloud Armor and Cloudflare, or ISP and scrubbing services. In practice the load balancer is one layer of a DDoS defence.

Read full answer

Applies tocloud and self-managed load balancers, CDNs, WAFs

Attack Types and Defences

Attack Layer Load balancer / edge defence
UDP or ICMP floods, amplification (DNS, NTP, memcached) L3/4 Only listen on needed ports; anycast absorption; upstream scrubbing
SYN floods L4 SYN cookies, managed TCP termination, connection rate limits
TLS handshake floods L5/6 TLS termination at scale, session resumption, connection limits
HTTP floods (many valid-looking requests) L7 Rate limiting, WAF rules, bot detection, caching, challenges
Slowloris and slow POST L7 Header and body timeouts, minimum data rates, connection limits per IP
Expensive endpoints (search, login) L7 Path-specific rate limits, CAPTCHAs, caching, queueing

Example: nginx Limits

limit_req_zone  $binary_remote_addr zone=perip:10m rate=20r/s;
limit_conn_zone $binary_remote_addr zone=conns:10m;

server {
    listen 443 ssl;
    client_header_timeout 10s;          # defeats slow-header (Slowloris) attacks
    client_body_timeout   10s;
    client_max_body_size  2m;
    limit_conn conns 50;                # max concurrent connections per IP

    location /api/search {
        limit_req zone=perip burst=40 nodelay;
        limit_req_status 429;
        proxy_pass http://app;
    }
}

$binary_remote_addr must be the real client IP. Behind a CDN, configure real_ip_header and trusted proxy ranges, otherwise you rate-limit the CDN.

Layered Architecture

attackers -> anycast CDN / DDoS protection (Cloudflare, Shield Advanced, Cloud Armor)
              absorbs volumetric attacks, bot management, WAF
          -> cloud L7 load balancer (only 443 open; TLS; rate rules)
          -> autoscaled app servers (not publicly reachable: security groups allow only the LB)
          -> caches and databases (protected by app-level limits and queues)

Practices

  • Hide origins: back ends accept traffic only from the load balancer or CDN (security groups, private subnets, mTLS from the CDN).

  • Cache aggressively for public content, so attacks hit the cache rather than the application.

  • Prepare runbooks: who to call, how to enable stricter modes (“under attack” modes, geo blocks).

  • Monitor request rates, error rates and connection counts per source.

  • Load-test limits and autoscaling ahead of time.

Pitfalls

  • Origin IPs exposed (old DNS records, email headers), so attackers bypass the CDN and load balancer.

  • Rate limiting by the wrong IP behind proxies.

  • Autoscaling without cost caps, so an attack becomes a large bill (“economic DoS”).

  • Expecting a self-managed load balancer to survive multi-terabit attacks.

Interview Tips

  • Explain absorption, filtering, rate limiting, WAF integration and autoscaling.

  • Distinguish volumetric (L3/4) from application (L7) attacks, and their defences.

  • Stress upstream DDoS services, hidden origins and layered defence.

16

What is horizontal versus vertical scaling, and how does load balancing apply?

Vertical scaling (scaling up) means giving one machine more resources: more CPU, RAM, faster disks or a bigger instance type. It needs no application changes and suits systems that are hard to distribute, such as many relational databases. But it has a hard ceiling (the biggest machine), often requires downtime or a restart to resize, gets expensive at the top end, and leaves a single point of failure. Horizontal scaling (scaling out) means adding more machines or instances and spreading the work across them. It offers near-linear capacity growth, fault tolerance through redundancy, and elasticity (add or remove instances automatically). It requires stateless or partitioned application design.

Load balancing is what makes horizontal scaling work: it gives the pool one entry point, distributes requests across instances, adds new instances as they become healthy (autoscaling integration) and removes failed or retired ones. With vertical scaling there is only one server, so a load balancer adds little for distribution. It can still help with failover to a standby, TLS offload and zero-downtime resizing (bring up a bigger instance, shift traffic, retire the old one). Most systems combine both: scale stateless tiers out behind load balancers, and scale stateful tiers up, adding replicas or shards when needed.

Read full answer

Applies tocapacity planning and architecture (vendor-neutral)

Comparison

Aspect Vertical (scale up) Horizontal (scale out)
How Bigger machine More machines
Application changes Usually none Statelessness, shared state, partitioning
Limit Largest available machine Very high (coordination costs eventually)
Fault tolerance Single point of failure Redundant; survives node loss
Elasticity Resizing often needs a restart Add or remove instances automatically
Cost curve Rises steeply at the top end Roughly linear; commodity instances
Load balancer role Minor (failover to a standby, TLS offload) Essential: distribution, health checks, scaling integration
Typical tiers Primary databases, legacy monoliths, in-memory analytics Web and API servers, workers, caches (sharded), stateless microservices

How Load Balancing Supports Horizontal Scaling

+--> app-1
client -> LB ----+--> app-2        autoscaling adds app-4 when CPU > 60%:
                 +--> app-3          1. instance boots, passes health checks
                                     2. LB adds it to the pool (with slow start)
                                     3. traffic redistributes automatically
                 scale-in: LB drains app-4's connections, then it terminates

Example: Kubernetes Horizontal Pod Autoscaler Behind a Service

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

The Service (and Ingress or Gateway in front) automatically includes every ready pod, so new replicas start receiving traffic as soon as their readiness probes pass.

Scaling Stateful Tiers

Tier Vertical Horizontal
Relational database Bigger instance Read replicas (read-traffic load balancing), sharding, distributed SQL
Cache Larger memory Cluster with consistent hashing (Redis Cluster)
Message broker Bigger broker More partitions and brokers
Sessions Single in-memory store Externalised, replicated store

Choosing

  • Start simple: scaling up a database is often cheapest early on.

  • Design web and API tiers to be stateless from the start, so they can scale out.

  • Plan for failure: even when vertical scaling suffices for load, have at least two instances behind a load balancer for availability.

Pitfalls

  • Scaling out stateful apps without externalising state (sticky-session dependence).

  • Bottlenecks that move: scaling out the web tier until the database saturates.

  • Ignoring the load balancer’s own capacity (connection limits, TLS CPU).

  • Vertical scaling without a failover plan.

Interview Tips

  • Define both, with limits, cost and fault tolerance.

  • Explain that load balancing is the enabler of horizontal scaling (distribution, health checks, autoscaling).

  • Mention hybrid strategies: scale out stateless tiers, scale up or shard stateful ones.

17

What factors should you consider when choosing a load balancing method for an application?

Choose a load balancing method by matching it to how the application behaves and what the business needs. The main factors: - Request profile: short, uniform requests suit round robin; variable or long-lived ones (WebSockets, streaming, slow reports) suit least connections or least outstanding requests. - State and affinity: stateless services can go anywhere. Server-local state needs sticky sessions; sharded data or caches need consistent hashing. - Server capacity: identical servers or mixed sizes, which calls for weights. - Protocol and layer: L4 (TCP/UDP: fast, content-blind) or L7 (HTTP routing by path, header or cookie; TLS termination). For HTTP/2 and gRPC, balance per request, not per connection. - Scale and topology: one balancer or many independent instances (where power of two choices and hashing work better than global counters), and multiple regions (DNS/GSLB, anycast). - Performance and latency goals: tail latency, throughput, and whether to add latency-aware methods (EWMA, outlier detection). - Availability: health checks, failover, slow start and draining support. - Deployment strategy: canaries and blue-green need weighted routing. - Security: TLS termination or passthrough, WAF, rate limiting, DDoS protection. - Operations and cost: managed or self-hosted, observability, team skills, licence and traffic costs.

The best choice is usually the simplest method that meets these needs, validated by load testing and production metrics.

Read full answer

Applies tosystem design and load balancer selection (vendor-neutral)

Decision Guide

Situation Suggested method
Stateless API, uniform short requests Round robin or random
Variable or long-lived connections Least connections / least outstanding requests
Different server sizes Weighted round robin or weighted least connections
Many balancer instances (sidecars, client-side LB) Power of two choices (P2C)
Caches, sharded state, per-room or per-tenant servers Consistent hashing on a key
Legacy app with in-memory sessions Sticky sessions (cookie), ideally temporary
Canary or migration Weighted routing
Users worldwide GSLB / latency-based DNS or anycast, then local L7
Servers with fluctuating speed Least response time / EWMA with outlier detection

Questions to Ask

1. Are servers interchangeable for every request?       no  -> hashing or stickiness
2. Do request durations vary a lot?                     yes -> least connections / requests
3. Are servers equal in capacity?                       no  -> weights
4. Is traffic HTTP/2, gRPC or WebSockets?               yes -> L7, per-request or connection-aware
5. How many balancers make decisions independently?     many -> P2C, avoid global counters
6. Do we need canaries or blue-green?                   yes -> weighted routes
7. Where are users?                                     global -> DNS/anycast + regional LBs

Factors and Trade-offs

Factor Trade-off to consider
Simplicity Round robin is predictable but load-blind
Load awareness Dynamic methods need state and accurate signals
Affinity Locality and correctness versus even load and failover
Layer L7 features versus L4 speed and protocol neutrality
Managed service Less operations work versus less control and cost per LCU or GB
Hashing Stable mapping versus hot keys

Validating the Choice

  • Load-test with realistic traffic (mix of request types, connection lengths, burstiness).

  • Watch per-server utilisation spread, p99 latency, error rates and retry counts.

  • Test failure scenarios: a server dying, a slow server, scaling events.

  • Revisit as traffic patterns change.

Pitfalls

  • Defaulting to round robin for long-lived connections.

  • Sticky sessions chosen as a quick fix that then blocks scaling.

  • Connection-based balancing for HTTP/2 and gRPC, which pins many requests to one server.

  • Over-engineering with complex latency-based algorithms that oscillate.

Interview Tips

  • List request profile, state, capacity, protocol, scale, availability, deployment, security and cost.

  • Map situations to methods with clear reasoning.

  • Emphasise validation with metrics and load tests.

18

Discuss how global server load balancing differs from local server load balancing.

Local server load balancing (LSLB) distributes traffic among servers within one site: a data centre, cloud region or cluster. It works at L4 or L7, makes a decision per connection or per request, sees real-time server health and load, and reacts within seconds. Examples: nginx, HAProxy, F5 BIG-IP LTM, AWS ALB/NLB, Kubernetes Services. Global server load balancing (GSLB) distributes traffic across multiple sites or regions. It decides which site a user should reach, usually via DNS (geo, latency, weighted and failover answers) or anycast (the same IP advertised from many locations, with BGP routing users to the nearest one). Its goals are proximity and low latency, disaster recovery when a whole site fails, data residency and capacity spreading between regions. GSLB works at coarse granularity (per DNS lookup or network route), and DNS-based GSLB is limited by caching and TTLs. The two are complementary: GSLB picks the region or site, then LSLB inside it picks the server. Examples of GSLB: Route 53 routing policies, Azure Traffic Manager and Front Door, Google Cloud’s global external load balancer, Cloudflare Load Balancing, F5 BIG-IP DNS (GTM).

Read full answer

Applies tomulti-region and multi-data-centre architectures

Comparison

Aspect Local (LSLB) Global (GSLB)
Scope Servers within one site or region Sites, regions or data centres
Mechanism Reverse proxy or L4 forwarding DNS answers, anycast, or a global edge proxy
Decision unit Per connection or request Per DNS resolution, per route, or per edge request
Inputs Server health, connections, latency Site health, user location, region latency, capacity, policy
Reaction time Seconds DNS: TTL plus resolver caching (minutes); anycast or edge proxy: seconds
Goal Spread load, server failover Proximity, regional failover, residency, DR
Examples nginx, HAProxy, ALB, NLB, F5 LTM Route 53, Traffic Manager, Front Door, Cloudflare LB, F5 GTM

Layered Architecture

user in Mumbai
   |
   v  GSLB: DNS latency routing -> ap-south-1 (or anycast edge nearest Mumbai)
regional LB (ALB in ap-south-1)
   |
   v  LSLB: least outstanding requests across healthy targets
app servers / pods in ap-south-1

If ap-south-1 fails its health checks, GSLB answers with the next-best region (for example eu-west-1). Local balancers there absorb the extra traffic.

GSLB Techniques

Technique Notes
Geo-DNS Region by resolver or client subnet (EDNS Client Subnet)
Latency-based DNS Uses measured latency between networks and regions
Weighted DNS Gradual migration between regions or providers
Failover DNS Active-passive DR
Anycast Same IP from many places; BGP picks the route; fast failover
Global proxy (edge) One anycast IP terminates at the edge, then forwards to the best region (Google global LB, Front Door, Cloudflare)

Design Considerations for GSLB

  • Data: can every region serve every user? Multi-region data replication, conflict handling and residency rules decide this.

  • Session continuity: users moved between regions may lose sessions unless state is global.

  • Capacity: each region needs headroom to absorb a failed region’s load.

  • Health checks should reflect full regional health (dependencies, not just the load balancer).

  • TTL choices: short enough for failover, aware that some clients ignore them.

Pitfalls

  • Assuming GSLB failover is instant with DNS.

  • Active-active regions without replicated data, giving inconsistent user experiences.

  • Health checks that only test the regional LB, missing a broken database.

  • No capacity planning for failover, so the surviving region collapses.

Interview Tips

  • LSLB: servers within a site, fine-grained, real-time. GSLB: sites and regions, DNS or anycast, proximity and DR.

  • Show the layered model and name example products.

  • Discuss TTL limits, data replication and capacity headroom.

19

What metrics would you monitor to assess the performance of a load balancer?

Monitor a load balancer from four angles: - Traffic: request rate (RPS), new and active connections, throughput (bytes in and out), TLS handshakes per second. - Latency: total request latency (p50, p95, p99), split into time spent in the load balancer and back-end response time, plus connect time to back ends and TLS handshake time. - Errors: 5xx split by origin: generated by the LB itself (502 bad gateway, 503 no healthy back ends, 504 timeout) versus returned by back ends. Also 4xx rates, connection errors, resets, rejected connections, TLS failures and retry counts. - Saturation and health: LB CPU and memory (self-managed), connection table usage, queue lengths (requests waiting for a back-end slot), port exhaustion, healthy versus unhealthy back ends, health-check failures, ejections, and capacity units for managed LBs.

Also watch distribution quality: requests and connections per back end to detect imbalance from stickiness, hashing or weights. These map to the four golden signals: latency, traffic, errors and saturation. Alert on user-facing symptoms (error rate and p99 latency against SLOs), and use the other metrics for diagnosis.

Read full answer

Applies tonginx, HAProxy, Envoy, AWS ELB, Azure and GCP load balancers

Key Metrics

Category Metric Why it matters
Traffic Requests/s, active and new connections Load and growth trends
Traffic Bytes processed Bandwidth and cost
Latency p50/p95/p99 total latency User experience
Latency Back-end response time vs LB processing time Locates the slowness
Errors LB-generated 502/503/504 Back-end failures, timeouts, empty pools
Errors Back-end 5xx Application bugs or overload
Errors Retries, connection resets Hidden instability
Saturation Queue length, max connections reached Overload ahead of errors
Saturation CPU, memory, file descriptors, ephemeral ports LB’s own limits
Health Healthy host count, ejections, health-check failures Capacity and stability
Balance Requests per back end, standard deviation Uneven distribution
Security Rate-limited or WAF-blocked requests Attacks and abuse

Examples by Platform

Platform Source Example metrics
AWS ALB CloudWatch RequestCount, TargetResponseTime, HTTPCode_ELB_5XX_Count, HTTPCode_Target_5XX_Count, HealthyHostCount, RejectedConnectionCount
HAProxy Stats page, Prometheus exporter haproxy_backend_current_queue, haproxy_server_response_time_average_seconds, haproxy_backend_http_responses_total{code="5xx"}
Envoy Admin /stats, Prometheus upstream_rq_time, upstream_cx_active, upstream_rq_retry, outlier_detection.ejections_active
nginx Access logs ($request_time, $upstream_response_time), stub_status Active connections, latency from logs

Example: Prometheus Alerts for HAProxy

groups:
  - name: loadbalancer
    rules:
      - alert: HighBackend5xxRate
        expr: |
          sum(rate(haproxy_backend_http_responses_total{code="5xx"}[5m])) by (proxy)
            / sum(rate(haproxy_backend_http_responses_total[5m])) by (proxy) > 0.02
        for: 5m
        labels: { severity: page }
      - alert: BackendQueueBuilding
        expr: max(haproxy_backend_current_queue) by (proxy) > 50
        for: 2m
        labels: { severity: warn }

nginx Log Format With Timing

log_format timing '$remote_addr "$request" $status rt=$request_time '
                  'uct=$upstream_connect_time urt=$upstream_response_time ua="$upstream_addr"';
access_log /var/log/nginx/access.log timing;

A large gap between rt (total) and urt (upstream) points to slow clients or the proxy itself. A large uct points to back-end connection problems.

Diagnosing With Metrics

Symptom Likely cause
LB 504s rising, back-end latency high Slow back ends or timeouts set too low
LB 502s, back ends report no errors Back ends closing keep-alive connections, crashes, protocol mismatch
503 “no healthy upstream” All back ends failing checks
One back end at 3× the requests Stickiness, hash hot key, weight misconfiguration
Rejected connections LB connection limit or capacity exhausted

Pitfalls

  • Watching averages instead of percentiles.

  • Not separating LB-generated from back-end errors.

  • Missing saturation signals (queues), so problems surface only as errors.

  • Alerting on every metric rather than on SLO-related symptoms.

Interview Tips

  • Organise by golden signals: traffic, latency, errors and saturation, plus health and distribution.

  • Name concrete metrics (CloudWatch, HAProxy, Envoy).

  • Explain how metrics help diagnose issues and how to alert on SLOs.

20

Describe one method to handle SSL/TLS traffic in load balancing.

One common method is TLS termination (SSL offloading) at the load balancer. The load balancer holds the certificate and private key, completes the TLS handshake with the client, decrypts the request, and forwards it to the back end either as plain HTTP on a private network or re-encrypted (TLS re-encryption, also called TLS bridging). Benefits: - Offloads CPU-heavy handshakes from application servers. - Centralises certificate management, including automated renewal (ACM, Let’s Encrypt, cert-manager). - Enables L7 features: routing by path, header or cookie, cookie stickiness, WAF inspection, compression, caching, HTTP/2 and HTTP/3 to clients. - Applies consistent TLS policy in one place (TLS 1.2+/1.3, strong ciphers, HSTS, OCSP stapling).

Trade-offs: traffic between the load balancer and back ends is unencrypted unless re-encrypted, which many compliance regimes and zero-trust designs disallow. The load balancer becomes a high-value target holding private keys. Back ends must learn the original client details from X-Forwarded-For and X-Forwarded-Proto headers. Alternatives are TLS passthrough (L4: the back end terminates TLS, routing by SNI only) and end-to-end re-encryption or mTLS to back ends (service meshes).

Read full answer

Applies tonginx, HAProxy, Envoy, AWS ALB, Azure Application Gateway, GCP HTTPS LB

Flow

client --TLS 1.3 (cert for shop.example.com)--> LB [terminates TLS, inspects HTTP]
                                                 |-- HTTP (private network) --> app1
                                                 |   or HTTPS/mTLS (re-encrypted) --> app2
headers added: X-Forwarded-For: 203.0.113.7, X-Forwarded-Proto: https

Example: nginx TLS Termination With Re-encryption

server {
    listen 443 ssl;
    http2 on;
    server_name shop.example.com;

    ssl_certificate     /etc/ssl/shop/fullchain.pem;
    ssl_certificate_key /etc/ssl/shop/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_session_cache   shared:SSL:20m;          # session resumption: fewer full handshakes
    add_header Strict-Transport-Security "max-age=31536000" always;

    location / {
        proxy_pass https://app_backend;           # re-encrypt to back ends
        proxy_ssl_verify on;
        proxy_ssl_trusted_certificate /etc/ssl/internal-ca.pem;
        proxy_ssl_server_name on;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
    }
}
server {
    listen 80;
    server_name shop.example.com;
    return 301 https://$host$request_uri;         # force HTTPS
}
openssl s_client -connect shop.example.com:443 -servername shop.example.com -tls1_3 </dev/null \
  | grep -E 'Protocol|Cipher|Verify return code'

Methods Compared

Method LB sees HTTP? Back-end encryption Typical use
Termination (offload) Yes None (private network) Most web apps in trusted VPCs
Termination + re-encryption Yes TLS or mTLS Compliance (PCI DSS), zero trust
Passthrough (L4, SNI routing) No End-to-end from client Strict end-to-end needs, client certificates at the app, non-HTTP TLS

Best Practices

  • TLS 1.2 minimum, prefer 1.3; disable weak ciphers; enable OCSP stapling and session resumption.

  • Automate certificate renewal and alert on expiry.

  • Protect private keys: managed certificate services, HSMs or KMS-backed keys, tight access control.

  • Make back ends trust forwarded headers only from the load balancer.

  • Redirect HTTP to HTTPS and set HSTS.

Pitfalls

  • Expired certificates: a common cause of outages.

  • Apps generating http:// links or redirect loops because they ignore X-Forwarded-Proto.

  • Plaintext back-end traffic crossing untrusted networks.

  • Trusting X-Forwarded-For from clients, allowing IP spoofing.

Interview Tips

  • Describe termination: the LB handles the handshake, decrypts, and forwards.

  • Give benefits (offload, central certificates, L7 features) and risks (plaintext back-end hop, key protection).

  • Mention re-encryption and passthrough as alternatives.

21

Explain the concept of direct server return and its significance in load balancing.

Direct server return (DSR) is an L4 load balancing mode in which requests pass through the load balancer, but responses go from the back-end server directly to the client, bypassing the load balancer. How it works (the classic L2 DSR, or “direct routing”): 1. The client sends a packet to the virtual IP (VIP). 2. The load balancer picks a server and rewrites only the destination MAC address, forwarding the packet unchanged at L3. Tunnelled DSR variants (IP-in-IP, GRE, GUE) encapsulate the packet instead, so servers can be on other subnets. 3. Each server has the VIP configured on a loopback interface, with ARP replies for it suppressed, so it accepts packets addressed to the VIP. 4. The server replies directly to the client, using the VIP as its source address, so the client sees a normal conversation.

Significance: in most web workloads, responses are far larger than requests (videos, downloads, pages). With DSR the load balancer only handles the small inbound traffic, so it can serve much higher throughput with less bandwidth and CPU, and adds less latency on the response path. That is why DSR is used by CDNs, streaming, gaming and large L4 balancers: Linux IPVS -g (gatekeeper/direct routing) mode, Facebook’s Katran, GitHub’s GLB Director and F5 nPath.

Read full answer

Limitations: the load balancer sees only one direction, so no TLS termination, HTTP inspection, response rewriting, cookie stickiness or L7 features. It needs network configuration on every server (loopback VIP, ARP settings, or tunnel decapsulation). Port translation is generally not possible. Troubleshooting and health tracking are harder, and asymmetric routing can trip firewalls and stateful devices.

Applies to: L4 load balancers (Linux IPVS/LVS, Katran, GLB Director, F5, Citrix)

Packet Flow

Request:   client 203.0.113.7 -> VIP 198.51.100.10:443 -> LB (rewrites MAC) -> server2 (VIP on lo)
Response:  server2 (src 198.51.100.10:443) -> client 203.0.113.7    (LB not involved)

Example: Linux IPVS Direct Routing

# On the load balancer (director)
ipvsadm -A -t 198.51.100.10:443 -s wlc                    # virtual service, weighted least connections
ipvsadm -a -t 198.51.100.10:443 -r 10.0.0.21:443 -g -w 1  # -g = direct routing (DSR)
ipvsadm -a -t 198.51.100.10:443 -r 10.0.0.22:443 -g -w 1

# On each real server: accept the VIP, but never answer ARP for it
ip addr add 198.51.100.10/32 dev lo
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2

With -g, the director and real servers must share an L2 segment. IPVS tunnelling mode (-i, IP-in-IP) removes that requirement.

DSR Versus Other Modes

Mode Return path LB load L7 features Server changes
Full proxy (L7) Through LB High (both directions, TLS) Yes None
NAT (L4) Through LB Medium No Default gateway is the LB
DSR Direct to client Low (inbound only) No VIP on loopback, ARP tuning or decapsulation

When to Use DSR

Good fit Poor fit
High-bandwidth responses (video, downloads, game state) Need TLS termination or HTTP routing at the LB
Very high connection rates at L4 Need response inspection (WAF, compression)
Edge networks fronting L7 proxies (Katran → proxy tier) Environments where server network config is restricted (many managed clouds)

A common pattern combines both: an L4 DSR tier spreads traffic across a fleet of L7 proxies, which terminate TLS.

Pitfalls

  • ARP problems: servers answering ARP for the VIP, causing traffic to bypass the LB unpredictably.

  • Firewalls dropping asymmetric flows.

  • Expecting L7 features in DSR mode.

  • MTU issues with tunnelled DSR (encapsulation overhead).

Interview Tips

  • Define DSR: requests via the LB, responses directly to the client.

  • Explain how (MAC rewrite or tunnelling, VIP on loopback, ARP suppression).

  • Give the significance (throughput for asymmetric traffic) and the limitations (no L7, network setup).

22

How can caching be integrated with load balancing to improve performance?

Caching and load balancing combine at several layers: - Cache at the load balancer or reverse proxy: nginx proxy_cache, Varnish, HAProxy’s small object cache and Envoy’s cache filter store cacheable responses. Repeated requests are answered without reaching back ends, which reduces latency and back-end load. - CDN in front of the load balancer: edge caches serve static assets and cacheable pages close to users, so the load balancer and origins see only cache misses. - Cache-aware routing to back-end or cache servers: consistent hashing sends each key or URL to the same cache node, which maximises hit rates and avoids duplicating data across nodes, while minimising reshuffling on scaling. Examples: nginx hash $request_uri consistent in front of Varnish nodes, Envoy ring hash or Maglev. - Server-local caches: with stickiness or hashing, each app server keeps warm in-process caches for its users or keys. - Shared distributed caches (Redis, Memcached) behind stateless app servers: any server can use the cache, so the load balancer can route freely.

Supporting techniques: - Cache-Control headers, so every layer knows what to cache and for how long. - Request coalescing (collapsing concurrent misses for the same key) and stale-while-revalidate, to avoid thundering herds. - Vary-correct cache keys, so responses that differ by user, language or encoding aren’t mixed up.

Read full answer

Applies tonginx, Varnish, Envoy, HAProxy, CDNs (Cloudflare, CloudFront, Fastly)

Architecture

users -> CDN edge cache (static, public pages)
      -> L7 load balancer with micro-cache (1–10 s for hot API responses)
      -> hash on URL -> cache tier (Varnish nodes)  -- miss --> app servers
                                                             -> Redis (shared data cache) -> DB

Example: nginx Caching Load Balancer

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api:50m max_size=2g inactive=10m;

upstream app {
    least_conn;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

server {
    listen 443 ssl;
    location /api/products {
        proxy_cache api;
        proxy_cache_key "$scheme$host$request_uri";
        proxy_cache_valid 200 30s;
        proxy_cache_lock on;                                     # coalesce concurrent misses
        proxy_cache_use_stale error timeout updating http_500 http_503;
        proxy_cache_background_update on;                        # serve stale while refreshing
        add_header X-Cache-Status $upstream_cache_status;
        proxy_pass http://app;
    }
    location /api/cart {
        proxy_pass http://app;                                   # personalised: never cache
    }
}

Example: Hashing Requests Across Cache Nodes

upstream varnish_tier {
    hash $request_uri consistent;     # same URL -> same cache node; ~1/N keys move on scaling
    server 10.0.2.11:6081;
    server 10.0.2.12:6081;
    server 10.0.2.13:6081;
}

With round robin, each cache node would hold its own copy of every popular object, giving lower hit rates and N× origin fetches on misses.

Integration Options

Layer Caches Benefit Watch out for
CDN Static assets, public pages, some APIs Global latency, origin offload Invalidation, personalised content
LB / reverse proxy Hot responses, micro-caching Fewer back-end calls Cache keys, auth headers
Hash-routed cache tier Large object sets High hit rate, no duplication Hot keys
App-local cache with affinity Per-user or per-key data Fastest access Imbalance, loss on failover
Shared Redis/Memcached Data and sessions Stateless app tier Network hop, cache stampedes

Pitfalls

  • Caching personalised responses (with Set-Cookie or Authorization), which leaks data between users.

  • Missing Vary handling (encoding, language).

  • Thundering herds when popular entries expire: use locks, coalescing or stale serving.

  • Round robin across cache nodes, which dilutes hit rates.

  • No invalidation strategy for content that must update quickly.

Interview Tips

  • Describe the layers: CDN, load balancer cache, hashed cache tier, shared caches.

  • Explain consistent hashing for cache locality.

  • Cover Cache-Control, coalescing, stale-while-revalidate and personalisation safety.

23

Discuss the challenges of load balancing microservice architectures.

Microservices turn load balancing from one edge problem into a many-to-many problem: hundreds of services, thousands of instances and east-west traffic between services. The main challenges: - Dynamic endpoints: instances start, stop, autoscale and move constantly, so load balancers need service discovery (Kubernetes endpoints, Consul, Eureka, DNS) and must keep up without sending traffic to dead pods. - Where to balance: centralised proxies (an extra hop, a bottleneck, a single point of failure), client-side libraries (language-specific: gRPC, older Netflix Ribbon), or sidecar or ambient service meshes (Istio, Linkerd, Envoy), which add operational complexity and resource overhead. - Long-lived, multiplexed connections: HTTP/2 and gRPC keep one connection open for many requests. Kubernetes Services balance at L4 per connection, so traffic can pile onto a few pods. You need L7, per-request balancing. - Many independent balancers: every client instance makes its own decisions with partial information. Power of two choices and least outstanding requests cope better than global counters. - Cascading failures and retries: retries at several layers multiply load (retry storms), and slow dependencies tie up resources. This calls for timeouts, retry budgets, circuit breakers, outlier detection and bulkheads. - Heterogeneous services: different protocols, latency profiles and scaling behaviours need different policies. - Deployments: canaries, blue-green and versioned routing across many services, with graceful draining during frequent rollouts. - Security: mTLS and authorisation between services, with certificate rotation. - Observability: distributed tracing, per-service metrics and correlation IDs are needed to understand where latency and errors come from. - Locality and cost: cross-zone traffic adds latency and data-transfer costs, so zone-aware routing helps.

Read full answer

Applies toKubernetes, service meshes (Istio, Linkerd), Envoy, gRPC

Typical Layout

internet -> edge LB / API gateway (north-south: TLS, auth, rate limits)
         -> service A pods --(sidecar or client LB: per-request, P2C, mTLS)--> service B pods
                                                              \--> service C pods
service discovery (Kubernetes API / xDS control plane) keeps endpoint lists current

Example: Istio DestinationRule

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payments
spec:
  host: payments.prod.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST           # per-request, P2C-based least request in Envoy
      localityLbSetting:
        enabled: true                 # prefer same-zone endpoints
    connectionPool:
      http:
        http2MaxRequests: 1000
        maxRequestsPerConnection: 100
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Locality-aware load balancing in Istio only takes effect when outlier detection is configured, as it is here.

Challenges and Mitigations

Challenge Mitigation
Churning endpoints Service discovery, readiness probes, endpoint updates via the xDS API
gRPC/HTTP/2 imbalance with kube-proxy Headless Services + client-side LB, or mesh L7 balancing
Retry storms Retry budgets, retries at one layer, jittered backoff
Slow or failing instances Outlier detection, circuit breakers, timeouts
Cross-zone cost and latency Locality-aware routing with failover
Frequent deployments Draining (preStop hooks), weighted canaries
Service-to-service security mTLS via mesh, short-lived certificates
Debugging Distributed tracing (OpenTelemetry), per-route metrics

Pitfalls

  • Relying on kube-proxy for gRPC, so one pod receives most traffic.

  • Retries everywhere (client, mesh, gateway), multiplying traffic during incidents.

  • No timeouts, so threads pile up waiting on slow services.

  • Mesh adoption without the operational capacity to run it.

  • Ignoring draining, causing errors on every deployment.

Interview Tips

  • Highlight east-west traffic, dynamic discovery, gRPC/HTTP/2 imbalance and cascading failures.

  • Compare central proxies, client-side load balancing and service meshes.

  • Give mitigations: L7 per-request balancing, outlier detection, retry budgets, locality, mTLS and tracing.

24

How do you configure a health check on a load balancer?

Configuring a health check means deciding what to probe, how often, and when to change a server’s state: 1. Choose the check type: TCP (port accepts connections), HTTP/HTTPS (expect a status code and optionally a body), gRPC health checking protocol, or a custom script. Prefer HTTP for web services, since a TCP check misses hung applications. 2. Expose a health endpoint in the application, such as /healthz (liveness) and /ready (readiness). It should be fast and cheap, check critical local readiness (started, warm, can reach essential dependencies), and not cascade: a shared database blip shouldn’t fail every server at once. 3. Set timing and thresholds: - interval (for example 5 s); - timeout (for example 2 s, shorter than the interval); - unhealthy threshold (fall, for example 3 consecutive failures); - healthy threshold (rise, for example 2 consecutive passes). These trade detection speed against flapping. 4. Define the success criteria: status codes (200 or a range), an optional body match, and the port and host header. 5. Add passive checks and outlier detection (real-traffic errors) to catch problems between probes. 6. Plan recovery: slow start for recovered servers, connection draining for removed ones, and fail-open behaviour if every server fails. 7. Secure and monitor: restrict the endpoint to the load balancer’s network, don’t expose sensitive details, and alert on unhealthy host counts.

Read full answer

Applies tonginx, HAProxy, AWS ALB/NLB, Kubernetes probes (Python 3.12 simulation)

Code Example: Rise/Fall Thresholds

# Health-check state machine with rise/fall thresholds (like HAProxy rise 2 fall 3)
class Backend:
    def __init__(self, name, rise=2, fall=3):
        self.name, self.rise, self.fall = name, rise, fall
        self.up, self.streak = True, 0          # streak counts results that disagree with the state

    def record(self, ok):
        if ok == self.up:
            self.streak = 0                     # result agrees with the current state
            return
        self.streak += 1
        if self.up and self.streak >= self.fall:
            self.up, self.streak = False, 0
        elif not self.up and self.streak >= self.rise:
            self.up, self.streak = True, 0

b = Backend("app2")
probes = [1, 0, 1, 0, 0, 0, 1, 0, 1, 1]        # 1 = probe passed, 0 = failed
for t, ok in enumerate(probes):
    b.record(bool(ok))
    print(f"probe {t}: {'pass' if ok else 'FAIL'} -> {'UP' if b.up else 'DOWN'}")
# probe 1: FAIL -> UP     (one failure is ignored: no flapping)
# probe 5: FAIL -> DOWN   (third consecutive failure)
# probe 9: pass -> UP     (second consecutive pass)

Single failed probes are ignored, so a server isn’t removed for one blip. Recovery needs consecutive passes, so a flapping server stays out.

HAProxy

backend app
    option httpchk
    http-check send meth GET uri /ready ver HTTP/1.1 hdr Host app.internal
    http-check expect status 200
    default-server inter 5s fastinter 1s downinter 10s fall 3 rise 2 slowstart 30s
    server app1 10.0.1.11:8080 check
    server app2 10.0.1.12:8080 check

fastinter probes faster while a server is changing state; downinter probes less often while it is down.

AWS Application Load Balancer

aws elbv2 modify-target-group --target-group-arn "$TG" \
  --health-check-protocol HTTP --health-check-path /ready \
  --health-check-interval-seconds 10 --health-check-timeout-seconds 5 \
  --healthy-threshold-count 2 --unhealthy-threshold-count 3 \
  --matcher HttpCode=200

Kubernetes Readiness and Liveness Probes

readinessProbe:            # failing -> removed from Service endpoints (no restart)
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 3
livenessProbe:             # failing -> container restarted; keep it simple
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3

Open-source nginx only has passive checks (max_fails, fail_timeout); active health_check is an NGINX Plus feature.

Choosing Settings

Setting Lower value Higher value
Interval Faster detection, more probe traffic Less load, slower detection
Timeout Detects slow servers Tolerates brief slowness
Unhealthy threshold Quick removal, risk of flapping Stable, slower removal
Healthy threshold Quick return Ensures real recovery

Worst-case detection time ≈ interval × unhealthy threshold (plus timeout): 5 s × 3 = about 15 s.

What the Endpoint Should Check

Check Include?
Process up and serving HTTP Yes
Warm-up done (caches, connections) Yes (readiness)
Local disk or memory critically low Yes
Shared database reachable Carefully: often better as degraded mode than a failure
Third-party APIs Usually no

Pitfalls

  • TCP-only checks for HTTP apps (miss hung processes).

  • Deep checks on shared dependencies, so a database blip removes every server: a self-inflicted outage.

  • Heavy health endpoints that add load.

  • Liveness probes that check dependencies, causing restart loops.

  • Thresholds too low, so servers flap.

Interview Tips

  • Walk through type, endpoint, interval, timeout, thresholds and success criteria.

  • Show a concrete config (HAProxy, ALB, Kubernetes).

  • Discuss readiness versus liveness, shallow versus deep checks, passive checks and slow start.

25

What is connection draining, and why is it used in load balancing?

Connection draining (called deregistration delay in AWS, and graceful drain or “draining” mode in HAProxy, Envoy and F5) is how a load balancer removes a back end gracefully: - it stops sending new requests or connections to the server; - it lets existing in-flight requests and open connections finish until they complete or a drain timeout expires; - only then does it fully remove the server.

It is used so that routine operations don’t break user requests: - deployments and rolling updates (old instances drain while new ones take over); - autoscaling scale-in; - maintenance and patching; - instance replacement (spot interruption notices, host retirements).

Read full answer

Without draining, removing a server would cut active requests (failed uploads, half-finished checkouts, dropped WebSockets), causing 5xx errors and resets on every deploy. The drain timeout should match typical request duration: short (10–30 s) for APIs, longer for uploads, while long-lived connections like WebSockets need application help, such as asking clients to reconnect. Draining works together with the application’s graceful shutdown: on SIGTERM it fails readiness, finishes work, then exits.

Applies to: AWS ELB, HAProxy, Envoy, nginx, Kubernetes

Lifecycle

t=0     deploy starts: instance i-2 deregistered            state: draining
        LB sends no new requests to i-2; 14 requests still in flight
t=0-8s  in-flight requests complete normally
t=8s    no active requests -> i-2 removed (or at deregistration delay of 30 s at the latest)
t=8s+   instance terminated

Configuration Examples

# AWS target group: wait up to 30 s for in-flight requests (default 300 s)
aws elbv2 modify-target-group-attributes --target-group-arn "$TG" \
  --attributes Key=deregistration_delay.timeout_seconds,Value=30
# HAProxy runtime API: put app2 in drain mode (no new traffic, existing sessions continue)
echo "set server app/app2 state drain" | socat stdio /run/haproxy/admin.sock
echo "show servers state app" | socat stdio /run/haproxy/admin.sock    # watch connections fall
echo "set server app/app2 state maint" | socat stdio /run/haproxy/admin.sock
# Kubernetes: delay shutdown so endpoints update, and allow time to finish work
spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "10"]     # endpoint removal propagates before SIGTERM

Kubernetes removes a terminating pod from Service endpoints, but that update reaches kube-proxy, ingress controllers and cloud LBs asynchronously. The preStop sleep covers that gap.

Choosing the Drain Timeout

Workload Typical timeout Note
Fast APIs 10–30 s Default 300 s in AWS slows deploys unnecessarily
File uploads, reports Minutes Or design for resumable uploads
WebSockets, streaming Application-driven reconnect Send a close or reconnect message
Batch workers Until the current job finishes Pull-based queues: stop taking jobs

Draining Versus Related Concepts

Concept Meaning
Connection draining No new traffic; finish existing
Health-check failure Server removed because it failed probes (may cut connections)
Maintenance mode Fully out of rotation
Slow start Opposite direction: gradually add traffic to a new server

Pitfalls

  • Drain timeout too short, cutting long requests.

  • Drain timeout too long, so deployments and scale-in crawl.

  • App exits immediately on SIGTERM, ignoring the drain.

  • Sticky sessions that keep users on draining servers longer than expected.

  • Long-lived connections never closing, so the drain always hits its timeout.

Interview Tips

  • Define draining: stop new traffic, let existing traffic finish, then remove.

  • Explain why: zero-downtime deployments, scale-in and maintenance.

  • Mention timeouts, deregistration delay, preStop hooks and app graceful shutdown.

26

How do cloud load balancers like AWS ELB differ from on-premise solutions?

Cloud load balancers (AWS Elastic Load Balancing: ALB, NLB, GWLB; Azure Load Balancer and Application Gateway; Google Cloud Load Balancing) are managed services: - the provider runs, patches and scales them automatically, with built-in high availability across zones; - they are configured through APIs, consoles and infrastructure as code; - they are billed pay-as-you-go (hours plus capacity units or data processed); - they are tightly integrated with autoscaling groups, certificate managers (ACM, Key Vault), WAF and DDoS protection (AWS WAF and Shield, Azure Front Door, Cloud Armor), IAM, monitoring and container platforms.

On-premise solutions are hardware appliances (F5 BIG-IP, Citrix NetScaler, A10) or software you run yourself (HAProxy, nginx, Envoy, Linux IPVS, Kemp). You own sizing, redundancy (active-passive pairs), upgrades, monitoring and capacity planning. In return you get full control over features and tuning, often richer or more customisable functionality (iRules scripting, advanced traffic manipulation, full TCP and TLS tuning), predictable costs at steady high volume, and the ability to meet data-locality or regulatory constraints.

Read full answer

Trade-offs: - Cloud: less control, provider-specific limits and features, per-GB costs at scale, and lock-in. - On-premise: upfront capital cost, slower scaling, and operational burden.

Many organisations use both: cloud LBs at the edge, and self-managed software proxies (Envoy, HAProxy, nginx) inside clusters, or hybrid setups.

Applies to: AWS ELB, Azure Load Balancer and Application Gateway, Google Cloud LB versus F5, Citrix, HAProxy, nginx

Comparison

Aspect Cloud managed LB On-premise (appliance or self-hosted software)
Operation Provider runs, patches and scales You install, patch, monitor and scale
Scaling Automatic and elastic (pre-warming rarely needed) Capacity planned; buy or deploy more
HA Built-in multi-zone You build it (active-passive pairs, VRRP, clustering)
Cost model Opex: hourly + LCU/data processed Capex (hardware, licences) + staff; cheaper at steady high volume
Provisioning Minutes via API/IaC Days to weeks (hardware), faster for software
Integration Autoscaling, certificates, IAM, WAF, logs, Kubernetes controllers Manual or via vendor tooling and automation
Customisation Limited to provider features Deep: scripting (iRules, Lua), custom modules, kernel tuning
Visibility Provider metrics and logs Full access to internals
Limits Rule counts, timeouts, quotas set by provider Hardware and licence limits
Lock-in Provider-specific config Vendor-specific (appliances) or portable (open source)
Compliance Shared responsibility model Full control over location and data

AWS ELB Types

Type Layer Use
Application Load Balancer L7 HTTP/HTTPS, gRPC Path, host and header routing; WAF; Lambda targets
Network Load Balancer L4 TCP/UDP/TLS Very high throughput, static IPs, low latency
Gateway Load Balancer L3 (GENEVE) Inserting firewall and inspection appliances
Classic Load Balancer L4/L7 legacy Previous generation, being retired

Example: Cloud LB as Infrastructure as Code (Terraform)

resource "aws_lb" "web" {
  name               = "web-alb"
  load_balancer_type = "application"
  subnets            = var.public_subnet_ids        # spread across availability zones
  security_groups    = [aws_security_group.alb.id]
}

resource "aws_lb_target_group" "app" {
  name     = "app"
  port     = 8080
  protocol = "HTTP"
  vpc_id   = var.vpc_id
  health_check { path = "/ready" }
}

The equivalent on-premise setup means installing and configuring HAProxy or an F5 pair, configuring keepalived or VRRP for a floating IP, and planning capacity.

Choosing

Choose cloud LB when Choose on-premise or self-managed when
Workloads run in that cloud Workloads are in your data centre
Elastic, variable traffic Steady, very high traffic where per-GB fees dominate
Small ops team Need deep customisation or specific vendor features
Fast provisioning matters Strict data-locality or air-gapped environments

Pitfalls

  • Assuming infinite instant scale: cloud LBs still have quotas, and sudden spikes may need provider coordination.

  • Ignoring data-processing costs at high volume.

  • Single on-premise appliance without an HA pair.

  • Lifting appliance-specific scripting into clouds that cannot replicate it.

Interview Tips

  • Contrast managed, elastic, integrated, opex with controlled, customisable, capex, self-operated.

  • Name ALB, NLB and GWLB and their layers.

  • Discuss cost, compliance and hybrid patterns.

27

Explain the role of load balancing in auto-scaling groups and clusters.

Load balancing is the front door that makes auto-scaling useful. Auto-scaling changes the number of instances based on demand (CPU, request count, queue depth, schedules), and the load balancer: - Registers new instances automatically once they pass health checks, then spreads traffic across them, optionally with slow start so cold instances aren’t flooded. - Drains and deregisters instances during scale-in, so in-flight requests complete before termination. - Provides a stable endpoint (DNS name or VIP), so clients never need to know which instances exist. - Supplies health signals: in AWS, an Auto Scaling group can use ELB health checks to replace instances that the load balancer marks unhealthy, not just ones whose host failed. - Supplies scaling metrics: request count per target, latency or connection counts can drive scaling policies (AWS ALBRequestCountPerTarget, Kubernetes HPA with custom metrics). - Balances across zones, so scaling and failures in one zone don’t overload others.

In Kubernetes clusters, the same roles are split: the HPA scales pods, the cluster autoscaler or Karpenter scales nodes, and Services, Ingress or Gateway controllers route to ready pods only. Together they give elasticity without client-side changes.

Read full answer

Applies toAWS Auto Scaling groups with ELB, Azure VM Scale Sets, GCP managed instance groups, Kubernetes

Lifecycle

scale-out:  metric high -> ASG launches instance -> boots, app warms up
            -> LB health check passes (healthy threshold) -> registered -> traffic (slow start)
scale-in:   metric low -> ASG picks instance -> LB deregisters (draining, delay 30 s)
            -> in-flight requests finish -> instance terminated
failure:    LB health check fails -> ASG (ELB health type) replaces the instance

Example: AWS Target Tracking on Requests per Target

aws autoscaling put-scaling-policy --auto-scaling-group-name web-asg \
  --policy-name req-per-target --policy-type TargetTrackingScaling \
  --target-tracking-configuration '{
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ALBRequestCountPerTarget",
      "ResourceLabel": "app/web-alb/50dc6c495c0c9188/targetgroup/app/943f017f100becff"
    },
    "TargetValue": 800
  }'

aws autoscaling update-auto-scaling-group --auto-scaling-group-name web-asg \
  --health-check-type ELB --health-check-grace-period 120

The ASG keeps about 800 requests per instance per minute. With ELB health checks, an instance that fails the load balancer’s checks is replaced. The grace period stops new instances being killed while they boot.

Responsibilities

Concern Auto-scaler Load balancer
How many instances Yes No (provides metrics)
Which instance gets a request No Yes
Adding new capacity safely Launches Health checks, slow start, registration
Removing capacity safely Terminates Draining / deregistration delay
Replacing broken instances Yes (with LB health info) Stops routing to them
Stable client endpoint No Yes

Tuning Tips

  • Make health-check grace periods and readiness reflect real start-up and warm-up time.

  • Keep scale-in protection or cooldowns to avoid thrashing.

  • Choose scaling metrics that reflect user load (requests per target, latency) rather than only CPU.

  • Use multiple zones and cross-zone balancing.

  • Pre-scale for known events, because autoscaling reacts with a delay.

Pitfalls

  • New instances receiving traffic before they are warm: high latency or errors on scale-out.

  • No deregistration delay, so scale-in drops requests.

  • Health checks that fail under load, causing replacement cascades.

  • Stateful instances in groups that scale in, losing sessions.

Interview Tips

  • Explain how the LB registers, health-checks, drains and spreads traffic as instances come and go.

  • Mention ELB health checks driving replacement and LB metrics driving scaling.

  • Cover the Kubernetes equivalents: HPA, cluster autoscaler, readiness and Services.

28

Describe the process of load balancing in a multi-cloud environment.

In a multi-cloud environment, an application runs on two or more providers (for example AWS and Azure, or GCP plus Oracle Cloud). Load balancing works in layers: 1. Global traffic steering across clouds: a provider-neutral DNS/GSLB or edge service decides which cloud and region serves each user. Examples: Cloudflare Load Balancing, NS1, Akamai, F5 Distributed Cloud, or Route 53 used neutrally. It routes by health, geography, latency, weights or cost, and fails over if one cloud degrades. 2. Regional load balancing inside each cloud: each cloud’s native LB (ALB/NLB, Azure Application Gateway or Load Balancer, Google Cloud LB), or self-managed proxies, distributes traffic to instances, containers or functions there. 3. Service-to-service traffic: inside and between clouds, a service mesh (Istio multi-cluster, Consul, Cilium Cluster Mesh) or private connectivity (VPN, Interconnect, ExpressRoute, Direct Connect, or a cloud exchange) handles east-west traffic with mTLS, locality-aware routing and failover.

The process also needs: - consistent health checks that test the full stack in each cloud; - shared configuration as code (Terraform, Kubernetes manifests); - centralised observability; - a data strategy: replication across clouds, which consistency model, and which writes go where.

Read full answer

Without the data strategy, active-active routing breaks consistency.

Applies to: multi-cloud architectures (Cloudflare, NS1, F5 Distributed Cloud, Terraform, Istio)

Architecture

users -> global DNS/GSLB or anycast edge (Cloudflare LB: health checks against both clouds)
          |-- 60% --> AWS us-east-1: ALB -> EKS services
          |-- 40% --> Azure East US: Application Gateway -> AKS services
          (failover: if AWS pool unhealthy, 100% to Azure)
east-west: Istio multi-cluster / private interconnect with mTLS
data: replicated database (e.g. CockroachDB, YugabyteDB) or a primary in one cloud with replicas elsewhere

Example: Provider-Neutral Pools (Pseudo-Config)

load_balancer: api.example.com
steering_policy: dynamic_latency        # or geo, weighted, failover
pools:
  - name: aws-us-east
    origins: [ { address: alb-123.us-east-1.elb.amazonaws.com, weight: 1 } ]
    monitor: { path: /healthz/deep, interval: 15s, expected_codes: "200" }
  - name: azure-east-us
    origins: [ { address: api-eastus.azurefd.net, weight: 1 } ]
    monitor: { path: /healthz/deep, interval: 15s, expected_codes: "200" }
fallback_pool: azure-east-us
session_affinity: cookie

Each provider has its own exact syntax (Cloudflare pools and monitors, NS1 filter chains), but the concepts are the same: pools per cloud, health monitors, a steering policy and a fallback.

Steps in Practice

Step Details
Define goals Resilience to a provider outage, latency, cost, regulation, avoiding lock-in
Choose a global layer Neutral DNS/edge (not dependent on one cloud)
Standardise the platform Kubernetes, containers, IaC across clouds
Configure regional LBs Native or portable (Envoy, nginx, HAProxy) per cloud
Connect networks Private interconnects, consistent addressing, mTLS
Design data Replication, partitioning by region, conflict handling
Health and failover Deep health checks; tested failover runbooks
Observe Unified metrics, logs and traces (OpenTelemetry)

Challenges

Challenge Mitigation
Different LB features per cloud Use common denominators or portable proxies
Data consistency and egress costs Keep users’ data near their primary cloud; cache; budget egress
Latency between clouds Avoid chatty cross-cloud calls; locality-aware routing
Operational complexity IaC, GitOps, shared observability, practice drills
DNS caching slowing failover Short TTLs; anycast edge proxies

Pitfalls

  • Global DNS hosted in one of the clouds you are trying to survive losing.

  • Active-active without data replication that supports it.

  • Untested failover, which fails when needed.

  • Unexpected egress bills from cross-cloud traffic.

Interview Tips

  • Describe the layers: global steering, per-cloud LBs, east-west mesh.

  • Stress neutral health-checked DNS/edge, consistent IaC and data strategy.

  • Mention costs, latency and failover testing.

29

How does load balancing work in a hybrid cloud with private and public resources?

In a hybrid cloud, an application spans a private environment (on-premise data centre or private cloud) and a public cloud. Load balancing then has to steer traffic between environments, as well as within each one. Common patterns: - Global or DNS-level steering: a GSLB (F5 BIG-IP DNS, Cloudflare, NS1, Route 53) or an edge service directs users to on-premise or cloud endpoints by health, weights, geography or capacity. - Cloud bursting: the private data centre handles baseline load. When it nears capacity, overflow goes to the public cloud, using weights, priority pools or an on-premise LB with cloud back ends in its pool. - Active-passive DR: on-premise is primary and the cloud is a warm standby (or the reverse). Health-checked failover switches traffic. - Migration: weighted shifting (90/10 → 50/50 → 0/100) moves traffic gradually from on-premise to cloud. - Tiered split: the front end runs in the cloud near users, behind a cloud LB, while some back ends (mainframes, regulated data stores) stay on-premise, reached over private links (Direct Connect, ExpressRoute, Cloud Interconnect, VPN) through internal load balancers.

Key concerns: - network connectivity and latency between environments; - consistent security (TLS, identity, firewalls); - data locality and compliance; - health checks that see both sides; - unified observability.

Read full answer

Applies tohybrid architectures (F5, HAProxy, nginx, AWS Outposts, Azure Arc, Anthos/GKE Enterprise)

Architecture

users -> GSLB / edge (health checks on both)
          |-- primary --> on-prem F5/HAProxy VIP -> private servers (baseline capacity)
          |-- overflow -> cloud ALB -> autoscaled instances
cloud app tier --(Direct Connect / ExpressRoute)--> internal LB on-prem -> regulated database

Example: On-Premise HAProxy Bursting to Cloud

backend web
    balance leastconn
    # Private servers take traffic first
    server dc1 10.10.0.11:8080 check maxconn 500
    server dc2 10.10.0.12:8080 check maxconn 500
    # Cloud capacity, used only when all private servers are down
    server aws1 10.200.1.21:8080 check backup
    server aws2 10.200.1.22:8080 check backup

backup servers are used only when no primary server is available, which fits failover. For overflow while private servers are merely busy, keep separate private and cloud backends and switch with an ACL:

frontend fe
    bind :80
    acl dc_busy avg_queue(web_private) gt 20
    use_backend web_cloud if dc_busy
    default_backend web_private

Patterns Compared

Pattern Traffic split Benefit Watch out for
Cloud bursting Private first, overflow to cloud Pay for cloud only at peaks Data access latency from cloud to on-prem
Active-passive DR All to primary; failover Resilience Standby freshness, failover tests
Active-active Split by geo or weight Capacity and resilience Data replication and consistency
Migration Gradual weight shift Low-risk move Two environments to operate
Tiered split Front end in cloud, data on-prem Compliance Private link capacity and latency

Connectivity and Security

  • Private links with redundancy, for predictable latency and bandwidth.

  • Consistent IP planning: no overlapping CIDRs.

  • mTLS or IPsec between environments; unified identity (federated IAM, workload identity).

  • Firewall rules that allow health checks from both sides.

Pitfalls

  • Bursting stateless front ends that depend on on-premise data, so latency jumps and the link saturates.

  • Health checks that only see one side.

  • Single private link as a single point of failure.

  • Different configs drifting between environments.

Interview Tips

  • Describe GSLB steering, bursting, DR, migration and tiered splits.

  • Show concrete config (backup or overflow pools, weights).

  • Emphasise connectivity, data locality, security and consistent observability.

30

What is cross-region load balancing, and in what scenario would it be necessary?

Cross-region load balancing distributes traffic across multiple geographic regions (for example us-east-1, eu-west-1 and ap-south-1) rather than only across zones or servers in one region. It works through: - DNS-based routing (latency, geo, weighted, failover policies); - anycast global load balancers that advertise one IP worldwide and route to the nearest healthy region (Google Cloud global external Application Load Balancer, AWS Global Accelerator, Azure Front Door, Cloudflare); - global edge proxies that terminate connections near users and forward to a chosen region.

Regional load balancers then distribute traffic to servers within each region. Scenarios where it is necessary: - Regional disaster recovery: surviving the outage of an entire region (cloud incidents, natural disasters), where business requirements set an RTO and RPO that one region cannot meet. - Global users and latency: serving users on different continents within latency targets, such as under 100 ms. - Data residency and regulation: EU users served and stored in the EU (GDPR), and in-country requirements in India, China and elsewhere. - Capacity: a single region’s quotas or capacity are not enough for peak traffic. - Follow-the-sun or cost strategies, and gradual regional rollouts of new versions.

Read full answer

It brings complexity: data replication across regions (latency, consistency choices, conflict resolution), capacity headroom to absorb a failed region, cross-region data transfer costs and harder testing.

Applies to: AWS Global Accelerator and Route 53, Google Cloud global LB, Azure Front Door and Traffic Manager, Cloudflare

Architecture

anycast IP / latency DNS
users EU ---------> eu-west-1  ALB -> app -> DB replica (or regional primary)
users US ---------> us-east-1  ALB -> app -> DB primary
users APAC -------> ap-south-1 ALB -> app -> DB replica
region outage: health checks fail -> traffic shifts to the next nearest healthy region

Example: AWS Global Accelerator Endpoint Groups

aws globalaccelerator create-endpoint-group --listener-arn "$LISTENER" \
  --endpoint-group-region eu-west-1 --traffic-dial-percentage 100 \
  --endpoint-configurations EndpointId="$EU_ALB_ARN",Weight=100

aws globalaccelerator create-endpoint-group --listener-arn "$LISTENER" \
  --endpoint-group-region us-east-1 --traffic-dial-percentage 100 \
  --endpoint-configurations EndpointId="$US_ALB_ARN",Weight=100

Global Accelerator gives two static anycast IPs. Traffic enters AWS’s network at the nearest edge and goes to the closest healthy endpoint group. For ALB and NLB endpoints, Global Accelerator uses the load balancer’s own target health, so a region whose targets are unhealthy is skipped. The traffic dial reduces a region’s share (for example to 0% during maintenance), and failover doesn’t depend on DNS TTLs.

Active-Active Versus Active-Passive

Model How Pros Cons
Active-active All regions serve traffic Low latency everywhere; capacity used Multi-region data writes, conflicts
Active-passive One primary; failover to standby Simpler data (single writer) Standby cost, failover time, untested standby
Read-local, write-global Reads in every region, writes to one Good compromise Write latency for distant users

When It Is Necessary

Scenario Necessary?
Regional SaaS with small user base, RTO of hours No: multi-AZ is enough
Global consumer app with strict latency targets Yes
Banking or healthcare with regional outage SLAs Yes
EU and US customers with residency rules Yes (often with per-region data partitioning)
Internal tool used in one country Usually no

Pitfalls

  • Multi-region front ends with a single-region database, so a region outage still takes the app down.

  • No capacity headroom, so the failover region collapses.

  • Ignoring data transfer costs and replication lag.

  • Never running failover drills.

Interview Tips

  • Define it as distributing traffic across regions, via DNS, anycast or global proxies.

  • Give scenarios: region DR, global latency, data residency, capacity.

  • Discuss data replication, headroom, active-active versus active-passive and costs.

31

Explain how a load balancer can make routing decisions based on content type.

An L7 (application-layer) load balancer terminates the connection, parses the HTTP request, and chooses a back-end pool based on its content. This is called content-based routing. Signals it can use: - the URL path or file extension (/static/*, *.jpg, /api/*, /video/*); - the Host header or SNI (virtual hosting: api.example.com versus www.example.com); - the request Content-Type (for example application/grpc, multipart/form-data uploads, application/json); - the Accept header (requests for text/html versus application/json or images); - other headers and cookies (API version, user tier, Upgrade: websocket, A/B cohort); - the HTTP method and query parameters.

This lets each kind of content go to the best-suited servers: - static files to a cache or object store; - images to an image-processing service; - uploads to servers with large body limits; - gRPC to HTTP/2-capable back ends; - WebSockets to connection-optimised servers; - APIs to application servers.

Read full answer

Benefits: independent scaling, specialised tuning, caching and simpler services. Trade-offs: L7 processing cost (TLS termination and parsing), more complex configuration, and headers can be spoofed, so never use them alone for security decisions. L4 load balancers cannot do this because they don’t see HTTP content.

Applies to: nginx, HAProxy, Envoy, AWS ALB listener rules, Kubernetes Ingress and Gateway API

Example: HAProxy Content Routing

frontend web
    bind :443 ssl crt /etc/haproxy/certs/site.pem alpn h2,http/1.1
    acl is_static   path_end -i .css .js .png .jpg .webp .svg .woff2
    acl is_upload   req.hdr(content-type) -m beg multipart/form-data
    acl is_grpc     req.hdr(content-type) -m beg application/grpc
    acl is_ws       req.hdr(upgrade) -i websocket
    acl is_api      path_beg /api/

    use_backend static_cache  if is_static
    use_backend uploads       if is_upload
    use_backend grpc_services if is_grpc
    use_backend websockets    if is_ws
    use_backend api           if is_api
    default_backend web_app

backend grpc_services
    server g1 10.0.3.11:50051 check proto h2       # HTTP/2 to back end for gRPC

Example: nginx by Path and Accept Header

map $http_accept $html_or_api {
    default          api_backend;
    ~text/html       ssr_backend;      # browsers asking for HTML -> server-side rendering
}

server {
    location /images/ { proxy_pass http://image_service; }
    location ~* \.(css|js|woff2)$ { root /srv/static; expires 30d; }
    location / { proxy_pass http://$html_or_api; }
}

ssr_backend and api_backend are upstream blocks, so proxy_pass http://$html_or_api picks one per request.

AWS ALB Listener Rule (Header Condition)

aws elbv2 create-rule --listener-arn "$LISTENER" --priority 10 \
  --conditions '[{"Field":"http-header","HttpHeaderConfig":{"HttpHeaderName":"Content-Type","Values":["application/grpc*"]}}]' \
  --actions Type=forward,TargetGroupArn="$GRPC_TG"

Common Routing Rules

Content Detected by Routed to
Static assets Extension or /static/ CDN, cache, object storage
Images and video Path or extension Media service, streaming servers
File uploads Content-Type: multipart/form-data, method POST Upload service with large body limits
gRPC Content-Type: application/grpc HTTP/2 back ends
WebSockets Upgrade: websocket Real-time servers, long timeouts
API versions Path /v2/ or Accept-Version header Versioned services
HTML vs JSON Accept header SSR app versus API servers

Pitfalls

  • Relying on client-supplied headers for security (authorisation must happen in the service).

  • Rule ordering mistakes: a broad rule matches before a specific one.

  • Inconsistent caching when responses vary by Accept without Vary: Accept.

  • Too many complex rules, making config hard to reason about. Prefer clear path conventions.

Interview Tips

  • Explain L7 parsing and the signals: path, host, Content-Type, Accept, headers, method.

  • Give examples (static vs API, gRPC, uploads, WebSockets) with config.

  • Mention benefits (specialisation, scaling) and trade-offs (cost, complexity, spoofing).

32

How would you enable graceful shutdown of servers in a load-balanced environment?

Graceful shutdown means removing a server without failing any requests. It needs the load balancer and the application to cooperate: 1. Stop new traffic first: deregister the server or put it in drain mode at the load balancer, or have the app fail its readiness check, so the LB marks it out after the unhealthy threshold. 2. Wait for propagation: load balancers, kube-proxy and ingress controllers learn about the change asynchronously, so keep serving for a short period (Kubernetes preStop sleep, or about interval × fall threshold). 3. Finish in-flight work: the app handles SIGTERM by stopping its listener from accepting new connections, completing active requests, closing keep-alive connections (respond with Connection: close), telling WebSocket clients to reconnect, and finishing or requeueing background jobs. 4. Respect timeouts: the LB’s drain or deregistration delay and the platform’s termination grace period must be longer than the app’s shutdown window, and longer than typical request duration. 5. Release resources: flush logs and metrics, close database connections and commit offsets. 6. Exit, and only then terminate the instance or container.

Also use rolling deployments with max-unavailable limits, health-checked new instances before old ones go, and slow start for replacements.

Read full answer

Applies toKubernetes, AWS ELB, HAProxy, nginx, systemd services (Python 3.12 simulation)

Code Example: Readiness-Driven Shutdown

# Graceful shutdown: fail readiness, wait for the LB to notice, finish in-flight work, then exit
import signal, threading, time, urllib.request, urllib.error
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler

draining = threading.Event()
inflight = 0
lock = threading.Lock()

class Handler(BaseHTTPRequestHandler):
    def log_message(self, *args):
        pass
    def do_GET(self):
        global inflight
        if self.path == "/ready":                      # the LB health check
            code = 503 if draining.is_set() else 200
            self.send_response(code); self.end_headers()
            return
        with lock:
            inflight += 1
        time.sleep(0.5)                                # simulated slow request
        self.send_response(200); self.end_headers(); self.wfile.write(b"done")
        with lock:
            inflight -= 1

server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()

def shutdown(signum=None, frame=None):
    draining.set()                                     # 1. readiness now fails
    time.sleep(0.3)                                    # 2. wait for LB to stop sending (fall x interval)
    while True:                                        # 3. finish in-flight requests
        with lock:
            if inflight == 0:
                break
        time.sleep(0.05)
    server.shutdown()                                  # 4. stop accepting and exit
    print("shutdown complete, in-flight requests finished")

signal.signal(signal.SIGTERM, shutdown)               # Kubernetes and systemd send SIGTERM

def get(path):
    try:
        with urllib.request.urlopen(f"http://127.0.0.1:{port}{path}") as r:
            return r.status, r.read().decode()
    except urllib.error.HTTPError as e:
        return e.code, ""

results = []
slow = threading.Thread(target=lambda: results.append(get("/work")))
slow.start()
time.sleep(0.1)                                        # request is now in flight
print("ready before:", get("/ready")[0])               # 200
t = threading.Thread(target=shutdown)                  # same as receiving SIGTERM
t.start()
time.sleep(0.1)
print("ready while draining:", get("/ready")[0])       # 503
slow.join(); t.join()
print("in-flight request result:", results[0])         # (200, 'done')

The in-flight request started before shutdown completes successfully, and the readiness endpoint returns 503, so the load balancer stops sending new requests.

Kubernetes Setup

spec:
  terminationGracePeriodSeconds: 60          # longer than preStop + app drain
  containers:
    - name: api
      readinessProbe:
        httpGet: { path: /ready, port: 8080 }
        periodSeconds: 5
        failureThreshold: 1
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "15"]         # let endpoint removal propagate

Sequence: the pod is marked terminating → removed from endpoints (asynchronously) → preStop sleep → SIGTERM → app drains → exits, or receives SIGKILL after 60 s.

Server and Framework Settings

Server Graceful option
nginx nginx -s quit (SIGQUIT): workers finish current requests
HAProxy set server ... state drain; soft-stop on SIGUSR1 (with hard-stop-after)
Puma SIGTERM stops accepting and finishes requests (persistent_timeout, worker shutdown timeout)
Gunicorn SIGTERM with --graceful-timeout
Node.js server.close(), plus closing idle keep-alive connections (server.closeIdleConnections())
Go http.Server.Shutdown(ctx)
Spring Boot server.shutdown=graceful with spring.lifecycle.timeout-per-shutdown-phase

Timeout Alignment

Timer Example Rule
Health check detection (interval × fall) 5 s × 1 Or deregister explicitly
preStop delay 15 s ≥ endpoint propagation time
App drain timeout 30 s ≥ typical long request
LB deregistration delay 45 s ≥ preStop + app drain
Termination grace period 60 s > everything above

Pitfalls

  • App exits immediately on SIGTERM, so in-flight requests fail with 502.

  • SIGTERM arrives before the LB stops routing, so new requests hit a closing server: add the preStop delay.

  • PID 1 issues in containers: shell wrappers that don’t forward signals (use exec or an init such as tini).

  • Grace periods shorter than drain timeouts, so SIGKILL cuts work.

  • Long-lived connections never closed, so every shutdown hits the timeout.

Interview Tips

  • Give the ordered steps: stop new traffic, propagate, drain in-flight work, release resources, exit.

  • Show app signal handling and readiness failure.

  • Align timeouts across LB, platform and app; mention preStop hooks and PID 1 signal handling.

33

Discuss the role of API gateways in conjunction with load balancers.

An API gateway and a load balancer often sit together in the request path, but they do different jobs. A load balancer focuses on distributing traffic across healthy instances: L4 or L7, health checks, TLS termination, high availability and scaling. An API gateway is an API management layer (Kong, AWS API Gateway, Apigee, Azure API Management, Tyk, Envoy-based gateways) that handles cross-cutting API concerns: - authentication and authorisation (API keys, OAuth 2.0 and JWT validation); - rate limiting and quotas per client or plan; - request routing to microservices by path, method or version; - request and response transformation, protocol translation (REST to gRPC), and aggregation; - caching, API versioning, a developer portal, usage analytics and monetisation.

They combine in layers. A typical layout is: edge load balancer or CDN (TLS, DDoS and WAF, spreading load across gateway instances) → API gateway (auth, limits, routing) → internal load balancing to service instances, done by the gateway itself, Kubernetes Services or a mesh. The load balancer makes the gateway highly available and scalable, and the gateway turns many services into one coherent API. Some products blur the lines: an ALB can do path routing and OIDC authentication, and a gateway such as Kong or Envoy also load-balances upstreams. The decision depends on which features are needed and where.

Read full answer

Applies toKong, AWS API Gateway, Apigee, Azure API Management, Envoy Gateway, NGINX

Typical Architecture

clients -> CDN / WAF -> L4/L7 load balancer (TLS, HA)
        -> API gateway cluster (3+ instances): JWT validation, rate limits, routing, transforms
            |-- /orders/*   -> orders service (load-balanced across pods)
            |-- /payments/* -> payments service
            |-- /v1/legacy  -> legacy monolith

Example: Kong Declarative Config

_format_version: "3.0"
upstreams:
  - name: orders-upstream
    algorithm: least-connections
    targets:
      - target: orders-1.internal:8080
      - target: orders-2.internal:8080
    healthchecks:
      active:
        http_path: /ready
        healthy:   { interval: 5, successes: 2 }
        unhealthy: { interval: 5, http_failures: 3 }
services:
  - name: orders
    host: orders-upstream            # the gateway load-balances across the upstream's targets
    routes:
      - name: orders-route
        paths: ["/orders"]
    plugins:
      - name: jwt
      - name: rate-limiting
        config: { minute: 600, policy: local }

Responsibilities Compared

Concern Load balancer API gateway
Spreading traffic and health checks Primary job Often also does it for upstreams
TLS termination Yes Yes
Path and host routing L7 LBs: yes Yes, richer (methods, versions, rewrites)
AuthN/AuthZ (API keys, OAuth, JWT) Limited (ALB OIDC) Yes
Rate limiting and quotas per consumer Basic (per IP) Yes, per key or plan
Transformation and aggregation No Yes
Developer portal, analytics, monetisation No Yes
L4 / non-HTTP protocols Yes Usually no

When to Use What

Situation Choice
Single web app, few routes L7 load balancer is enough
Public API with partners and plans API gateway behind a load balancer
Many microservices, internal traffic Gateway at the edge; service mesh internally
Serverless back ends Managed API gateway (AWS API Gateway)

Pitfalls

  • Gateway as a single point of failure: run several instances behind a load balancer.

  • Business logic in the gateway, making it a bottleneck “monolith”.

  • Duplicated auth and limits at many layers, inconsistently configured.

  • Extra latency from too many hops (CDN → LB → gateway → mesh → service).

Interview Tips

  • Distinguish traffic distribution and availability (LB) from API management (gateway).

  • Draw the layered path and explain how the LB makes the gateway highly available.

  • Mention overlaps and avoiding single points of failure.

34

What is load balancer affinity, and when might you enforce it?

Load balancer affinity is a rule that binds a client, session or key to a specific back end, so related requests keep going to the same server instead of being freely distributed. Forms: - Session (client) affinity / sticky sessions: bound by a cookie (LB-generated or application cookie), or by source IP (ClientIP affinity in Kubernetes Services, source-IP hash). - Key-based affinity: hashing a header, user ID, tenant ID or URL (consistent hashing) to choose the server. - Connection affinity: inherent in long-lived connections (WebSockets, gRPC streams), where all messages on a connection go to one server. - Locality affinity: preferring servers in the same zone or region as the client or caller (topology-aware routing), which reduces latency and cost.

When to enforce it: - Server-local session state (legacy apps storing sessions in memory or on local disk). - Multi-step flows (checkout, file uploads in chunks, SAML or OAuth flows that keep state locally). - Cache locality: per-user or per-key data warm on one server (recommendation models, per-tenant caches). - Stateful real-time systems: game rooms, chat rooms or collaborative documents held in one server’s memory. - Protocol requirements: Socket.IO polling fallback, some NTLM or Kerberos setups that authenticate per connection. - Experiments: keeping a user in the same A/B or canary cohort.

Read full answer

Avoid affinity when services are stateless: it unbalances load, causes session loss on failover, slows scale-in and deployments, and IP affinity breaks behind NAT and proxies. Prefer externalised state, using affinity as a performance optimisation, not a correctness crutch.

Applies to: HAProxy, nginx, AWS ALB stickiness, Kubernetes Services and Ingress, Envoy

Configuration Examples

backend app
    balance roundrobin
    cookie SRV insert indirect nocache httponly secure    # LB-inserted affinity cookie
    option redispatch                                     # rebind if the server is down
    server app1 10.0.1.11:8080 check cookie a1
    server app2 10.0.1.12:8080 check cookie a2
# Kubernetes Service: source-IP affinity for 3 hours
apiVersion: v1
kind: Service
metadata:
  name: legacy-app
spec:
  selector: { app: legacy }
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP: { timeoutSeconds: 10800 }
  ports:
    - port: 80
      targetPort: 8080
# AWS ALB: duration-based (LB cookie) stickiness for one hour
aws elbv2 modify-target-group-attributes --target-group-arn "$TG" --attributes \
  Key=stickiness.enabled,Value=true Key=stickiness.type,Value=lb_cookie \
  Key=stickiness.lb_cookie.duration_seconds,Value=3600

Affinity Types

Type Key Strength Weakness
Cookie LB or app cookie Precise per browser HTTP only; cookies may be blocked or stripped
Source IP Client IP Works at L4, any protocol NAT/CGNAT clumping, mobile IP changes
Header or key hash User, tenant, URL Stable for APIs and caches Hot keys
Locality Zone or region Latency, cost Needs spillover when a zone is overloaded

Enforcement Checklist

  • Set an expiry that matches session lifetime.

  • Redispatch on failure, and accept that state on a failed server is lost.

  • Drain servers before removal, so pinned users finish.

  • Monitor per-server load for skew.

  • Plan to remove the need for affinity (Redis sessions, JWTs).

Pitfalls

  • IP affinity for mobile and corporate users.

  • Infinite stickiness, so new servers get no existing users after scaling out.

  • Treating affinity as reliability.

  • Cookie affinity behind caches that share cookies between users.

Interview Tips

  • Define affinity as binding a client or key to the same back end.

  • Explain the mechanisms: cookie, IP, hash, locality.

  • Give when to enforce (local state, flows, cache locality, real-time rooms) and its costs; recommend stateless designs.

35

How does load balancing work in containerized environments like Docker and Kubernetes?

In containerised environments, load balancing has to keep up with containers that start, stop and move constantly and get new IP addresses. Platforms solve this with service discovery plus virtual addresses. - Docker Compose: on a user-defined network, containers reach each other by service name, and Docker’s embedded DNS returns the IPs of all replicas. Spreading is basic (clients may pick the first IP), so you often add a proxy (nginx, Traefik, HAProxy). - Docker Swarm: each service gets a virtual IP (VIP), and IPVS spreads connections across tasks. The routing mesh publishes ports on every node and forwards to any task. - Kubernetes works in layers: - Service (ClusterIP): a stable virtual IP and DNS name (orders.prod.svc.cluster.local). kube-proxy (iptables, IPVS or nftables mode) or Cilium’s eBPF data plane spreads connections across ready pod endpoints, tracked by EndpointSlices. - Readiness probes decide whether a pod receives traffic. - NodePort and LoadBalancer Services expose Services outside the cluster; a cloud controller provisions an external LB. - Ingress or the Gateway API, implemented by controllers such as NGINX, Envoy Gateway, Traefik or cloud LB controllers, provide L7 routing (host, path, TLS, weighted canaries). - Service meshes (Istio, Linkerd) add per-request L7 balancing, retries, outlier detection and mTLS between services. - Headless Services (clusterIP: None) return pod IPs directly for client-side load balancing (gRPC, databases).

Key detail: kube-proxy balances per connection at L4, so long-lived HTTP/2 or gRPC connections can stick to a few pods. Use a mesh, an L7 ingress or client-side balancing for those.

Read full answer

Applies toDocker Compose and Swarm, Kubernetes 1.30+

Kubernetes Traffic Path

internet -> cloud LB (Service type LoadBalancer or Ingress/Gateway controller)
         -> ingress pods (L7: host/path rules, TLS)
         -> Service ClusterIP (kube-proxy / eBPF picks a ready pod per connection)
         -> pod (readinessProbe passed)

Example: Kubernetes Service and Gateway API Route

apiVersion: v1
kind: Service
metadata:
  name: orders
spec:
  selector: { app: orders }
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
spec:
  parentRefs: [ { name: public-gateway } ]
  hostnames: ["shop.example.com"]
  rules:
    - matches: [ { path: { type: PathPrefix, value: /orders } } ]
      backendRefs: [ { name: orders, port: 80 } ]
kubectl get endpointslices -l kubernetes.io/service-name=orders   # pods currently receiving traffic
kubectl scale deploy/orders --replicas=6                          # new pods join once ready

Example: Docker Compose With Replicas Behind nginx

services:
  api:
    image: example/api:1.4
    deploy:
      replicas: 3
  lb:
    image: nginx:1.27
    ports: ["8080:80"]
    volumes: ["./nginx.conf:/etc/nginx/conf.d/default.conf:ro"]

In nginx.conf, proxy_pass http://api:8080; resolves the service name through Docker DNS. nginx resolves it at startup, so if replicas change you need a resolver 127.0.0.11 with a variable in proxy_pass, or a reload.

Layers Compared

Layer Balances Level
Docker DNS Name to IPs DNS (client picks)
Swarm VIP / IPVS Connections to tasks L4
Kubernetes Service (kube-proxy, eBPF) Connections to ready pods L4
Ingress / Gateway controller HTTP requests by host, path, weight L7
Service mesh sidecar or ambient proxy Each request, with retries and outlier detection L7
Client-side (gRPC with headless Service) Each request or stream L7 in the client

Pitfalls

  • gRPC through ClusterIP: one connection per client pins traffic to one pod.

  • No readiness probes: pods receive traffic before they are ready.

  • Deployments dropping requests without preStop delays and graceful shutdown.

  • externalTrafficPolicy: Cluster losing client source IPs, or Local causing uneven spread.

Interview Tips

  • Explain service discovery + virtual IPs: Docker DNS, Swarm VIP, Kubernetes Services.

  • Walk through Service → kube-proxy → ready pods, plus Ingress/Gateway for L7.

  • Mention gRPC long-lived connection imbalance, headless Services and meshes.

36

What is a reverse proxy, and how does it relate to load balancing?

A reverse proxy is a server that sits in front of one or more back-end servers and receives client requests on their behalf. Clients connect to the proxy, which forwards each request to a back end and returns the response, so clients never talk to the origin directly. (A forward proxy is the opposite: it sits in front of clients and makes requests to the internet for them.) A reverse proxy commonly provides: - TLS termination and HTTP/2 or HTTP/3 to clients; - routing by host or path to different applications; - caching and compression; - security: hiding origin servers, WAF integration, rate limiting, request size limits, header sanitising; - buffering slow clients, so application workers aren’t tied up; - authentication offload, and logging and metrics.

Relationship: a load balancer distributes traffic across multiple servers. Most L7 load balancers are reverse proxies with a pool of back ends plus balancing algorithms and health checks: nginx, HAProxy, Envoy, Traefik, AWS ALB. A reverse proxy in front of one server is still useful (TLS, caching, security) but does no balancing. Conversely, some load balancers are not reverse proxies: L4 balancers (IPVS, AWS NLB in pass-through mode, DSR setups) forward packets or connections without terminating HTTP, and DNS-based balancing just returns addresses.

Read full answer

Applies tonginx, HAProxy, Envoy, Traefik, Caddy, Apache httpd (mod_proxy)

Diagram

Forward proxy:  [clients] -> forward proxy -> internet servers      (protects/controls clients)
Reverse proxy:  internet clients -> reverse proxy -> [your servers]  (protects/fronts servers)
Reverse proxy as LB:  clients -> nginx (TLS, cache) -> app1 | app2 | app3 (round robin + health)

Example: nginx as Reverse Proxy, Then as Load Balancer

# Reverse proxy to a single app
server {
    listen 443 ssl;
    server_name blog.example.com;
    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# Same proxy, now load-balancing across a pool
upstream shop_pool {
    least_conn;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    keepalive 64;                       # reuse back-end connections
}
server {
    listen 443 ssl;
    server_name shop.example.com;
    location / {
        proxy_pass http://shop_pool;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

Comparison

Aspect Reverse proxy Load balancer
Core purpose Front servers: termination, routing, caching, security Spread load across servers, availability
Number of back ends One or many Many (by definition)
Layer L7 (HTTP), sometimes L4 TCP proxying L4 or L7, or DNS
Terminates client connection Yes L7: yes; L4 pass-through or DSR: no
Health checks and algorithms Optional Central
Examples nginx, Caddy, Apache, Varnish HAProxy, nginx upstream, Envoy, ALB, NLB, IPVS, F5

Typical Stack

CDN (a globally distributed reverse proxy with caching)
-> cloud L4/L7 load balancer
-> ingress reverse proxy (nginx/Envoy) with routing + load balancing to pods

Pitfalls

  • Losing client information: set and trust X-Forwarded-* or the PROXY protocol only from your proxies.

  • Single reverse proxy in front of many servers: a single point of failure unless made redundant.

  • Buffering or timeouts that break streaming, WebSockets or large uploads if not tuned.

  • Caching personalised responses at the proxy.

Interview Tips

  • Define a reverse proxy as an intermediary in front of servers, and contrast it with a forward proxy.

  • Explain that L7 load balancers are reverse proxies with pools and algorithms.

  • Note that L4 and DNS load balancing are not reverse proxies, and list reverse proxy features (TLS, caching, security).

37

Explain how load balancing works with WebSockets and other persistent connection protocols.

WebSockets, Server-Sent Events, gRPC streams, MQTT and long polling keep a connection open for minutes to hours. Load balancing them differs from short HTTP requests: - The balancing decision happens once per connection: the WebSocket starts as an HTTP request with Upgrade: websocket. The load balancer picks a back end, the handshake completes (101 Switching Protocols), and all later messages stay on that server for the connection’s lifetime. Balance improves only as connections churn. - Use connection-aware algorithms: least connections suits long-lived connections far better than round robin, which can pile connections onto some servers over time. - Configure timeouts and upgrades: the proxy must forward the Upgrade and Connection headers and use long idle timeouts, plus ping/pong heartbeats (application or protocol level), so idle connections aren’t cut. Examples: nginx proxy_read_timeout, HAProxy timeout tunnel, ALB idle timeout. - Plan capacity per connection: each connection holds memory and file descriptors. Monitor concurrent connections per server, and tune OS limits and ephemeral ports. - Share state across servers: users in the same chat room may be on different servers, so broadcasts go through pub/sub (Redis, NATS, Kafka), or rooms are routed by consistent hashing to one server. - Use stickiness for fallbacks: libraries like Socket.IO with HTTP long-polling fallback need sticky sessions, because a session’s polling requests must hit the same server. - Make deployments and scaling graceful: draining can take hours, so servers send close or “reconnect” messages, and clients reconnect with jittered exponential backoff (avoiding reconnect storms) and land on new servers. New servers start empty, so rebalancing happens as clients reconnect.

Read full answer

Applies tonginx, HAProxy, Envoy, AWS ALB/NLB, Socket.IO, gRPC

nginx WebSocket Proxy

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}
upstream ws_backend {
    least_conn;
    server 10.0.1.11:9000;
    server 10.0.1.12:9000;
}
server {
    listen 443 ssl;
    location /ws/ {
        proxy_pass http://ws_backend;
        proxy_http_version 1.1;                         # required for Upgrade
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_read_timeout 1h;                          # default 60 s would cut idle sockets
        proxy_send_timeout 1h;
    }
}

HAProxy

backend websockets
    balance leastconn
    timeout tunnel 1h            # applies after the upgrade
    timeout server 30s
    server ws1 10.0.1.11:9000 check
    server ws2 10.0.1.12:9000 check

Client Reconnect With Backoff (JavaScript)

function connect(attempt = 0) {
  const ws = new WebSocket("wss://chat.example.com/ws/");
  ws.onopen = () => { attempt = 0; };
  ws.onclose = () => {
    const delay = Math.min(30000, 1000 * 2 ** attempt) * (0.5 + Math.random() / 2); // jitter
    setTimeout(() => connect(attempt + 1), delay);
  };
}
connect();

Protocol Notes

Protocol Notes for load balancing
WebSockets Upgrade headers, long timeouts, least connections, pub/sub fan-out
Server-Sent Events Plain long HTTP response; disable proxy buffering (proxy_buffering off)
gRPC streams / HTTP/2 Many streams per connection: use L7 per-request or per-stream balancing
MQTT L4 or MQTT-aware brokers; shared subscriptions; clustering
Long polling Behaves like repeated requests; stickiness if the server keeps per-client state

Scaling Problems and Fixes

Problem Fix
Old servers keep all connections after scale-out Periodic connection rebalancing (max connection age), reconnect hints
Deploys drop thousands of sockets at once Gradual draining, staggered close, client jitter
Cross-server messages Redis or NATS pub/sub, or hash rooms to servers
Idle disconnects by intermediaries Heartbeats under the idle timeout

Pitfalls

  • Round robin with long-lived connections, creating long-term imbalance.

  • Default 60 s proxy timeouts silently closing idle sockets.

  • Missing Connection: upgrade forwarding.

  • Reconnect storms without jitter after a deploy or outage.

Interview Tips

  • Explain that the decision is per connection, not per message.

  • Cover upgrade headers, timeouts, least connections, heartbeats, pub/sub and stickiness for fallbacks.

  • Discuss draining, reconnection with backoff, and rebalancing after scaling.

38

Discuss how machine learning can be applied to improve load balancing decisions.

Classic algorithms react to current, simple signals (connection counts, health). Machine learning can use richer data and prediction: - Latency and load prediction: models estimate each back end’s response time for a request from request features (endpoint, payload size, user tier) and server state (CPU, queue depth, recent latencies), then route to the lowest predicted latency or the best expected SLO compliance. - Online learning and bandits: multi-armed bandit or reinforcement learning policies try back ends, observe outcomes and adapt continuously, balancing exploration (keeping estimates fresh) against exploitation (sending traffic to the best server). EWMA-based “peak EWMA” balancing, as in Finagle and Linkerd, is a simple form. - Traffic forecasting for capacity: time-series models predict demand (daily cycles, events) to pre-scale instances before spikes, as predictive autoscaling does in AWS and Google Cloud. - Anomaly detection: detecting degraded or “grey failure” servers (slow but passing health checks) and abnormal traffic (DDoS, bots) faster than fixed thresholds. - Smarter global steering: choosing regions or CDN paths from measured network performance (real user monitoring). - Cost and energy-aware routing: balancing latency against cost, carbon intensity or spot-instance availability.

Challenges: - ML decisions must be fast (microseconds on the data path), so models are often trained offline with only lightweight scoring inline, or they adjust weights periodically rather than per request. - Feedback loops and herding: everyone chasing the “best” server overloads it, so add randomisation (power of two choices) and load terms. - Explainability, safety and fallback to simple algorithms are needed when models fail. - It needs good telemetry.

Read full answer

In practice, ML works best alongside robust baselines (least requests, P2C, outlier detection), not replacing them.

Applies to: adaptive load balancing, predictive autoscaling (Python 3.12 simulation)

Code Example: Online-Learned Latency Routing

# Learned latency model: route to the server with the lowest *predicted* latency (an EWMA
# updated online from every response), versus round robin. Server C silently becomes
# 4x slower halfway through, and B is always a little slower.
import random, statistics

random.seed(7)
BASE = {"A": 20.0, "B": 25.0, "C": 20.0}             # true mean latency (ms)

def true_latency(s, t):
    mean = BASE[s] * (4 if s == "C" and t > 2000 else 1)
    return random.expovariate(1 / mean)

def run(policy, n=4000, alpha=0.1, explore=0.05):
    est = {s: 20.0 for s in BASE}                     # learned estimate per server
    lat = []
    for t in range(n):
        if policy == "round_robin":
            s = "ABC"[t % 3]
        elif random.random() < explore:               # small exploration keeps estimates fresh
            s = random.choice("ABC")
        else:
            s = min(est, key=est.get)
        x = true_latency(s, t)
        est[s] = (1 - alpha) * est[s] + alpha * x     # online learning from each response
        lat.append(x)
    lat.sort()
    return round(statistics.mean(lat), 1), round(lat[int(0.99 * n)], 1)

print("round robin     (avg, p99 ms):", run("round_robin"))   # (31.3, 203.3)
print("learned latency (avg, p99 ms):", run("learned"))       # (21.8, 105.0)

The learned policy notices that C degraded and that B is slower, and shifts traffic away, cutting average and tail latency. Note that a real system must also model load: routing everything to the currently fastest server would overload it. That’s why production designs combine estimates with P2C and concurrency terms.

Where ML Fits

Area Technique Example
Per-request routing EWMA latency, bandits, learned cost models Linkerd EWMA, Finagle aperture + P2C
Weight tuning Periodic optimisation from metrics Adjust weights every 30 s from latency and CPU
Autoscaling Forecasting (seasonal models, gradient boosting, neural networks) AWS predictive scaling, GCP predictive autoscaling
Health Anomaly detection on latency and error patterns Ejecting grey-failure hosts
Security Bot and DDoS classification Edge bot management scores
Global routing Learned path performance CDN and anycast route selection

Design Principles

  • Keep the data path simple: score cheaply, or push weights from an offline or nearline controller.

  • Guardrails: cap weight changes, keep minimum traffic to every healthy server, and add automatic fallback to static algorithms.

  • Avoid herding: randomise between top candidates, and include queue length or in-flight requests.

  • Evaluate safely: shadow mode, canaries and A/B comparison against baseline p99 and error rates.

  • Monitor model drift as traffic patterns change.

Pitfalls

  • Oscillation: all traffic moves to one server, it slows, and traffic moves back.

  • Stale or noisy features causing bad decisions.

  • Complexity without measurable gains over P2C or least requests.

  • Opaque decisions that make incidents harder to debug.

Interview Tips

  • Mention latency prediction, bandits, forecasting for predictive scaling and anomaly detection.

  • Discuss real-time constraints, herding, guardrails and fallback.

  • Position ML as augmenting proven algorithms, validated by experiments.

39

What strategies might you employ to handle sudden spikes in traffic using load balancing?

Handling sudden spikes (“flash crowds”, launches, viral posts, retries after an outage) needs several strategies together: - Absorb at the edge: CDN caching of static and cacheable content, and micro-caching hot API responses for a few seconds at the load balancer, so most spike traffic never reaches the application. - Spread efficiently: use least outstanding requests or power of two choices, not round robin, so slow servers don’t pile up. Enable cross-zone balancing. Use anycast or GSLB to spread across regions. - Scale fast: autoscaling on request rate or latency with pre-warmed capacity or warm pools. Schedule scaling for known events, and use predictive scaling for recurring patterns. Managed LBs may need pre-warming for extreme, instant spikes. - Control admission: apply rate limiting (token buckets per client, plus a global limit), connection limits and queues with bounded size. Return 429 or 503 with Retry-After instead of letting everything time out. Use a virtual waiting room for ticket sales and launches. - Shed and degrade gracefully: prioritise critical traffic (checkout over recommendations), turn off expensive features with feature flags, and serve stale cache or static fallbacks. - Protect dependencies: circuit breakers, timeouts, retry budgets and bulkheads prevent the spike from cascading into databases and third parties. - Prepare: load-test to known limits, keep capacity headroom, and practise runbooks.

Read full answer

Applies toCDNs, nginx/HAProxy/Envoy, cloud LBs, autoscaling (Python 3.12 simulation)

Code Example: Token Bucket Admission Control

# Token bucket at the load balancer: absorb short bursts, shed sustained overload with 429s
class TokenBucket:
    def __init__(self, rate, burst):
        self.rate, self.capacity, self.tokens = rate, burst, burst

    def tick(self):                                   # called once per second
        self.tokens = min(self.capacity, self.tokens + self.rate)

    def take(self, n):                                # returns (accepted, rejected)
        ok = min(n, int(self.tokens))
        self.tokens -= ok
        return ok, n - ok

bucket = TokenBucket(rate=100, burst=300)             # 100 req/s sustained, bursts up to 300
traffic = [80, 90, 350, 120, 100, 400, 400, 400, 90, 80]
for sec, n in enumerate(traffic):
    ok, shed = bucket.take(n)
    print(f"t={sec}s offered={n:3d} served={ok:3d} rejected_429={shed:3d}")
    bucket.tick()
# t=2s offered=350 served=300 rejected_429= 50   (burst capacity absorbs most of the spike)
# t=5s offered=400 served=100 rejected_429=300   (sustained overload: only the steady rate is served)

Short bursts are absorbed by the bucket’s burst capacity. During a sustained overload, only the sustainable rate is admitted and the rest get fast 429 responses, instead of every request slowing down and timing out.

nginx: Rate Limits With Burst and Queueing Limits

limit_req_zone $binary_remote_addr zone=perclient:20m rate=10r/s;
limit_req_zone $server_name        zone=global:1m    rate=2000r/s;

upstream app {
    least_conn;
    server 10.0.1.11:8080 max_conns=200;     # cap concurrency per server
    server 10.0.1.12:8080 max_conns=200;
}
server {
    location /api/ {
        limit_req zone=perclient burst=20 nodelay;
        limit_req zone=global burst=500;
        limit_req_status 429;
        proxy_pass http://app;
    }
    location /api/catalog/ {
        proxy_cache api; proxy_cache_valid 200 5s;   # micro-cache hot reads
        proxy_cache_lock on;
        proxy_cache_use_stale updating error timeout;
        proxy_pass http://app;
    }
}

max_conns in open-source nginx applies per worker process unless the upstream has a zone. Add zone app 64k; so the limit is shared across workers.

Strategy Matrix

Spike type Best strategies
Predictable (sale, launch, TV ad) Scheduled scaling, pre-warming, waiting room, cache warming
Unpredictable viral traffic CDN caching, autoscaling, rate limits, graceful degradation
Retry storm after outage Retry budgets, jittered client backoff, Retry-After
Bot or attack surge WAF, bot management, per-client limits
Single hot endpoint Micro-caching, request coalescing, endpoint-specific limits

Autoscaling Tips

Tip Why
Scale on requests per target or latency CPU lags behind request surges
Warm pools or pre-initialised instances Boot time is often minutes
Lower scale-out cooldown, higher scale-in cooldown Fast up, slow down
Scale dependencies too (DB connections, caches) Front ends scaling alone moves the bottleneck

Pitfalls

  • Relying only on autoscaling, which is too slow for second-scale spikes.

  • Unbounded queues: requests wait until they time out anyway.

  • Retries at every layer, amplifying the spike.

  • Scaling app servers into a database connection limit.

Interview Tips

  • Layer the answer: cache, spread, scale, admit, shed, protect, prepare.

  • Show rate limiting with bursts and 429/Retry-After.

  • Mention waiting rooms, predictive and scheduled scaling and load testing.

40

How do load balancers contribute to a system’s security posture?

Load balancers sit at the entry point of a system, which makes them a natural security control layer: - Reduced attack surface: back-end servers live in private subnets with no public IPs. Only the load balancer is exposed, on specific ports (usually 443), and security groups allow back-end traffic only from the LB. - TLS enforcement: centralised TLS termination with modern protocols and ciphers, HTTP-to-HTTPS redirects, HSTS, automated certificate management, and optional re-encryption or mTLS to back ends. - DDoS resilience: absorbing SYN floods and connection exhaustion, connection limits and timeouts against slow attacks (Slowloris), integration with AWS Shield, Azure DDoS Protection, Cloud Armor and Cloudflare, and scaling capacity. - Application-layer filtering: WAF integration (AWS WAF, ModSecurity / Coraza, Cloud Armor rules) against injection, XSS and known exploits; bot management; geo and IP allow/deny lists. - Rate limiting against brute-force login attempts, scraping and API abuse. - Authentication offload: OIDC or JWT validation at the edge (ALB OIDC authentication, Envoy JWT filter), and mTLS client certificates for partner APIs. - Request hygiene: limits on header and body size, rejection of malformed HTTP (preventing some request smuggling), and removal of internal headers such as Server, X-Powered-By and debug headers. - Visibility: access logs and metrics feeding SIEM systems for detection and forensics. - Availability as security: health checks and failover keep services available, which is the “A” in the CIA triad.

The load balancer is itself a critical asset: it holds private keys, so harden it, patch it and restrict admin access. It is one layer in defence in depth, not a replacement for secure application code.

Read full answer

Applies toAWS ALB/NLB, Azure Application Gateway, GCP Load Balancing, nginx, HAProxy, Envoy, F5

Example: Hardened nginx Edge

server {
    listen 443 ssl;
    http2 on;
    server_name api.example.com;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;

    server_tokens off;                     # hide version
    client_max_body_size 1m;               # request size limit
    large_client_header_buffers 4 8k;
    client_header_timeout 10s;             # slow-header protection

    location = /login {
        limit_req zone=login burst=5 nodelay;   # brute-force protection (zone defined in http{})
        proxy_pass http://app;
    }
    location /admin/ {
        allow 10.0.0.0/8;                  # internal only
        deny all;
        proxy_pass http://admin_app;
    }
    location / {
        proxy_hide_header X-Powered-By;
        proxy_pass http://app;
    }
}

Example: AWS Security Group Pattern

# Back ends accept traffic only from the ALB's security group, not from the internet
aws ec2 authorize-security-group-ingress --group-id "$APP_SG" \
  --protocol tcp --port 8080 --source-group "$ALB_SG"

Security Functions

Function Threat addressed
Private back ends, single exposed port Direct server attacks, scanning
TLS termination and policies Eavesdropping, downgrade attacks
WAF rules SQL injection, XSS, known CVE exploits
Rate limits, bot management Credential stuffing, scraping, abuse
Connection limits and timeouts Slowloris, connection exhaustion
mTLS / OIDC at the edge Unauthenticated access
Header and size limits, strict HTTP parsing Smuggling, buffer abuse
Logging Detection, incident response

Hardening the Load Balancer Itself

  • Patch promptly (proxy CVEs such as HTTP/2 Rapid Reset, CVE-2023-44487).

  • Restrict management interfaces and APIs; use IAM least privilege.

  • Store keys in KMS or HSMs, or use managed certificates.

  • Monitor configuration changes, and keep config in version control.

Pitfalls

  • Origins still publicly reachable, bypassing the LB and WAF.

  • Trusting client-supplied X-Forwarded-For for IP-based controls.

  • WAF in detection mode forever, never blocking.

  • Assuming the LB fixes app vulnerabilities.

Interview Tips

  • Cover attack surface reduction, TLS, DDoS, WAF, rate limiting, auth offload, request hygiene and logging.

  • Stress defence in depth and hardening the LB itself.

  • Mention security groups that allow traffic only from the LB.

41

Can load balancers protect against SQL injection or XSS attacks? If so, how?

Partly, and only with a WAF. A plain load balancer distributes traffic and does not understand application attacks. An L7 load balancer integrated with a Web Application Firewall (WAF) can inspect HTTP requests (URL, query string, headers, cookies, body) and block or flag those that match attack patterns. This works through: - managed rule sets such as the OWASP Core Rule Set (CRS), AWS Managed Rules (AWSManagedRulesSQLiRuleSet, AWSManagedRulesCommonRuleSet), Cloud Armor preconfigured rules, and Azure’s default rule set; - normalisation before matching: URL decoding, HTML entity decoding, case folding and whitespace compression, to defeat encoding tricks; - anomaly scoring, where several weak signals add up to a block; - rate limiting and bot management, which slow automated injection scanners; - virtual patching to block a known exploit quickly while the code fix ships.

The limits are important. WAFs work on patterns, not application semantics: - they can be bypassed with novel encodings, JSON or GraphQL payloads, or stored XSS delivered through channels they don’t see; - they produce false positives that need tuning; - they cannot see DOM-based XSS in browser JavaScript, or second-order injection.

Read full answer

The real defences are in the application: - parameterised queries or ORMs for SQL; - context-aware output encoding and safe templating for XSS; - input validation and Content Security Policy.

A WAF at the load balancer is a valuable extra layer (defence in depth), not a substitute.

Applies to: AWS WAF on ALB, Azure Application Gateway WAF, Google Cloud Armor, ModSecurity/Coraza with nginx, HAProxy or Envoy (Python 3.12 illustration)

Code Example: Why Normalisation Matters

# Toy WAF rule check: why WAFs normalise (decode) input before matching signatures
import re
from urllib.parse import unquote_plus

RULES = {
    "sqli": re.compile(r"('|%27)\s*(or|and)\s+\d+\s*=\s*\d+|union\s+select|;\s*drop\s+table", re.I),
    "xss":  re.compile(r"<\s*script|on\w+\s*=|javascript:", re.I),
}

def inspect(raw, normalise=True):
    value = raw
    if normalise:
        for _ in range(3):                      # repeated decoding catches double-encoding
            decoded = unquote_plus(value)
            if decoded == value:
                break
            value = decoded
    return [name for name, rx in RULES.items() if rx.search(value)] or ["allow"]

requests = [
    "q=laptops",
    "id=1' OR 1=1--",
    "id=1%2527%2520OR%25201%253D1--",           # double URL-encoded SQL injection
    "name=<script>alert(1)</script>",
    "name=%3Cimg%20src%3Dx%20onerror%3Dalert(1)%3E",
    "comment=Use the select tool, then union them",  # benign text (no false positive here)
]
for r in requests:
    print(f"{r[:40]:42} raw={inspect(r, normalise=False)} normalised={inspect(r)}")
# id=1%2527%2520OR%25201%253D1--   raw=['allow'] normalised=['sqli']   (encoding bypasses naive rules)
# name=%3Cimg%20src%3Dx%20onerror%3D raw=['allow'] normalised=['xss']

A rule matching only raw text misses double-encoded payloads. Real WAFs apply transformation chains before matching, and are still bypassable with techniques the rules don’t model.

Example: AWS WAF Managed SQL Injection Rules on an ALB

aws wafv2 create-web-acl --name app-acl --scope REGIONAL --default-action Allow={} \
  --visibility-config SampledRequestsEnabled=true,CloudWatchMetricsEnabled=true,MetricName=appAcl \
  --rules '[{
    "Name": "sqli", "Priority": 1,
    "Statement": {"ManagedRuleGroupStatement": {"VendorName": "AWS", "Name": "AWSManagedRulesSQLiRuleSet"}},
    "OverrideAction": {"None": {}},
    "VisibilityConfig": {"SampledRequestsEnabled": true, "CloudWatchMetricsEnabled": true, "MetricName": "sqli"}
  }]'
aws wafv2 associate-web-acl --web-acl-arn "$ACL_ARN" --resource-arn "$ALB_ARN"

Application Fixes (the Real Defence)

# Parameterised SQL and escaped HTML output: the payloads become harmless data
import html, sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE users (email TEXT, name TEXT)")
db.execute("INSERT INTO users VALUES ('a@example.com', 'Asha')")

email = "x' OR 1=1--"                                       # injection attempt
rows = db.execute("SELECT name FROM users WHERE email = ?", (email,)).fetchall()
print("rows returned:", rows)                               # [] (treated as a literal string)

name = "<script>alert(1)</script>"
print("rendered:", f"<p>Hello, {html.escape(name)}</p>")    # <p>Hello, &lt;script&gt;...</p>

A Content-Security-Policy header such as default-src 'self'; script-src 'self' limits what injected scripts can do. The load balancer can add it to every response.

What a WAF at the Load Balancer Can and Cannot Do

Can Cannot
Block common SQLi/XSS payloads in requests Understand your queries or templates
Normalise encodings before matching Catch every novel bypass
Virtually patch known CVEs fast Stop DOM-based XSS in the browser
Rate-limit scanners and bots Fix stored or second-order injection reliably
Add security headers (CSP, HSTS) Replace secure coding and testing

Deployment Tips

  • Start in count or detection mode, review logs, tune exclusions, then switch to blocking.

  • Inspect request bodies (check size limits: unchecked oversize bodies may pass uninspected).

  • Keep rules updated, and alert on blocked-request spikes.

  • Make sure origins can’t be reached directly, bypassing the WAF.

Pitfalls

  • WAF as the only defence against injection.

  • Leaving it in detection mode forever.

  • Body inspection limits silently skipping large payloads.

  • False positives blocking legitimate users, such as code snippets in forums.

Interview Tips

  • Answer “yes, with a WAF, partially”: signatures, normalisation, managed rules.

  • Explain limits (bypasses, false positives, DOM XSS).

  • Stress parameterised queries, output encoding and CSP as the primary defences.

42

How does a Web Application Firewall (WAF) integrate with load balancing solutions?

A WAF inspects HTTP(S) traffic for attacks (injection, XSS, known exploits, bots, abuse), so it must sit where traffic is decrypted and parsed, which is usually at or next to the L7 load balancer. Integration models: - Built into the load balancer: the WAF is a feature attached to the LB. - AWS WAF web ACLs on ALB, CloudFront and API Gateway. - Azure Application Gateway WAF and Front Door WAF. - Google Cloud Armor security policies on the external Application Load Balancer. - F5 Advanced WAF on BIG-IP. - NGINX App Protect. - ModSecurity or Coraza modules in nginx, HAProxy (SPOE) and Envoy (proxy-wasm or ext_authz).

The LB terminates TLS, the WAF evaluates the request, then it is blocked (403 or custom page), challenged (CAPTCHA) or forwarded to the pool. - Edge or CDN WAF in front of the load balancer: Cloudflare, Akamai, Fastly and CloudFront with AWS WAF filter at the edge. The origin LB then accepts traffic only from the CDN, via IP allowlists, authenticated origin pulls (mTLS) or secret headers. - Inline appliance or sandwich: a separate WAF tier between two load balancers, or behind a Gateway Load Balancer (AWS GWLB with GENEVE), so WAF instances scale and fail over independently. - Sidecar or in-mesh WAF for east-west traffic in Kubernetes.

Read full answer

Operational integration includes: - shared TLS certificates; - health and failover of the WAF tier (fail-open versus fail-closed policy); - passing the real client IP (X-Forwarded-For, PROXY protocol) so IP rules work; - logging to SIEM; - rule tuning (count mode, then block).

Applies to: AWS WAF + ALB, Azure Application Gateway WAF, Cloud Armor, ModSecurity/Coraza, F5, Cloudflare

Architectures

1. Integrated:   client -> ALB [TLS terminate -> AWS WAF web ACL -> listener rules] -> targets
2. Edge WAF:     client -> CDN edge WAF -> origin LB (allows only CDN IPs) -> app
3. WAF tier:     client -> L4 LB -> WAF instances (autoscaled) -> L7 LB -> app
4. GWLB insert:  client -> GWLB endpoint -> third-party WAF/firewall fleet -> back to ALB -> app

Example: ModSecurity (OWASP CRS) in nginx as a Load Balancer

load_module modules/ngx_http_modsecurity_module.so;

upstream app { least_conn; server 10.0.1.11:8080; server 10.0.1.12:8080; }

server {
    listen 443 ssl;
    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;   # includes CRS + local exclusions
    location / {
        proxy_pass http://app;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}
# /etc/nginx/modsec/main.conf (excerpt)
Include /etc/nginx/modsec/modsecurity.conf      # SecRuleEngine DetectionOnly -> On after tuning
Include /usr/share/modsecurity-crs/crs-setup.conf
Include /usr/share/modsecurity-crs/rules/*.conf
SecRuleRemoveById 942100                         # example tuned exclusion for a false positive

ModSecurity is now an OWASP project. OWASP Coraza is a Go-based, CRS-compatible alternative that is often used with Envoy and Caddy.

Example: Cloud Armor Rule on a Google Cloud Load Balancer

gcloud compute security-policies create web-policy
gcloud compute security-policies rules create 1000 --security-policy web-policy \
  --expression "evaluatePreconfiguredWaf('sqli-v33-stable')" --action deny-403
gcloud compute backend-services update web-backend --global --security-policy web-policy

Integration Considerations

Concern Guidance
TLS WAF needs decrypted traffic: integrated LBs or re-encryption to a WAF tier
Client IP Preserve it for geo and IP rules (XFF, PROXY protocol)
Latency Rule evaluation adds latency; managed WAFs typically add a few ms or less
Availability Scale WAF tiers horizontally; decide fail-open or fail-closed
Bypass Lock origins to the LB or CDN only
Tuning Count mode, exclusions, then block; review regularly
Logging Ship WAF logs to SIEM; alert on spikes

Pitfalls

  • Origins reachable directly, bypassing the WAF.

  • Wrong client IP, so rate and IP rules hit the CDN or LB address.

  • Body inspection limits that let large payloads through uninspected.

  • Untuned rules that block real users, or that never move to blocking mode.

Interview Tips

  • Explain that a WAF needs decrypted HTTP, so it sits at the L7 load balancer or edge.

  • Describe integrated, edge, WAF-tier and GWLB models.

  • Cover client IP preservation, origin lockdown, tuning and availability.

43

Describe the importance of properly configuring HTTPS certificates on load balancers.

When a load balancer terminates TLS, its certificate configuration decides whether every user can connect securely. Getting it right matters for several reasons: - Trust and availability: browsers and API clients reject connections with an expired, untrusted, mismatched or incomplete-chain certificate. A single mistake therefore causes a site-wide outage and frightening warnings. Expired certificates are one of the most common causes of preventable outages. - Correct names: the certificate must cover every hostname served (SANs, wildcards). With many domains, SNI lets one listener select the right certificate per hostname. - Complete chain: serve the leaf plus intermediate certificates. Some browsers can fetch missing intermediates, but many clients cannot (mobile apps, IoT, Java and curl), so an incomplete chain causes failures for only some users. - Strong security: key size or type (RSA 2048+, or ECDSA P-256), a modern TLS policy (TLS 1.2 minimum, TLS 1.3 preferred, strong ciphers, forward secrecy), HSTS, and OCSP stapling for faster, more private revocation checks. - Key protection: the private key at the LB can decrypt traffic for that domain. Store it in managed certificate services, KMS or HSMs, limit access, and rotate on compromise. - Automation: automatic renewal (ACM, Let’s Encrypt via ACME, cert-manager, Azure Key Vault integration) plus expiry monitoring, since certificate lifetimes are getting shorter: the CA/Browser Forum has approved a phased reduction to 47 days by 2029. - Compliance: standards like PCI DSS require strong TLS and secure key handling. - Consistency across instances: every LB node, region and listener must serve the same valid certificate.

Read full answer

Applies toAWS ALB/NLB with ACM, Azure Application Gateway, Google Cloud LB, nginx, HAProxy, Kubernetes Ingress with cert-manager

Example: cert-manager on Kubernetes Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod       # automatic issuance and renewal
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["shop.example.com", "www.shop.example.com"]
      secretName: shop-tls
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend: { service: { name: shop, port: { number: 80 } } }

Example: HAProxy With SNI Certificates

frontend https
    bind :443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1 ssl-min-ver TLSv1.2
    # every PEM in the directory (cert + chain + key) is loaded and selected by SNI
    http-response set-header Strict-Transport-Security "max-age=31536000"

Verifying the Configuration

# Hostname, chain and expiry as clients see them
openssl s_client -connect shop.example.com:443 -servername shop.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -enddate -ext subjectAltName

# Days until expiry (alert below 21)
echo | openssl s_client -connect shop.example.com:443 -servername shop.example.com 2>/dev/null \
  | openssl x509 -noout -checkend $((21*24*3600)) && echo "OK" || echo "EXPIRES SOON"

Also check Verify return code: 0 (ok) in s_client output: a non-zero code often means a missing intermediate.

Common Misconfigurations

Mistake Effect
Expired certificate Hard failures for every client
Missing intermediate Works in some browsers, fails in apps, curl and older devices
Hostname not in SAN Name mismatch error
Wrong cert on one LB node or region Intermittent errors
Old protocols (TLS 1.0/1.1) enabled Compliance failures, weak security
Private key in plain files with broad access Key theft risk
No HTTP → HTTPS redirect Users on insecure connections
Cert pinned in apps, then rotated App connections break

Best Practices

  • Automate issuance and renewal; monitor expiry independently, such as synthetic checks from outside.

  • Use managed certificates where possible; least privilege on key access.

  • Apply a current TLS policy (for example AWS ELBSecurityPolicy-TLS13-1-2-2021-06).

  • Test with external scanners such as SSL Labs, plus openssl checks in CI.

  • Keep an inventory of certificates, owners and expiry dates.

Pitfalls

  • Manual renewals tracked in calendars.

  • Forgetting internal certificates (LB-to-back-end re-encryption).

  • Wildcard certificates shared widely, increasing the blast radius.

  • Not testing non-browser clients.

Interview Tips

  • Explain trust, availability, security and compliance impacts.

  • Cover SANs and SNI, full chains, TLS policy, HSTS, OCSP stapling and key protection.

  • Emphasise automation and expiry monitoring.

44

Discuss the security implications of SSL termination at the load balancer.

SSL/TLS termination at the load balancer means the LB decrypts client traffic, so traffic between the LB and back ends may travel unencrypted unless re-encrypted. The security implications cut both ways.

Benefits: - central, consistent TLS policy (TLS 1.2/1.3, strong ciphers, HSTS); - one place to manage certificates and renewals; - visibility for security controls, since a WAF, bot detection, rate limiting and request validation all need decrypted traffic; - reduced attack surface on back ends, which hold no public certificates.

Read full answer

Risks and considerations: - Plaintext in the internal network: anyone with access to that network segment (a compromised host, a misconfigured mirror port, a malicious insider) could read or alter traffic, including credentials, tokens and personal data. Zero-trust and many compliance regimes (PCI DSS, HIPAA guidance) expect encryption in transit everywhere, so re-encrypt (TLS bridging) or use mTLS / a service mesh to back ends. - The LB holds private keys: it becomes a high-value target. Protect keys with managed certificate services, KMS or HSMs, harden and patch the LB, and restrict admin access. - Trust in forwarded headers: back ends rely on X-Forwarded-For and X-Forwarded-Proto. If clients can reach back ends directly, or if headers aren’t overwritten, attackers can spoof IPs or trick apps into thinking a connection was HTTPS. - Back ends lose TLS context: client certificate details (for mTLS) must be passed securely in headers, and apps must build correct https:// URLs and secure cookies. - Logging exposure: decrypted requests may land in LB or WAF logs with sensitive data. Scrub or limit what is logged. - Concentration: a vulnerability in the TLS stack of the LB affects every service behind it, so patch quickly (Heartbleed in the past; HTTP/2 Rapid Reset).

The usual resolution: terminate at the LB for inspection, then re-encrypt to back ends (end-to-end encryption with an inspection point), or use TLS passthrough where no intermediary may decrypt.

Applies to: nginx, HAProxy, Envoy, AWS ALB, Azure Application Gateway, Google Cloud LB

Options

Termination only:     client ==TLS==> LB --HTTP--> app        (inspection; plaintext internally)
Re-encryption:        client ==TLS==> LB ==TLS/mTLS==> app    (inspection + encryption in transit)
Passthrough:          client ==========TLS==========> app     (no LB inspection; app holds keys)

Example: Envoy Re-encrypting With mTLS to the Back End

clusters:
  - name: payments
    type: STRICT_DNS
    load_assignment:
      cluster_name: payments
      endpoints:
        - lb_endpoints:
            - endpoint: { address: { socket_address: { address: payments.internal, port_value: 8443 } } }
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        sni: payments.internal
        common_tls_context:
          tls_certificates:                         # Envoy's client certificate (mTLS)
            - certificate_chain: { filename: /etc/envoy/client.crt }
              private_key: { filename: /etc/envoy/client.key }
          validation_context:
            trusted_ca: { filename: /etc/envoy/internal-ca.pem }   # verify the back end

Header Hygiene in nginx

proxy_set_header X-Forwarded-For   $remote_addr;   # overwrite, don't append client-supplied values
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Client-Cert     "";             # strip spoofed client-cert headers

$proxy_add_x_forwarded_for appends to whatever the client sent. That is fine behind trusted proxies, but at the outer edge, overwrite.

Implications Summary

Implication Mitigation
Plaintext LB-to-back-end traffic Re-encrypt, mTLS, service mesh, private network segmentation
Keys on the LB Managed certificates, KMS/HSM, least privilege, rotation
Header spoofing Overwrite headers at the edge; back ends accept only from the LB
Sensitive data in logs Redact; restrict log access
Single TLS stack for everything Fast patching, monitoring, defence in depth
Compliance requirements Document data flows; encrypt in transit end to end

Pitfalls

  • “It’s an internal network” as the reason for plaintext.

  • Back ends reachable directly, bypassing the LB’s controls.

  • Apps generating insecure redirects because they ignore X-Forwarded-Proto.

  • Long-lived, unmanaged private keys on LB hosts.

Interview Tips

  • Present benefits (central policy, inspection) and risks (plaintext hop, key concentration, header trust).

  • Explain re-encryption and passthrough as alternatives.

  • Mention compliance and zero-trust expectations.

45

Explain how rate limiting works in the context of load balancing.

Rate limiting caps how many requests (or connections) a client, key or route may make in a given time. Load balancers and gateways are a natural place to enforce it, because they see all traffic before it reaches back ends. How it works: 1. Identify the client or key: IP address (the real client IP, using X-Forwarded-For or the PROXY protocol from trusted proxies), API key, user ID from a JWT, tenant, path or a combination. 2. Count with an algorithm: - Fixed window: simple, but allows bursts at window edges. - Sliding window log or counter: smoother. - Token bucket: a steady refill rate plus burst capacity. - Leaky bucket: smooths the output rate by queueing. 3. Decide: allow, delay or queue (shaping), or reject with HTTP 429 Too Many Requests, ideally with Retry-After and rate-limit headers. 4. Store counters: - Locally per LB instance: fast, but the effective limit is the limit times the number of instances, and it is uneven. - Centrally: Redis, or a rate-limit service such as Envoy’s global rate limit service. Accurate, but adds latency and a dependency. - Hybrid: local plus periodic sync.

Purposes: protecting back ends from overload, fairness between tenants, abuse prevention (brute force, scraping, credential stuffing), cost control, and API product tiers. Rate limiting complements connection limits, concurrency limits, WAF and DDoS protection. Applications should still enforce business limits, such as login attempts per account.

Read full answer

Applies tonginx, HAProxy, Envoy, Kong, AWS WAF rate-based rules, API gateways (Python 3.12 simulation)

Code Example: Sliding-Window Counter

# Sliding-window counter rate limiter (as used by many gateways): smooths the edges of
# fixed windows by weighting the previous window's count
import math

class SlidingWindow:
    def __init__(self, limit, window=60):
        self.limit, self.window = limit, window
        self.counts = {}                          # (client, window index) -> count

    def allow(self, client, now):
        idx = math.floor(now / self.window)
        elapsed = (now % self.window) / self.window
        prev = self.counts.get((client, idx - 1), 0)
        cur = self.counts.get((client, idx), 0)
        estimate = prev * (1 - elapsed) + cur     # requests in the last `window` seconds
        if estimate >= self.limit:
            return False
        self.counts[(client, idx)] = cur + 1
        return True

rl = SlidingWindow(limit=100, window=60)
# 100 requests at t=59 s, then 100 more at t=61 s: a fixed window would allow all 200
first = sum(rl.allow("ip-1", 59.0) for _ in range(100))
second = sum(rl.allow("ip-1", 61.0) for _ in range(100))
later = sum(rl.allow("ip-1", 110.0) for _ in range(100))
other = sum(rl.allow("ip-2", 61.0) for _ in range(100))
print("t=59s allowed:", first)      # 100
print("t=61s allowed:", second)     # 2  (previous window still weighs 100 x 59/60 = 98.3)
print("t=110s allowed:", later)     # 82 (previous window now weighs 100 x 10/60 = 16.7; 2 already counted)
print("other client allowed:", other)  # 100 (limits are per client)

A fixed 60-second window would allow 200 requests in 2 seconds across the boundary. The sliding estimate prevents that, and each client has its own budget.

nginx

limit_req_zone $binary_remote_addr zone=ip:10m   rate=10r/s;    # per client IP
limit_req_zone $http_x_api_key     zone=key:10m  rate=50r/s;    # per API key

server {
    location /api/ {
        limit_req zone=ip  burst=20 nodelay;    # token-bucket-like burst allowance
        limit_req zone=key burst=100;
        limit_req_status 429;
        proxy_pass http://api;
    }
    location = /login {
        limit_req zone=ip burst=3;              # stricter on authentication
        proxy_pass http://api;
    }
}

Each nginx instance keeps its own counters (shared across workers through the zone), so with three instances a client may get up to 3× the limit.

HAProxy (Stick Tables)

frontend fe
    bind :443 ssl crt /etc/haproxy/certs/site.pem
    stick-table type ip size 1m expire 10m store http_req_rate(10s)
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }
    default_backend app

Stick tables can be synchronised between HAProxy peers for a shared view.

Algorithms Compared

Algorithm Behaviour Notes
Fixed window Count per minute bucket Simple; 2× bursts at boundaries
Sliding log Exact timestamps Accurate; memory-heavy
Sliding window counter Weighted previous + current window Good accuracy, cheap
Token bucket Refill rate + burst size Allows bursts, common in proxies
Leaky bucket Queue drained at a fixed rate Smooths output; adds latency
Concurrency limit Max in-flight requests Protects against slow requests better than rates

Design Choices

Choice Options
Key IP, API key, user, tenant, route, or combined
Scope Per instance, global (Redis or a rate-limit service), or edge (CDN, WAF)
Response 429 + Retry-After, delay, CAPTCHA challenge
Tiers Different limits per plan or endpoint

Pitfalls

  • Limiting by the proxy’s IP instead of the real client IP.

  • NAT and corporate users sharing one IP and hitting limits together.

  • Per-instance counters giving inconsistent effective limits.

  • No Retry-After, so clients retry immediately and make things worse.

  • Central store outages: decide whether to fail open or closed.

Interview Tips

  • Explain identify, count, decide, store.

  • Compare fixed window, sliding window, token and leaky bucket.

  • Discuss local versus global counters, real client IPs and 429 with Retry-After.

46

How would you troubleshoot a situation where a load balancer is incorrectly routing traffic?

“Incorrect routing” can mean requests reaching the wrong back end or pool, the wrong version or region, uneven distribution, broken stickiness, or traffic still going to removed servers. Troubleshoot systematically from the client inward: 1. Define the symptom precisely: which hosts, paths, headers or clients are affected, what was expected and what actually happened, and since when. Correlate with recent changes (deployments, config, DNS, certificates, autoscaling events). Most routing bugs follow a change. 2. Check that traffic reaches the intended load balancer: DNS (dig; stale records, TTL caching, split-horizon DNS), CDN configuration, and whether clients hit the correct listener, port and IP. 3. Inspect the routing rules: - rule order and priority (a broad rule matching first); - host or path match types (prefix versus exact, trailing slashes, case); - header conditions, regexes, default actions; - SNI and certificate selection for HTTPS. 4. Check the target pools: registered targets, health status, ports and weights (a canary left at 50%), and stale endpoints (Kubernetes EndpointSlices, service discovery lag). 5. Check stickiness and hashing: affinity cookies pinning users to old servers, hash keys that changed, consistent-hash rings after scaling. 6. Reproduce with controlled requests: curl with explicit Host, headers and --resolve. Have back ends return an identifying header (server ID, version). 7. Read the LB logs: which rule matched, which target was chosen, the response code. Use tracing across hops. 8. Fix carefully: correct the rule or pool, roll back recent changes if needed, then verify with tests and metrics. Add config tests and change reviews to prevent recurrence.

Read full answer

Applies tonginx, HAProxy, Envoy/Istio, AWS ALB, Kubernetes Ingress and Gateway API

Useful Commands

# Which IPs do clients resolve? Compare resolvers.
dig +short shop.example.com @1.1.1.1 ; dig +short shop.example.com @8.8.8.8

# Bypass DNS and send a request to a specific LB IP with the right Host/SNI
curl -sv --resolve shop.example.com:443:203.0.113.10 https://shop.example.com/api/orders \
  -H "X-Debug: 1" -o /dev/null 2>&1 | grep -iE "< HTTP|< x-served-by|< x-version"

# Repeat to see the distribution across back ends
for i in $(seq 1 20); do curl -s -o /dev/null -w "%header{x-served-by}\n" https://shop.example.com/; done | sort | uniq -c

%header{...} in -w needs curl 7.84+.

Platform-Specific Checks

Platform What to inspect
nginx nginx -T (full effective config), location match order (exact =, ^~, regex, prefix), $upstream_addr in logs
HAProxy use_backend order and ACLs, show servers state, logs with %b/%s (backend/server)
Envoy / Istio istioctl proxy-config routes/clusters/endpoints <pod>, istioctl analyze, admin /config_dump
AWS ALB Listener rule priorities and conditions, target group health, access logs (target:port, matched_rule_priority)
Kubernetes kubectl describe ingress/httproute, kubectl get endpointslices, Service selectors and labels

Common Causes

Symptom Likely cause
Requests hit the default backend Host header mismatch, rule order, path type (Exact vs Prefix)
Traffic to old version Canary weight not reset, stale endpoints, sticky cookies
One server gets most traffic Hash on a low-cardinality key, IP affinity behind NAT, HTTP/2 long-lived connections at L4
Some users routed wrongly Split DNS, CDN cache keyed incorrectly, geo rules
Traffic to removed servers Long TTLs, deregistration not done, service discovery lag
Wrong certificate or 421 errors SNI and Host mismatch, connection coalescing in HTTP/2

A Kubernetes Service selector that matches pods from two Deployments (for example app: web on both v1 and v2) sends traffic to both; check labels carefully.

Pitfalls

  • Testing from a machine with cached DNS or hosts-file overrides.

  • Changing several things at once, losing track of the cause.

  • Ignoring rule precedence semantics, which differ by product.

  • No back-end identifying headers, making routing invisible.

Interview Tips

  • Use a systematic path: symptom → recent changes → DNS → listener → rules → pools/health → stickiness → logs.

  • Show concrete tools (curl --resolve, nginx -T, istioctl proxy-config, ALB logs).

  • Mention prevention: config tests, reviews, identifying headers, gradual rollouts.

47

What would you do if a load balancer marks a healthy backend as down?

If the load balancer marks a back end down while the server seems to be running, the health check is failing from the load balancer’s point of view. Reproduce the check exactly as the LB performs it, and work through the possible causes: 1. Read the LB’s reason: health-check status and reason codes (ALB Target.ResponseCodeMismatch, Target.Timeout, Target.FailedHealthChecks), HAProxy logs (L4TOUT, L7STS, L4CON), Envoy /clusters health flags. 2. Run the same check from the LB’s network: same IP, port, path, Host header, protocol and timeout: curl -v http://10.0.1.12:8080/ready from an LB host or a debug pod in the same subnet. “Running” on localhost proves little. 3. Check network reachability: security groups, NACLs and firewalls (allow health checks from the LB’s subnets or security group), NetworkPolicies in Kubernetes, routes, and whether the app listens on 0.0.0.0 rather than 127.0.0.1. 4. Check the health check configuration: - wrong port or path (/health versus /healthz); - expected status (the app returns 301 to HTTPS or 401, but the check expects 200); - a Host header the app rejects; - HTTP versus HTTPS mismatch; - timeouts shorter than the endpoint’s response time; - thresholds too strict. 5. Check the health endpoint’s logic: it may fail because of a dependency (database, cache) that is slow or down, or because readiness is deliberately failing (warm-up, draining, maintenance flag). 6. Check resources and timing: CPU throttling, GC pauses, thread or connection pool exhaustion making the endpoint slow under load, and new instances still in grace or start-up periods. 7. Fix the root cause (config, network rules, endpoint logic or capacity), watch the server pass the healthy threshold, and add monitoring and alerts on health-check failures.

Read full answer

Applies toAWS ALB/NLB, HAProxy, nginx, Envoy, Kubernetes readiness probes

Diagnostic Commands

# AWS: why is the target unhealthy?
aws elbv2 describe-target-health --target-group-arn "$TG" \
  --query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State,TargetHealth.Reason,TargetHealth.Description]'

# From a host in the LB's subnet: replicate the probe exactly
curl -sv -m 5 -H "Host: app.internal" http://10.0.1.12:8080/ready

# On the server: is it listening on the right interface and port?
ss -ltnp | grep 8080          # 127.0.0.1:8080 means unreachable from the LB

# Kubernetes: readiness probe failures
kubectl describe pod api-7d9f -n prod | sed -n '/Readiness/,+3p;/Events/,$p'

Common Causes

Cause Clue Fix
Firewall or security group blocks LB Target.Timeout, L4TOUT Allow the LB’s SG or subnet on the health port
App bound to 127.0.0.1 Works locally only Bind to 0.0.0.0
Wrong path or port 404 or connection refused Fix health check config
HTTP→HTTPS redirect on the health path 301/302 mismatch Exempt health path or accept 200–399
Auth required 401/403 Make health path unauthenticated (restricted by network)
Host-based routing in the app 404 or 400 for unknown Host Set the Host header in the check
Slow endpoint (deep dependency checks) Timeouts under load Shallow readiness checks, longer timeout
Grace period too short Fails only at start-up Increase grace period or initial delay

Prevention

  • Keep health endpoints cheap, unauthenticated (but network-restricted) and fast.

  • Document exact health check settings with the service, and test them in CI or staging.

  • Alert on unhealthy host count, and include reason codes in dashboards.

Pitfalls

  • Declaring the server “fine” because curl localhost works.

  • Lowering thresholds or disabling checks to make alarms go away.

  • Deep dependency checks taking all servers out when a shared database blips.

  • Fixing one instance manually when the cause is shared configuration.

Interview Tips

  • Emphasise reproducing the check from the LB’s perspective.

  • Walk through network, config, endpoint logic and resources.

  • Mention reason codes, tooling and prevention.

48

Describe how you might identify and resolve performance bottlenecks in a load-balanced system.

Performance work in a load-balanced system means finding which hop or resource limits throughput or latency, then fixing that, re-measuring and repeating. Approach: 1. Define the problem and baseline: which SLO is violated (p99 latency, throughput, errors), on which endpoints and when. Compare against normal behaviour. 2. Break latency down per hop: client → CDN → LB → back end → dependencies. Use LB metrics (LB processing time versus target response time), distributed tracing (OpenTelemetry) and timing fields in logs ($request_time versus $upstream_response_time). 3. Check each layer with the USE method (utilisation, saturation, errors): - Load balancer: CPU (TLS), connections, queues, bandwidth, capacity units, connection limits, ephemeral ports. - Distribution: per-server request counts and latency. Look for imbalance caused by stickiness, hashing, long-lived connections or poor algorithms. - Back ends: CPU, memory, GC, thread and worker pools, connection pools. - Dependencies: database (slow queries, locks, connection limits), caches (hit rate), external APIs. - Network: cross-zone hops, packet loss, MTU, DNS lookup times. 4. Look for queueing: latency rising sharply as utilisation nears 100% is the classic sign of a saturated stage. 5. Form a hypothesis, test it with load tests or targeted experiments, and fix: - scale out the saturated tier; - switch algorithms (least requests or P2C); - enable keep-alive and connection reuse; - cache; - tune pools and timeouts; - optimise queries; - fix imbalance (remove IP affinity, use L7 balancing for gRPC). 6. Verify and prevent: re-measure, add dashboards and alerts, and load-test regularly.

Read full answer

Applies toany load-balanced architecture (nginx, HAProxy, Envoy, cloud LBs) (Python 3.12 log analysis)

Code Example: Locating Latency From Proxy Logs

# Split total request time into "upstream" (back end) and "proxy/client" time from nginx-style
# logs, per upstream server, to see where latency comes from
import re, statistics
from collections import defaultdict

LOG = """\
10.1.1.5 "GET /api/items" 200 rt=0.212 uct=0.001 urt=0.205 ua="10.0.1.11:8080"
10.1.1.6 "GET /api/items" 200 rt=0.198 uct=0.001 urt=0.190 ua="10.0.1.12:8080"
10.1.1.7 "GET /api/items" 200 rt=1.402 uct=0.001 urt=1.380 ua="10.0.1.13:8080"
10.1.1.8 "POST /upload" 200 rt=3.950 uct=0.002 urt=0.310 ua="10.0.1.11:8080"
10.1.1.9 "GET /api/items" 200 rt=1.250 uct=0.002 urt=1.231 ua="10.0.1.13:8080"
10.1.1.5 "GET /api/items" 502 rt=1.003 uct=1.000 urt=1.000 ua="10.0.1.14:8080"
10.1.1.6 "GET /api/items" 200 rt=0.220 uct=0.001 urt=0.211 ua="10.0.1.12:8080"
"""
pat = re.compile(r'"(\S+) (\S+)" (\d+) rt=([\d.]+) uct=([\d.]+) urt=([\d.]+) ua="([^"]+)"')
by_server = defaultdict(list)
for line in LOG.splitlines():
    m = pat.search(line)
    method, path, status, rt, uct, urt, ua = m.groups()
    rt, uct, urt = float(rt), float(uct), float(urt)
    by_server[ua].append((int(status), rt, uct, urt))
    if rt - urt > 1.0:
        print(f"slow outside the back end: {method} {path} total={rt}s upstream={urt}s")
    if uct >= 1.0:
        print(f"connect timeout to {ua}: check back-end health, security groups, listen backlog")

for ua, rows in sorted(by_server.items()):
    med = statistics.median(r[3] for r in rows)
    errors = sum(r[0] >= 500 for r in rows)
    print(f"{ua}: requests={len(rows)} median_upstream={med:.3f}s errors={errors}")
# slow outside the back end: POST /upload total=3.95s upstream=0.31s   (slow client upload)
# connect timeout to 10.0.1.14:8080: ...                                (dead or unreachable)
# 10.0.1.13:8080: requests=2 median_upstream=1.305s errors=0           (one slow back end)

Three different bottlenecks appear in a few lines: - a slow client upload: time outside the back end, so consider buffering, upload limits or direct-to-storage uploads; - an unreachable back end: connect timeouts; - one slow server: an outlier among identical instances, so check that host, or eject it with outlier detection.

Signals and Likely Bottlenecks

Signal Likely bottleneck Fix
LB time high, target time normal LB itself (TLS CPU, limits) or slow clients Scale or upgrade the LB, session resumption, buffering
One server slow, others fine Bad host, noisy neighbour, imbalance Outlier detection, replace host, fix affinity
All servers slow, CPU high Back-end capacity Scale out, optimise code
All servers slow, CPU low Waiting on dependencies or pools Tune pools, fix slow queries, add caching
Latency jumps at a traffic threshold Queueing at a saturated stage Add capacity at that stage
Many new connections per second No keep-alive to back ends Enable upstream keep-alive
502/504 under load Timeouts, worker exhaustion Right-size workers and timeouts, add capacity

Example: Enable Upstream Keep-Alive in nginx

upstream app {
    least_conn;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
    keepalive 128;                    # pool of idle connections per worker
}
location / {
    proxy_pass http://app;
    proxy_http_version 1.1;
    proxy_set_header Connection "";   # don't send "close"
}

Without this, nginx opens a new TCP connection per request to back ends, adding latency and risking ephemeral port exhaustion.

Load Testing

# k6: ramp to find the knee of the latency curve
k6 run --vus 50 --duration 5m script.js
# Watch p95/p99 and per-tier utilisation together during the test

Pitfalls

  • Averages hiding tail latency.

  • Scaling the wrong tier (adding app servers when the database is saturated).

  • Testing with unrealistic traffic (no TLS, no keep-alive, uniform requests).

  • Fixing symptoms (longer timeouts) instead of causes.

Interview Tips

  • Describe per-hop latency breakdown, the USE method, tracing and queueing signs.

  • Give concrete bottleneck patterns and fixes (imbalance, keep-alive, pools, caching).

  • Emphasise hypothesis → test → fix → verify.

49

How would you handle a situation where the load balancer becomes the performance bottleneck?

First confirm that the load balancer really is the bottleneck: LB processing time rises while back-end response times stay normal; LB CPU (often TLS), connection counts, bandwidth or queues are saturated; you see rejected connections, surge-queue overflows or 5xx errors generated by the LB itself; or you hit limits such as file descriptors, conntrack tables, ephemeral ports or capacity units. Then address it at several levels: - Tune the existing balancer: - worker processes and threads matching cores; - higher file descriptor and connection limits; - keep-alive to clients and upstream connection reuse; - TLS efficiency: session resumption, TLS 1.3, ECDSA certificates (cheaper than RSA), hardware acceleration; - avoiding expensive features on hot paths, such as complex regexes, large Lua scripts, or logging synchronously to slow disks. - Scale up: larger instances with more cores and network bandwidth. - Scale out the LB tier: run several LB instances behind DNS round robin, anycast, ECMP routing, or an L4 balancer in front of L7 proxies (the “L4 then L7” pattern: IPVS, Katran or NLB spreading traffic across HAProxy, Envoy or nginx). Managed LBs scale automatically, but may need pre-warming for sudden very large spikes, or a switch to a higher-capacity type such as NLB for raw TCP. - Offload work: move caching and static content to a CDN, let the CDN or edge terminate TLS, and serve large downloads from object storage. Use DSR for response-heavy L4 traffic. - Reduce work per request: fewer rewrites, simpler rule sets, compression at the right place, and HTTP/2 or HTTP/3 multiplexing to cut connection overhead. - Split by function: separate LBs for different services or traffic types (API, WebSockets, static), or client-side load balancing and service meshes for east-west traffic, so a central LB isn’t in every internal call.

Afterwards, load-test the new design, and monitor LB saturation metrics with alerts before limits are hit.

Read full answer

Applies tonginx, HAProxy, Envoy, Linux kernel tuning, AWS ALB/NLB, CDNs

Confirming the Bottleneck

Evidence Points to
ALB TargetResponseTime flat, client latency up LB or network path
HAProxy queue (qcur) or scur near maxconn Connection limits
CPU at 100% in TLS functions (perf top) TLS handshakes
nf_conntrack: table full in kernel logs Connection tracking limit
connect() failed (99: Cannot assign requested address) Ephemeral port exhaustion to back ends
Bandwidth at NIC or instance limit Network capacity

Example Tuning (Self-Managed Linux + nginx)

worker_processes auto;
worker_rlimit_nofile 200000;
events { worker_connections 65535; }

http {
    ssl_session_cache shared:SSL:50m;      # resumption: fewer full handshakes
    ssl_session_timeout 1h;
    keepalive_timeout 65s;
    access_log /var/log/nginx/access.log main buffer=64k flush=5s;   # buffered logging
    upstream app { server 10.0.1.11:8080; server 10.0.1.12:8080; keepalive 256; }
}
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
sysctl -w net.netfilter.nf_conntrack_max=1048576

Scaling the LB Tier

anycast / DNS / ECMP
                 |
   L4 tier (NLB, IPVS, Katran: stateless, very fast)
     |            |            |
  L7 proxy-1   L7 proxy-2   L7 proxy-3   (TLS, routing; autoscaled)
     \____________|____________/
               app servers

Options Compared

Option Effort Gain
Tuning (workers, keep-alive, TLS resumption) Low Often large
Bigger instance Low Linear, limited
CDN offload Medium Removes most static traffic
L4 + L7 tiered scale-out Medium–high Near-linear scaling
Client-side LB / mesh for internal traffic High Removes central hop
DSR Medium Big for response-heavy traffic

Pitfalls

  • Assuming the LB is the problem without comparing LB and back-end timings.

  • Single LB instance with no horizontal path.

  • Synchronous logging or heavy scripting on every request.

  • Forgetting kernel limits (conntrack, ports, file descriptors).

Interview Tips

  • Start with confirmation via metrics.

  • Then give tune, scale up, scale out (L4 → L7 tiers, anycast), offload (CDN, DSR) and reduce work.

  • Finish with load testing and saturation alerts.

50

Suppose a load balancer is causing increased latency—what potential root causes would you investigate?

When latency rises at the load balancer, compare total latency with back-end response time. The gap is time spent in or around the LB. Root causes to investigate: - TLS overhead: full handshakes for every connection (no session resumption, keep-alive disabled), RSA 4096 keys, OCSP fetching without stapling, CPU saturation from handshakes. - Connection handling: no upstream keep-alive (a new TCP plus TLS connection to the back end per request), connection pool exhaustion, ephemeral port or conntrack exhaustion, low maxconn causing queueing. - Queueing and saturation: LB CPU, memory or bandwidth limits; worker or thread limits; managed LB scaling lag during spikes. - Algorithm and distribution: requests sent to slow or overloaded servers (round robin ignoring load, sticky sessions, hash hot spots, HTTP/2 or gRPC at L4), slow-start missing for cold instances, or health checks too slow to remove a degraded server. - Retries and timeouts: automatic retries hiding failures but adding latency; long connect timeouts to dead servers. - Buffering behaviour: request or response buffering adding delay for streaming (SSE, gRPC), or disabled buffering making slow clients hold back-end connections. - Expensive processing: complex regex rules, WAF inspection of large bodies, Lua or Wasm filters, compression at high levels, header rewriting, or synchronous logging to slow storage. - DNS issues: the LB resolving back-end names on every request, or slow DNS resolvers. - Network path: cross-zone or cross-region hops, MTU or fragmentation issues, packet loss, a geo-DNS mistake sending users to a distant region, or Nagle’s algorithm and delayed ACK interactions. - Recent changes: new rules, version upgrades, certificate swaps, configuration reloads.

Investigate with LB metrics, logs that split times (connect, TLS, upstream, total), tracing, and packet captures if needed. Then fix the specific cause and re-measure.

Read full answer

Applies tonginx, HAProxy, Envoy, AWS ALB/NLB, Azure and GCP load balancers

Breaking Down Where the Time Goes

# Client-side timing breakdown: DNS, TCP connect, TLS, time to first byte
curl -s -o /dev/null https://shop.example.com/api/items -w \
'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n'
# LB-side: total vs upstream connect/header/response times
log_format t '$request rt=$request_time uct=$upstream_connect_time '
             'uht=$upstream_header_time urt=$upstream_response_time ua=$upstream_addr cs=$upstream_cache_status';
# HAProxy timers: TR (request), Tw (queue), Tc (connect), Tr (server response), Ta (total)
log-format "%ci %b/%s %ST TR=%TR Tw=%Tw Tc=%Tc Tr=%Tr Ta=%Ta"

A high Tw in HAProxy means requests are queued waiting for a server slot. A high Tc means slow back-end connects. A high Tr means the back end is slow, not the LB.

Root Cause Checklist

Area Check Typical fix
TLS Handshake rate, resumption ratio, CPU Session cache/tickets, TLS 1.3, ECDSA certs, OCSP stapling
Upstream connections New connections per request keepalive pools, HTTP/1.1 or HTTP/2 upstream
Queues Queue time, maxconn Raise limits, add back ends, scale LB
Distribution Per-server latency spread Least requests or P2C, outlier detection, remove IP affinity
Retries Retry counters Retry budgets, fewer layers retrying
Processing Rule complexity, WAF body inspection Simplify rules, limit inspected size, optimise filters
Logging Disk I/O wait Buffered or async logging, ship via syslog
DNS Resolution per request Cache with resolver valid=, static upstreams
Network Cross-zone traffic, RTT, packet loss Zone-aware routing, fix MTU, check NICs
Capacity LB CPU, bandwidth, LCUs Scale up or out, pre-warm, CDN offload

Pitfalls

  • Blaming the LB without separating LB time from back-end time.

  • Increasing timeouts, which hides queueing and makes latency worse.

  • Ignoring the client side: slow clients inflate $request_time without any LB problem.

  • Only looking at averages, missing tail-latency causes.

Interview Tips

  • Start by separating LB time from back-end time.

  • Cover TLS, connection reuse, queueing, distribution, retries, processing, logging, DNS and network.

  • Show timing tools (curl -w, nginx and HAProxy timers) and specific fixes.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 50 Load Balancing answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All System Design subjects