Define load balancing in the context of modern web services.
KnownRevise
Load balancing is the practice of distributing incoming requests or connections across a pool of servers (instances, containers, pods) so that no single server is overwhelmed, the service stays available when servers fail, and it can scale horizontally by adding more servers behind a single entry point. A load balancer sits between clients and back ends, exposing one address (a virtual IP or DNS name). It: - chooses a back end for each request or connection using an algorithm (round robin, least connections, hashing); - health-checks back ends and stops sending traffic to unhealthy ones.
In modern web services, load balancing happens at several layers: - DNS and global load balancing across regions; - edge and CDN load balancing; - Layer 4 (TCP/UDP) load balancers; - Layer 7 (HTTP) load balancers and reverse proxies, which also handle TLS termination, routing by path or host, rate limiting and WAF rules; - inside clusters: Kubernetes Services, service meshes and client-side balancing in gRPC.
Open-source nginx marks servers down passively (after failed requests); active health checks need NGINX Plus, HAProxy, Envoy or a cloud load balancer.
What It Enables
Goal
How load balancing helps
Scalability
Add servers behind the same address
Availability
Route around failed instances and zones
Performance
Spread load; send users to nearer or less-busy servers
Zero-downtime deployments
Drain old instances, shift traffic gradually (canary, blue-green)
Security
Single hardened entry point: TLS, WAF, rate limits, DDoS absorption
Pitfalls
The load balancer as a single point of failure: run it redundantly (managed services, active-passive pairs, anycast).
Losing client information: forward X-Forwarded-For or use the PROXY protocol.
Stateful servers that break when requests move between instances.
Health checks that are too shallow or too deep.
Interview Tips
Define it as distributing traffic across servers for scalability, availability and performance.
Contrast L4 and L7, and name the layers (DNS/global, edge, cluster, mesh).
Mention health checks, algorithms and redundancy for the load balancer itself.
2
What are the primary objectives of implementing load balancing?
KnownRevise
The primary objectives of load balancing are: 1. High availability and fault tolerance: detect failed or unhealthy servers with health checks and route around them, across instances, zones and even regions, so users see no outage. 2. Scalability: serve more traffic by adding servers horizontally behind one stable endpoint, often automatically with autoscaling. 3. Performance and efficient resource use: spread work so no server is overloaded while others sit idle, keeping latency (especially tail latency) low, and route users to nearby or faster back ends. 4. Maintainability and safe change: take servers out for maintenance or deployments without downtime (connection draining, rolling, blue-green and canary releases). 5. Security and traffic management: provide one controlled entry point for TLS termination, WAF and rate limiting, absorb DDoS traffic, and hide internal topology. 6. Flexibility: route by content (path, host, header), version or tenant, and manage traffic across clouds and data centres.
TLS termination, WAF, rate limiting, DDoS protection, IP allow lists
Blocked attacks, certificate hygiene
Traffic control
Path/host routing, header routing, A/B splits
Correct routing; experiment control
Cost efficiency
Higher utilisation, fewer idle servers, cheaper regions
Cost per request
Example: Objectives in a Cloud Setup
Route 53 latency/failover records -> region choice + regional failover (availability, performance)
ALB in 3 availability zones -> zone failure tolerance (availability)
target group: auto scaling group -> add/remove instances with load (scalability, cost)
health check /ready every 10 s -> remove bad instances (availability)
deregistration delay 30 s -> drain during deploys (safe change)
HTTPS listener + WAF + rate rules -> single secured entry (security)
weighted target groups 95/5 -> canary release (safe change)
Trade-Offs Between Objectives
Tension
Example
Affinity versus even spread
Sticky sessions help caches and stateful apps but unbalance load
Deep health checks versus stability
Checking dependencies can remove every server when the database blips
Latency versus availability
Nearest-region routing versus failing over to a distant region
Security inspection versus latency
WAF and TLS termination add processing time
Pitfalls
Focusing only on spreading load and ignoring health checks and failover.
Making the load balancer a single point of failure.
Not planning capacity for failure: with N servers, losing one must not overload the rest (N+1 or N+2 sizing).
Interview Tips
List the objectives: availability, scalability, performance, maintainability, security and flexibility.
Map each one to concrete mechanisms.
Mention trade-offs such as stickiness and health-check depth.
3
Explain the difference between hardware and software load balancers.
KnownRevise
Hardware load balancers are dedicated physical appliances from vendors such as F5 BIG-IP, Citrix NetScaler (now Cloud Software Group) and A10, with specialised hardware (ASICs, FPGAs, crypto accelerators) for very high throughput, connection rates and TLS offload. They come with vendor support, rich traffic-management features, and often certified security. They are expensive, scale vertically (buy a bigger box), need physical installation and capacity planning, and fit poorly with elastic, automated cloud environments. Software load balancers run on commodity servers, VMs or containers: nginx, HAProxy, Envoy, Traefik, the Linux kernel’s IPVS, Katran (XDP/eBPF), and the managed cloud load balancers, which are software-defined underneath. They are cheap or free, elastic, programmable and automatable (API and configuration-as-code), and they scale horizontally. Their performance depends on the host, though modern software (eBPF, DPDK, multi-core proxies) reaches very high rates. Today most new deployments use software or cloud load balancers. Hardware appliances remain in large on-premises data centres, at telco and financial edges, and where existing investment or specific features matter. Most hardware vendors also sell virtual editions of the same software.
Read full answerHide full answer
Applies toon-premises and cloud load-balancing options
Comparison
Aspect
Hardware appliance
Software load balancer
Form
Physical device with specialised chips
Software on servers, VMs, containers; managed cloud services
Performance
Very high and predictable per box
High; scales out with more instances
Scaling
Vertical (bigger or more appliances)
Horizontal and elastic
Cost
High purchase and support cost
Free or open source, or pay-as-you-go
Deployment speed
Weeks (procurement, racking)
Minutes (automation, infrastructure as code)
Flexibility
Vendor features and scripting (for example iRules)
Highly configurable and programmable; integrates with CI/CD, Kubernetes, service discovery
High availability
Active-standby pairs, vendor clustering
Multiple instances, anycast, managed redundancy
Lock-in
Vendor-specific
Lower (open-source options)
Best for
High-volume on-premises data centres, TLS-heavy legacy environments, regulated setups
Cloud, containers, microservices, most modern applications
Software Load Balancer Examples
Tool
Layer
Notes
nginx
L4/L7
Web server and reverse proxy; very common
HAProxy
L4/L7
High performance, rich health checks and stick tables
Envoy
L4/L7
Dynamic configuration (xDS); basis of Istio and many gateways
Traefik
L7
Automatic discovery from Docker and Kubernetes
IPVS / LVS
L4
Linux kernel; used by kube-proxy in IPVS mode
Katran, Cilium (eBPF/XDP)
L4
Very fast packet-level load balancing on commodity hardware
frontend web
bind :443 ssl crt /etc/haproxy/certs/shop.pem
default_backend app
backend app
balance leastconn
option httpchk GET /ready
http-check expect status 200
server app1 10.0.1.11:8080 check inter 5s fall 3 rise 2
server app2 10.0.1.12:8080 check inter 5s fall 3 rise 2
Choosing
Situation
Choice
Cloud-native application
Managed cloud load balancer, plus Ingress or Gateway in Kubernetes
On-premises with automation
HAProxy, nginx or Envoy on VMs; eBPF L4 for very high volumes
Existing large F5 or NetScaler estate
Keep, or move to the vendors’ virtual and cloud editions gradually
Extreme L4 volume
eBPF/XDP-based load balancers or hardware
Pitfalls
Overbuying hardware for workloads that need elasticity.
Name examples: F5 and NetScaler versus nginx, HAProxy, Envoy and cloud LBs.
Note the trend: software and cloud dominate, and vendors offer virtual editions.
4
What are common load balancing algorithms and how do they work?
KnownRevise
Load balancing algorithms decide which back end receives the next request or connection. They fall into two families: - Static algorithms ignore current load: round robin (take turns), weighted round robin (more turns for bigger servers), random, and hash-based methods (source IP hash, or a URL or header hash, often with consistent hashing) so the same key lands on the same server. - Dynamic algorithms use live state: least connections (fewest active connections), weighted least connections, least outstanding requests, least response time or latency-aware (EWMA) methods, resource-based methods (CPU or agent-reported load), and power of two random choices (sample two servers, pick the less loaded), which gets near-optimal balance cheaply.
Static methods suit uniform, short requests; dynamic methods handle uneven or long requests; hashing gives affinity for caches and sessions.
import bisect
import hashlib
import itertools
import random
from collections import Counter
SERVERS = ["a", "b", "c"]
WEIGHTS = {"a": 5, "b": 3, "c": 2}
# Round robin: take turns
rr = itertools.cycle(SERVERS)
print("round robin:", [next(rr) for _ in range(6)]) # ['a', 'b', 'c', 'a', 'b', 'c']
# Smooth weighted round robin (as in nginx): spreads picks instead of bursting
def weighted_rr(weights):
current = dict.fromkeys(weights, 0)
total = sum(weights.values())
while True:
for s in weights:
current[s] += weights[s]
best = max(current, key=current.get)
current[best] -= total
yield best
w = weighted_rr(WEIGHTS)
print("weighted rr:", "".join(next(w) for _ in range(10))) # abcaabacba: a 5, b 3, c 2 per 10, interleaved
# Least connections: pick the server with the fewest active connections
active = {"a": 12, "b": 4, "c": 9}
print("least conn:", min(active, key=active.get)) # b
# Power of two choices: sample two at random, take the less loaded one (cheap, near-optimal)
rng = random.Random(7)
picks = Counter()
for _ in range(1000):
x, y = rng.sample(SERVERS, 2)
choice = x if active[x] <= active[y] else y
picks[choice] += 1
print("p2c:", dict(sorted(picks.items()))) # {'b': 649, 'c': 351}: never the busiest (a)
# IP hash / consistent hashing: the same client key maps to the same server
def h(key):
return int(hashlib.md5(key.encode()).hexdigest(), 16) # non-cryptographic use: placement only
ring = sorted((h(f"{s}#{v}"), s) for s in SERVERS for v in range(100)) # 100 virtual nodes each
points = [p for p, _ in ring]
def lookup(key):
i = bisect.bisect(points, h(key)) % len(ring)
return ring[i][1]
clients = [f"10.0.0.{i}" for i in range(1000)]
before = {c: lookup(c) for c in clients}
ring = sorted(ring + [(h(f"d#{v}"), "d") for v in range(100)]) # add a 4th server
points = [p for p, _ in ring]
moved = sum(before[c] != lookup(c) for c in clients)
print(f"consistent hash: {moved / len(clients):.0%} of clients moved after adding a server")
# 20%: close to the ideal 1/4; a plain "hash mod N" would move about 3/4 of clients
Smooth weighted round robin interleaves picks (a, b, c, a, a, …) instead of sending five requests in a row to a. Power of two choices never chooses the busiest server. Consistent hashing with virtual nodes moves only about 1/N of keys when a server is added, while plain hash mod N would move most of them.
Summary
Algorithm
How it works
Good for
Weakness
Round robin
Next server in order
Equal servers, uniform requests
Ignores load and request cost
Weighted round robin
Servers get turns in proportion to weights
Mixed server sizes, canaries
Weights are static
Random
Pick uniformly at random
Simple, stateless distributed balancers
Short-term imbalance
Least connections
Fewest active connections
Long or variable connections (databases, WebSockets)
Connection count may not reflect real load
Least outstanding requests (Envoy, AWS ALB)
Fewest in-flight requests
HTTP/2 and gRPC, variable latency
Needs per-request tracking
Least response time / EWMA
Lowest observed latency (often combined with active requests)
Heterogeneous back ends
Can herd onto a server that just got fast
Power of two choices
Sample two, choose the less loaded
Large pools, many balancers with stale information
Slightly worse than global least-loaded
Source IP hash
Hash of the client IP selects the server
Simple affinity
Uneven with NAT (many users behind one IP); remaps on pool change
upstream api {
least_conn; # or: hash $request_uri consistent; or: ip_hash;
server 10.0.1.11:8080 weight=5;
server 10.0.1.12:8080 weight=3;
server 10.0.1.13:8080 weight=2;
}
backend api
balance leastconn # roundrobin | static-rr | source | uri | hdr(X-Tenant) | random(2)
hash-type consistent
random(2) in HAProxy is the power-of-two-choices algorithm.
Choosing
Workload
Suggested algorithm
Uniform, short HTTP requests
Round robin or random
Variable request costs
Least outstanding requests, power of two choices
Long-lived connections (WebSockets, databases)
Least connections
Cache servers, sharded data
Consistent hashing
Servers of different sizes
Weighted variants
Canary releases
Weighted round robin (95/5)
Pitfalls
Round robin with very uneven request costs, which builds queues on unlucky servers.
IP hash behind corporate NAT or a CDN: many users share one IP.
Many balancers each running least connections with only their local view (power of two choices handles this better).
Interview Tips
Separate static (round robin, weighted, hash) from dynamic (least connections, latency, P2C).
Explain consistent hashing and power of two choices.
Match algorithms to workload characteristics.
5
Describe the term “sticky session” in load balancing.
KnownRevise
A sticky session (session affinity or session persistence) means the load balancer sends all requests from the same client session to the same back-end server, instead of choosing a server independently for each request. It is used when servers keep per-user state in local memory, such as an in-memory login session, shopping cart, upload in progress or WebSocket context, so a request landing on a different server would not find it. The load balancer identifies the client by: - a cookie: either one the load balancer inserts (for example AWS ALB’s AWSALB, or HAProxy and nginx inserted cookies), or the application’s own session cookie; - the source IP address (IP hash); - a header or URL parameter, such as a tenant or session ID.
Stickiness is easy to enable but has costs: uneven load (heavy users pinned to one server), lost sessions when a server fails or is replaced, and harder scaling and deployments. The preferred design is stateless servers with sessions in a shared store (Redis or a database) or in signed tokens. Use stickiness as an optimisation (cache locality) or for legacy apps, not as a correctness requirement.
1. First request: no cookie -> LB picks server B (normal algorithm)
2. LB response adds: Set-Cookie: SERVERID=B; Path=/; HttpOnly; Secure
3. Next requests carry SERVERID=B -> LB sends them to B
4. If B is unhealthy -> LB picks another server; the session state on B is lost
upstream app {
ip_hash; # open-source nginx: affinity by client IP
server 10.0.1.11:8080;
server 10.0.1.12:8080;
}
Cookie-based sticky directives are an NGINX Plus feature. In Kubernetes, Services support sessionAffinity: ClientIP, and Ingress controllers offer cookie affinity through annotations.
Methods Compared
Method
Pros
Cons
LB-inserted cookie
Precise per browser; works behind NAT
Needs L7; cookies must be accepted
Application cookie
Reuses the existing session ID
LB must parse it; the app controls the lifetime
Source IP hash
Works at L4; no cookies
Many users behind one NAT IP; mobile IPs change
Header or URL key
Good for APIs (tenant ID)
Clients must send it
Trade-Offs
Benefit
Cost
Works with in-memory sessions
Uneven load (“hot” servers)
Better local cache hit rates
Session loss on failure, scale-in or deployment
Keeps WebSocket and long-poll context together
Draining takes longer; autoscaling is less effective
Better Alternatives
Externalise session state to Redis, Memcached or a database.
Stateless tokens: signed cookies or JWTs with server-side revocation where needed.
Consistent hashing for cache locality, which degrades gracefully when servers change.
Pitfalls
Long stickiness durations that keep users on overloaded or old-version servers.
Assuming stickiness guarantees state survival: it does not survive server failure.
IP-based stickiness behind CDNs or carrier NAT.
Deployments that ignore stickiness and cut off pinned users abruptly (drain first).
Interview Tips
Define it as routing a session’s requests to the same server, usually via a cookie or IP hash.
Explain why it is used (local state) and its costs (imbalance, failover loss).
Recommend stateless servers with shared session stores as the better design.
6
How does load balancing improve application reliability?
KnownRevise
Load balancing improves reliability by removing single points of failure among application servers and reacting automatically to failures: - Redundancy: several instances serve the same application, ideally across availability zones, so losing one machine or even one zone does not take the service down. - Health checking: the load balancer probes each back end (active checks) and watches real traffic for errors and timeouts (passive checks, outlier detection). It removes failing instances from rotation within seconds and adds them back when they recover. - Failover and retries: requests are routed to healthy servers. Some L7 balancers retry idempotent requests on another back end. - Overload protection: spreading load prevents any server from being pushed into failure; connection limits, queues, rate limiting and load shedding protect the back ends. - Safe changes:connection draining, rolling, blue-green and canary releases let you deploy and patch without downtime, and roll back quickly. - Geographic resilience:global load balancing fails traffic over to another region during a regional outage.
The load balancer itself must also be highly available: redundant pairs, managed multi-zone services, or anycast.
Read full answerHide full answer
Applies toreliability engineering with load balancers (vendor-neutral)
Failure Scenarios
Failure
Without load balancing
With load balancing
One server crashes
Users on it get errors until manual action
Health check fails → traffic moves in seconds
Server overloaded or slow
Requests time out
Least-request routing avoids it; outlier detection ejects it
In the route configuration, a retry policy such as retry_on: "5xx,reset,connect-failure" with num_retries: 2 retries failed idempotent requests on other hosts. max_ejection_percent stops the balancer ejecting too many hosts at once, which could overload the rest.
Capacity Planning for Failure
Peak load needs 6 servers. To survive losing one zone out of 3:
each zone must carry 1/2 of peak when one is down -> 3 servers per zone -> 9 servers total (N+3 by zone)
Load balancing only helps if the remaining servers can absorb the redistributed traffic.
Reliability of the Load Balancer Itself
Approach
Example
Managed, multi-zone service
AWS ALB/NLB, Azure Load Balancer, Google Cloud LB
Active-passive pair with a floating IP
keepalived (VRRP) with HAProxy or nginx
Active-active with ECMP or anycast
BGP-announced VIPs to several LB nodes
Multiple layers
Global LB in front of regional LBs
Pitfalls
Health checks that only test the port, so hung applications still receive traffic.
Deep health checks that fail every server when a shared dependency blips.
Retries without limits, which amplify overload; use retry budgets.
No spare capacity, so failover overloads the survivors.
A single load balancer instance.
Interview Tips
Explain redundancy + health checks + automatic failover.
Mention outlier detection, retries, draining, canaries and global failover.
Stress capacity for failure and redundancy of the load balancer itself.
7
How do load balancers perform health checks on backend servers?
KnownRevise
Load balancers decide which back ends may receive traffic by checking their health, using two complementary methods: - Active health checks: the load balancer sends probes to each back end on a schedule (for example every 5–10 seconds). A probe can be a TCP connect, an HTTP(S) request to a path such as /healthz or /ready expecting a status code (200) or response content, a gRPC health check, or a custom script. A back end is marked unhealthy after N consecutive failures (unhealthy threshold) and healthy again after M consecutive successes (healthy threshold). Each probe has a timeout. - Passive health checks (outlier detection): the load balancer watches real traffic: connection failures, timeouts and 5xx rates. It temporarily ejects back ends that misbehave, even if the probes still pass.
Good health endpoints report whether this instance can serve requests: started, warmed up, not shutting down, and with critical local resources available. They should not fail because of a shared downstream dependency. Unhealthy back ends stop receiving new requests; in-flight connections are usually allowed to finish (draining).
Detection time ≈ interval × unhealthy threshold (for example 5 s × 3 = 15 s).
Examples
backend app
option httpchk GET /ready
http-check expect status 200
default-server inter 5s fall 3 rise 2
server app1 10.0.1.11:8080 check
server app2 10.0.1.12:8080 check
TCP-only checks on HTTP apps: a hung app still accepts connections.
Health checks touching shared dependencies, causing all-instance failures.
Thresholds too aggressive (flapping) or too lax (slow detection).
Health endpoints exposed publicly with sensitive details; restrict them or keep responses minimal.
Fail-open behaviour unknown: some load balancers send traffic to all targets when all are unhealthy (AWS ALB does). Know your platform’s behaviour.
Interview Tips
Explain active probes (interval, timeout, thresholds) and passive outlier detection.
Describe a good /ready endpoint, and why to avoid shared-dependency checks.
Mention draining, Kubernetes readiness versus liveness and fail-open behaviour.
8
What are the advantages and disadvantages of round-robin load balancing?
KnownRevise
Round robin sends each new request or connection to the next server in a fixed rotation: A, B, C, A, B, C, and so on. Its advantages: - it is simple and predictable; - it is fast, with almost no state or computation; - it is fair when servers are identical and requests are similar and short; - it needs no feedback from the servers; - it is easy to reason about and debug; - weighted round robin handles servers of different sizes.
Its disadvantages come from ignoring current load: - it treats all requests as equal, so with variable request costs (a heavy report next to tiny API calls) some servers build queues while others are idle, raising tail latency; - it ignores server speed, health degradation or warm-up (a slow or new server still gets its full share); - it handles long-lived connections poorly (WebSockets, HTTP/2, gRPC), because balancing per connection creates lasting imbalance; - it provides no affinity; - several independent balancers can end up synchronised.
Read full answerHide full answer
For variable workloads, least outstanding requests or power of two choices usually perform better.
Applies to: load balancing algorithm selection (Python 3.12 simulation)
Code Example: Round Robin Versus Least-Loaded With Uneven Requests
import random
def simulate(policy, n_requests=20_000, servers=4, seed=1):
rng = random.Random(seed)
busy_until = [0.0] * servers # when each server finishes its queued work
waits = []
t = 0.0
turn = 0
for _ in range(n_requests):
t += rng.expovariate(2.6) # arrivals: 2.6 per time unit (about 80% utilisation)
# 90% cheap requests (0.5), 10% expensive ones (8.0): uneven work per request
cost = 8.0 if rng.random() < 0.1 else 0.5
if policy == "round_robin":
s = turn % servers
turn += 1
else: # least loaded (least outstanding work)
s = min(range(servers), key=lambda i: busy_until[i])
start = max(t, busy_until[s])
busy_until[s] = start + cost
waits.append(start - t)
waits.sort()
return sum(waits) / len(waits), waits[int(len(waits) * 0.99)]
for policy in ("round_robin", "least_loaded"):
avg, p99 = simulate(policy)
print(f"{policy:13s} avg wait {avg:5.2f} p99 wait {p99:6.2f}")
# round_robin avg wait 11.06 p99 wait 64.57
# least_loaded avg wait 2.58 p99 wait 14.66
Both policies serve the same traffic at about 80% utilisation, where 10% of requests are 16 times more expensive. Round robin ignores the queues, so expensive requests pile up on whichever server is next. Its average wait is about 4 times higher and its p99 wait about 4 times higher than a load-aware policy.
Advantages and Disadvantages
Advantages
Disadvantages
Very simple; minimal state
Ignores current load and queue length
Low CPU overhead at very high rates
Poor with variable request costs (higher tail latency)
Even distribution of request counts
Even counts ≠ even work
Works well for homogeneous, stateless, short requests
DNS round robin for coarse distribution (with health-checked records).
Canary splits with weights (95/5).
Very high request rates where per-request load tracking is too costly.
Better Alternatives When It Struggles
Problem
Alternative
Variable request durations
Least outstanding requests, power of two choices
Long-lived connections
Least connections; L7 per-request balancing for HTTP/2 and gRPC
Heterogeneous or degrading servers
Latency-aware (EWMA) balancing, outlier detection
Cache locality
Consistent hashing
New instances cold
Slow start or warm-up (HAProxy slowstart, ALB slow start mode)
Pitfalls
Assuming equal request counts mean equal load.
Round robin over HTTP/2 or gRPC connections at L4: one connection can carry most of the traffic.
No health checks, so round robin keeps sending traffic to failed servers.
Interview Tips
Describe round robin as simple, stateless and fair for uniform work.
Explain the main weakness, ignoring load with uneven requests, using the queueing intuition.
Suggest weighted, least-requests or P2C alternatives depending on the workload.
9
In load balancing, what is the significance of the least connections method?
KnownRevise
Least connections sends each new connection or request to the back end with the fewest active connections at that moment. Its significance is that it is the simplest dynamic, load-aware algorithm: it uses live state instead of a fixed rotation, so it adapts automatically when connections have very different durations or servers process at different speeds. A server stuck with slow or long-lived work accumulates connections and stops receiving new ones until it catches up. This makes it a strong default for: - long-lived connections: WebSockets, database and message-broker connections, streaming, gRPC streams; - variable request times: reports next to fast API calls; - mixed server performance.
Weighted least connections divides active connections by server capacity. Close relatives are least outstanding requests (per HTTP request rather than per TCP connection, used for HTTP/2 and gRPC in Envoy and AWS ALB) and power of two choices, which samples two back ends and picks the one with fewer connections, scaling better across many load balancers. Limitations: it counts connections, not actual work; it needs state shared across balancer instances, otherwise each sees only its own connections; and it can flood a newly added or recovered server, which has zero connections, unless slow start is used.
import heapq
import random
def run(policy, servers=4, n=5_000, seed=3):
rng = random.Random(seed)
active = [0] * servers
ends = [] # heap of (end_time, server)
peak = [0] * servers
t, turn = 0.0, 0
for _ in range(n):
t += rng.expovariate(1.0)
while ends and ends[0][0] <= t: # close finished connections
_, s = heapq.heappop(ends)
active[s] -= 1
# Mixed connection lengths: most short API calls, some long-lived WebSocket sessions
duration = rng.expovariate(1 / 300) if rng.random() < 0.05 else rng.expovariate(1 / 2)
if policy == "round_robin":
s = turn % servers
turn += 1
else:
s = min(range(servers), key=lambda i: active[i])
active[s] += 1
peak[s] = max(peak[s], active[s])
heapq.heappush(ends, (t + duration, s))
return peak
print("round robin peak connections: ", run("round_robin")) # [8, 11, 9, 10]
print("least connections peak connections:", run("least_conn")) # [7, 7, 7, 6]
With 5% long-lived sessions mixed into short calls, round robin piles long sessions onto some servers by chance. Least connections keeps the peak connection counts nearly equal.
Configuration
upstream ws_backend {
least_conn;
server 10.0.1.11:8080 weight=2; # weighted least connections: bigger server
server 10.0.1.12:8080;
server 10.0.1.13:8080;
}
backend websockets
balance leastconn
timeout tunnel 1h # keep idle WebSocket tunnels open
server ws1 10.0.1.11:8080 check slowstart 60s
server ws2 10.0.1.12:8080 check slowstart 60s
slowstart ramps a recovered server’s weight up over 60 seconds, so it is not flooded the moment it returns with zero connections.
Least Connections Versus Alternatives
Algorithm
Unit
Strength
Weakness
Round robin
Request or connection
Simple; good for uniform short requests
Ignores load
Least connections
Active connections
Adapts to long or uneven connections
Counts, not work; new-server flood
Weighted least connections
Connections ÷ weight
Mixed server sizes
Static weights
Least outstanding requests
In-flight HTTP requests
Correct for HTTP/2 and gRPC multiplexing
Needs L7
Power of two choices
Two random samples
Works with many independent balancers
Slightly less precise
Least response time / EWMA
Latency
Accounts for slow servers
Can herd and oscillate
When to Use It
Workload
Least connections?
WebSockets, SSE, long polling
Yes
Database or broker proxies (PgBouncer, ProxySQL in front of replicas)
Yes
Mixed long and short HTTP requests
Yes (or least outstanding requests)
Uniform, very short requests at huge rates
Round robin is simpler and similar
Cache servers
No: use consistent hashing for locality
Pitfalls
Many balancer instances each counting only their own connections: the global view is lost (P2C handles this better).
HTTP/2 or gRPC at L4: one connection carries many requests, so connection counts mislead.
No slow start: a recovered or newly scaled server receives a burst of connections.
Ignoring weights when servers differ in size.
Interview Tips
Define it as “send to the server with the fewest active connections”: a dynamic algorithm.
Explain why it suits long-lived and uneven connections.
Mention weighted variants, least outstanding requests, P2C and slow start.
10
Explain how a load balancer handles the failure of one of its servers.
KnownRevise
When a back-end server fails, a load balancer typically goes through this sequence: 1. Detect: active health checks (missed probes beyond the unhealthy threshold) or passive detection (connection refused, timeouts, a burst of 5xx errors) mark the server down. With outlier detection this takes seconds. 2. Stop sending new traffic: the server is removed from the rotation, and the algorithm only chooses among healthy servers. With session persistence, affected sessions are reassigned to other servers; their in-memory state is lost unless it is stored externally. 3. Handle in-flight requests: requests already sent to the failed server fail or time out. An L7 load balancer can retry idempotent requests on another server (limited by retry budgets), so users often see nothing. 4. Protect the rest: the other servers absorb the load. Connection limits, queues and autoscaling prevent a cascade. If too many servers fail, some balancers fail open (send traffic to all servers) to avoid a total outage, or serve a fallback or maintenance response. 5. Recover: once health checks pass the healthy threshold, the server is gradually reintroduced, ideally with slow start. Orchestrators or autoscaling groups replace dead instances automatically. 6. Alert and record: metrics and logs show the event for investigation.
t=0s app2 crashes
t=0-1s in-flight requests to app2 fail -> LB retries GETs on app1/app3 (POSTs return 502)
t=5s probe 1 fails; t=10s probe 2 fails; t=15s probe 3 fails -> app2 marked DOWN
(passive detection may eject it after the first few connection errors, sooner)
t=15s+ all new traffic goes to app1/app3; autoscaling group launches app4
t=90s app4 passes 2 health checks -> added with slow start
Configuration Examples
upstream app {
server 10.0.1.11:8080 max_fails=3 fail_timeout=10s; # passive: 3 failures -> skip for 10 s
server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
server 10.0.1.13:8080 backup; # used only when the others are down
}
server {
location / {
proxy_pass http://app;
proxy_next_upstream error timeout http_502 http_503; # retry on another server
proxy_next_upstream_tries 2;
proxy_connect_timeout 1s;
}
}
By default nginx retries only idempotent methods with proxy_next_upstream; non_idempotent must be added explicitly to retry POSTs, which is usually unsafe.
backend app
option httpchk GET /ready
option redispatch # move persistent sessions to a healthy server
retries 2
default-server inter 3s fall 3 rise 2 slowstart 30s
server app1 10.0.1.11:8080 check
server app2 10.0.1.12:8080 check
Handling Different Failure Types
Failure
Detection
Handling
Crash or host down
Connection refused, failed probes
Remove; retry elsewhere
Hung process (accepts, never replies)
Timeouts; HTTP checks (TCP checks miss it)
Remove; set short connect and read timeouts
Slow or degraded
Latency-aware outlier detection
Eject temporarily or reduce weight
Partial errors (some 500s)
Passive 5xx tracking
Eject when over threshold
Whole zone fails
Many servers fail at once
Cross-zone pools; global LB fails over
All back ends fail
All checks fail
Fail open, or serve a static error or maintenance page; alert
Pitfalls
Retrying non-idempotent requests, causing duplicate orders or payments.
Slow detection (long intervals, high thresholds), or flapping with thresholds too low.
No spare capacity, so the remaining servers overload.
Session loss with stickiness and in-memory state.
Re-adding recovered servers at full weight immediately.
Interview Tips
Walk through detect → remove → retry → absorb → recover → alert.
Distinguish active and passive detection and idempotent retries.
Mention slow start, fail-open behaviour, capacity headroom and automatic replacement.
11
How does a load balancer distribute traffic in a stateless vs stateful scenario?
KnownRevise
In a stateless scenario, each request carries everything needed to process it (an auth token, parameters), and any shared state lives in external stores (databases, Redis, object storage). The load balancer can therefore send every request to any healthy server using a pure load-oriented algorithm: round robin, least requests or power of two choices. This gives the best balance, simplest failover (a failed server loses nothing), easy autoscaling and zero-downtime deployments. In a stateful scenario, a server keeps per-client or per-key state locally: in-memory sessions, a WebSocket connection, a game room, a cached shard, a partition of data. Requests for that client or key must reach the server that holds the state. The load balancer then routes by affinity: - sticky sessions via cookies or source IP for user sessions; - consistent hashing on a key (user ID, room ID, cache key) for sharded state; - connection-level routing for long-lived connections, where all messages on one connection stay on its server.
Stateful routing trades away even load and easy failover for locality. Designs aim to make app servers stateless and keep state in purpose-built stateful systems with their own replication.
Read full answerHide full answer
Applies toapplication and load balancer architecture
Comparison
Aspect
Stateless
Stateful
Routing
Any healthy server
Same server per session or key
Algorithms
Round robin, least requests, P2C, random
Cookie stickiness, IP hash, consistent hashing, header-based routing
Load balance
Even
Can be skewed (hot users or keys)
Server failure
Transparent (retry elsewhere)
Sessions or shards lost or must be rebuilt; clients reconnect
Scaling out or in
Instant
Requires rebalancing or draining; keys move
Deployments
Rolling, easy
Need draining, state migration or reconnect handling
# Stateless API: any server
upstream api {
least_conn;
server 10.0.1.11:8080;
server 10.0.1.12:8080;
}
# Stateful by key: game rooms, consistent hashing on the room ID in the URL
upstream rooms {
hash $arg_room consistent;
server 10.0.2.11:9000;
server 10.0.2.12:9000;
server 10.0.2.13:9000;
}
hash ... consistent uses a ketama-style ring, so adding a server moves only about 1/N of the rooms.
Making Stateful Workloads Easier to Balance
Technique
Effect
Externalise sessions (Redis, database)
Turns web tiers stateless
Signed tokens (JWT, signed cookies)
Clients carry state securely
Pub/sub for real-time fan-out (Redis, NATS, Kafka)
WebSocket servers become interchangeable
Consistent hashing with virtual nodes
Small, predictable movement on scaling
Replicated state (Raft, primary-replica)
Failover without losing data
Graceful draining and reconnection logic
Clients move smoothly during deployments
Kubernetes Note
Deployments behind a Service suit stateless pods: any ready pod receives traffic. StatefulSets give pods stable names (db-0, db-1), and clients or headless Services address specific replicas. Stateful systems (databases, Kafka) usually handle their own routing (leaders, partitions) rather than relying on a generic load balancer.
Pitfalls
Accidentally stateful apps (in-memory sessions or file uploads on local disk) behind round-robin balancers, causing random logouts or errors.
IP-hash stickiness with NAT or mobile clients.
Hot keys overloading one server under consistent hashing.
Scaling stateful tiers like stateless ones without moving their state.
Interview Tips
Stateless: any server, load-based algorithms, easy failover. Stateful: affinity, sticky sessions or consistent hashing.
Discuss trade-offs (balance, failover, scaling).
Recommend externalising state, and consistent hashing where locality is needed.
12
What is the concept of session persistence, and why is it important?
KnownRevise
Session persistence (session affinity, “sticky sessions”) is the load balancer’s ability to keep sending a client’s requests to the same back-end server for the duration of a session. The client is identified by a cookie (inserted by the load balancer or set by the application), the source IP, or a header or URL token. It is important when correctness or performance depends on server-local state: - in-memory user sessions (login state, shopping carts) in applications that do not share session storage; - multi-step workflows that keep temporary data on one server (wizards, chunked uploads); - long-lived or stateful protocols (WebSockets, SignalR and Socket.IO fallbacks, some SSO and SAML flows); - performance: better local cache hit rates, warm per-user data, fewer repeated authentications; - legacy applications that were never designed for horizontal scaling.
Without persistence, such applications show random logouts, lost carts and failed uploads. Persistence is not a guarantee of state survival: if the server fails, its sessions are gone. It also skews load and complicates scaling and deployments. Modern designs therefore treat persistence as an optimisation and keep important session state in shared stores or tokens.
Read full answerHide full answer
Applies toL7 and L4 load balancers, Kubernetes Ingress and Service affinity
At L4, a Kubernetes Service can use sessionAffinity: ClientIP with a timeout.
Why It Matters, and the Costs
Benefit
Cost
Correctness for apps with local session state
Uneven load: heavy users pinned to one server
Better cache locality and performance
Session loss when that server fails or is scaled in
Required by some protocols and legacy apps
Slower draining and deployments
Simple to enable
Masks architectural debt (stateful web tiers)
Best Practices
Keep stickiness duration short, and set it in line with application session timeouts.
Drain servers before removing them, so sticky users finish their work.
Allow redispatch when the pinned server is down (HAProxy option redispatch).
Prefer externalised sessions (Redis, a database) or signed tokens, using stickiness only for performance.
Monitor per-server load to spot imbalance caused by affinity.
Pitfalls
Treating stickiness as reliability: it is the opposite; failures lose sessions.
IP affinity for users behind CGNAT or corporate proxies, which pins thousands of users to one server.
Cookie affinity with CDNs or caches that strip or share cookies.
Very long affinity that keeps users on old versions after deployments.
Interview Tips
Define it as keeping a client’s requests on the same server.
Explain why: local session state, protocols, cache locality.
Cover methods (cookie, IP, header), costs, and the preferred stateless design with shared session stores.
13
Discuss the role of DNS in load balancing.
KnownRevise
DNS is often the first layer of load balancing: it decides which IP address, and therefore which data centre, region, load balancer or CDN edge, a client connects to. Techniques: - DNS round robin: several A or AAAA records for one name, rotated in responses. - Weighted records: traffic split by proportion (canaries, migrations). - Geo-DNS / geolocation routing: answers based on the resolver’s or client’s location. - Latency-based routing: the region with the lowest measured latency. - Health-checked failover: remove records for unhealthy endpoints; primary/secondary failover. - Global server load balancing (GSLB): combines these with health and capacity data.
Managed DNS services provide them: Amazon Route 53, Azure Traffic Manager, Google Cloud DNS routing policies, Cloudflare, NS1.
Read full answerHide full answer
DNS load balancing is cheap, global and scalable, but coarse and slow to react: - caching and TTLs mean clients keep old answers (often longer than the TTL, because some resolvers and apps ignore it); - DNS sees the resolver’s location, not the user’s (partly fixed by EDNS Client Subnet); - it knows nothing about per-request load; - clients may pick addresses in different orders.
In practice DNS distributes traffic between regions or sites, and regional load balancers (L4/L7) handle fine-grained, per-request balancing. Anycast is the complementary network-level technique.
api.example.com A ALIAS alb-eu-west-1 routing: latency (eu-west-1), health check: /healthz
api.example.com A ALIAS alb-us-east-1 routing: latency (us-east-1), health check: /healthz
api.example.com A ALIAS alb-ap-south-1 routing: latency (ap-south-1), health check: /healthz
TTL: 60 seconds
A client in Paris resolves to the eu-west-1 load balancer. If its health check fails, Route 53 answers with the next-best healthy region.
dig +short api.example.com # see which addresses this resolver returns
dig api.example.com +ttlunits # remaining TTL on the cached answer
Strengths and Limitations
Strengths
Limitations
Global reach; no extra network hop
Caching: changes take TTL + resolver/client caching to apply
Very scalable and cheap
Coarse: per resolver, not per request or per user
Works across clouds and data centres
No awareness of server load (unless integrated with GSLB data)
Simple failover between regions
Clients may cache failed addresses; some retry poorly
Enables geo and latency steering
Resolver location ≠ user location (EDNS Client Subnet helps)
DNS Combined With Other Layers
DNS / GSLB (region choice, failover) -> anycast edge or CDN -> regional L7 load balancer
-> Kubernetes Service / service mesh -> pods
Best Practices
Use short TTLs (30–60 s) for records that may fail over, balanced against query cost.
Point DNS at highly available load balancers, not at individual servers.
Use health checks that reflect real service health in each region.
Plan for clients that ignore TTLs: keep old endpoints serving or redirecting during migrations.
DNS round robin to individual servers without health checks: failed servers keep receiving traffic.
Very long TTLs that make failover take hours.
Treating DNS as precise load balancing.
A single DNS provider as a single point of failure.
Interview Tips
Explain DNS as global, coarse-grained balancing (round robin, weighted, geo, latency, failover).
Discuss TTL and caching limits and resolver location.
Show the layered design: DNS for regions, L4/L7 load balancers within regions.
14
In what scenarios would you use weighted load balancing?
KnownRevise
Weighted load balancing assigns each back end a weight, and traffic is split in proportion to the weights: weights 3:1 send about 75% and 25%. It is used when back ends should not receive equal shares: - heterogeneous capacity: servers or instance types of different sizes (8 cores versus 2), or mixed hardware generations; - canary releases and progressive delivery: send 1–5% of traffic to a new version, then ramp up (10%, 25%, 50%, 100%) while watching metrics; - blue-green cut-overs and migrations: shift traffic gradually between environments, clusters, clouds or data centres, including DNS-weighted moves between providers; - A/B testing: fixed percentages to experiment variants (often combined with stickiness so users stay in their variant); - warming up new or recovered instances (slow start ramps the weight up); - draining or de-prioritising back ends: lower weights for servers being phased out, under maintenance or in a costlier region; - cost or licence balancing: favour cheaper capacity (reserved instances, on-premises) and overflow to more expensive capacity.
Weighted versions exist for most algorithms: weighted round robin, weighted least connections, weighted DNS records, and weighted target groups or routes (AWS ALB, Istio, Gateway API, Argo Rollouts).
Show examples (nginx weights, Gateway API, ALB weighted target groups), and mention monitoring and stickiness interactions.
15
How can load balancers help mitigate DDoS attacks?
KnownRevise
Load balancers help against distributed denial-of-service (DDoS) attacks in several ways: - Absorbing and spreading volume: large managed and anycast load balancers (cloud LBs, CDN edges) have huge capacity spread across many locations, so attack traffic is diluted. Back-end servers are hidden behind them. - Filtering at the edge: dropping malformed and non-application traffic. An L7 load balancer accepts only valid HTTP(S) on allowed ports, and its TCP stack absorbs SYN floods (SYN cookies, managed connection tracking) before anything reaches the servers. - Rate limiting and connection limits per IP, per client, per path or globally, plus request size limits and timeouts against slow attacks such as Slowloris and slow POSTs. - Integration with WAF and bot management: blocking known bad IPs, countries, signatures and HTTP floods; challenges such as CAPTCHAs or JavaScript checks. - Autoscaling the back ends behind the balancer, with caching offloading repeated requests. - Health-based isolation and graceful degradation: shed low-priority traffic and keep critical paths alive.
A load balancer alone cannot stop very large volumetric attacks. Those need upstream scrubbing: cloud DDoS protection such as AWS Shield, Azure DDoS Protection, Google Cloud Armor and Cloudflare, or ISP and scrubbing services. In practice the load balancer is one layer of a DDoS defence.
Read full answerHide full answer
Applies tocloud and self-managed load balancers, CDNs, WAFs
Attack Types and Defences
Attack
Layer
Load balancer / edge defence
UDP or ICMP floods, amplification (DNS, NTP, memcached)
L3/4
Only listen on needed ports; anycast absorption; upstream scrubbing
SYN floods
L4
SYN cookies, managed TCP termination, connection rate limits
TLS handshake floods
L5/6
TLS termination at scale, session resumption, connection limits
$binary_remote_addr must be the real client IP. Behind a CDN, configure real_ip_header and trusted proxy ranges, otherwise you rate-limit the CDN.
Layered Architecture
attackers -> anycast CDN / DDoS protection (Cloudflare, Shield Advanced, Cloud Armor)
absorbs volumetric attacks, bot management, WAF
-> cloud L7 load balancer (only 443 open; TLS; rate rules)
-> autoscaled app servers (not publicly reachable: security groups allow only the LB)
-> caches and databases (protected by app-level limits and queues)
Practices
Hide origins: back ends accept traffic only from the load balancer or CDN (security groups, private subnets, mTLS from the CDN).
Cache aggressively for public content, so attacks hit the cache rather than the application.
Prepare runbooks: who to call, how to enable stricter modes (“under attack” modes, geo blocks).
Monitor request rates, error rates and connection counts per source.
Load-test limits and autoscaling ahead of time.
Pitfalls
Origin IPs exposed (old DNS records, email headers), so attackers bypass the CDN and load balancer.
Rate limiting by the wrong IP behind proxies.
Autoscaling without cost caps, so an attack becomes a large bill (“economic DoS”).
Expecting a self-managed load balancer to survive multi-terabit attacks.
Interview Tips
Explain absorption, filtering, rate limiting, WAF integration and autoscaling.
Distinguish volumetric (L3/4) from application (L7) attacks, and their defences.
Stress upstream DDoS services, hidden origins and layered defence.
16
What is horizontal versus vertical scaling, and how does load balancing apply?
KnownRevise
Vertical scaling (scaling up) means giving one machine more resources: more CPU, RAM, faster disks or a bigger instance type. It needs no application changes and suits systems that are hard to distribute, such as many relational databases. But it has a hard ceiling (the biggest machine), often requires downtime or a restart to resize, gets expensive at the top end, and leaves a single point of failure. Horizontal scaling (scaling out) means adding more machines or instances and spreading the work across them. It offers near-linear capacity growth, fault tolerance through redundancy, and elasticity (add or remove instances automatically). It requires stateless or partitioned application design.
Load balancing is what makes horizontal scaling work: it gives the pool one entry point, distributes requests across instances, adds new instances as they become healthy (autoscaling integration) and removes failed or retired ones. With vertical scaling there is only one server, so a load balancer adds little for distribution. It can still help with failover to a standby, TLS offload and zero-downtime resizing (bring up a bigger instance, shift traffic, retire the old one). Most systems combine both: scale stateless tiers out behind load balancers, and scale stateful tiers up, adding replicas or shards when needed.
Read full answerHide full answer
Applies tocapacity planning and architecture (vendor-neutral)
Comparison
Aspect
Vertical (scale up)
Horizontal (scale out)
How
Bigger machine
More machines
Application changes
Usually none
Statelessness, shared state, partitioning
Limit
Largest available machine
Very high (coordination costs eventually)
Fault tolerance
Single point of failure
Redundant; survives node loss
Elasticity
Resizing often needs a restart
Add or remove instances automatically
Cost curve
Rises steeply at the top end
Roughly linear; commodity instances
Load balancer role
Minor (failover to a standby, TLS offload)
Essential: distribution, health checks, scaling integration
Web and API servers, workers, caches (sharded), stateless microservices
How Load Balancing Supports Horizontal Scaling
+--> app-1
client -> LB ----+--> app-2 autoscaling adds app-4 when CPU > 60%:
+--> app-3 1. instance boots, passes health checks
2. LB adds it to the pool (with slow start)
3. traffic redistributes automatically
scale-in: LB drains app-4's connections, then it terminates
Example: Kubernetes Horizontal Pod Autoscaler Behind a Service
The Service (and Ingress or Gateway in front) automatically includes every ready pod, so new replicas start receiving traffic as soon as their readiness probes pass.
Start simple: scaling up a database is often cheapest early on.
Design web and API tiers to be stateless from the start, so they can scale out.
Plan for failure: even when vertical scaling suffices for load, have at least two instances behind a load balancer for availability.
Pitfalls
Scaling out stateful apps without externalising state (sticky-session dependence).
Bottlenecks that move: scaling out the web tier until the database saturates.
Ignoring the load balancer’s own capacity (connection limits, TLS CPU).
Vertical scaling without a failover plan.
Interview Tips
Define both, with limits, cost and fault tolerance.
Explain that load balancing is the enabler of horizontal scaling (distribution, health checks, autoscaling).
Mention hybrid strategies: scale out stateless tiers, scale up or shard stateful ones.
17
What factors should you consider when choosing a load balancing method for an application?
KnownRevise
Choose a load balancing method by matching it to how the application behaves and what the business needs. The main factors: - Request profile: short, uniform requests suit round robin; variable or long-lived ones (WebSockets, streaming, slow reports) suit least connections or least outstanding requests. - State and affinity: stateless services can go anywhere. Server-local state needs sticky sessions; sharded data or caches need consistent hashing. - Server capacity: identical servers or mixed sizes, which calls for weights. - Protocol and layer: L4 (TCP/UDP: fast, content-blind) or L7 (HTTP routing by path, header or cookie; TLS termination). For HTTP/2 and gRPC, balance per request, not per connection. - Scale and topology: one balancer or many independent instances (where power of two choices and hashing work better than global counters), and multiple regions (DNS/GSLB, anycast). - Performance and latency goals: tail latency, throughput, and whether to add latency-aware methods (EWMA, outlier detection). - Availability: health checks, failover, slow start and draining support. - Deployment strategy: canaries and blue-green need weighted routing. - Security: TLS termination or passthrough, WAF, rate limiting, DDoS protection. - Operations and cost: managed or self-hosted, observability, team skills, licence and traffic costs.
The best choice is usually the simplest method that meets these needs, validated by load testing and production metrics.
Read full answerHide full answer
Applies tosystem design and load balancer selection (vendor-neutral)
Decision Guide
Situation
Suggested method
Stateless API, uniform short requests
Round robin or random
Variable or long-lived connections
Least connections / least outstanding requests
Different server sizes
Weighted round robin or weighted least connections
Many balancer instances (sidecars, client-side LB)
Power of two choices (P2C)
Caches, sharded state, per-room or per-tenant servers
Consistent hashing on a key
Legacy app with in-memory sessions
Sticky sessions (cookie), ideally temporary
Canary or migration
Weighted routing
Users worldwide
GSLB / latency-based DNS or anycast, then local L7
Servers with fluctuating speed
Least response time / EWMA with outlier detection
Questions to Ask
1. Are servers interchangeable for every request? no -> hashing or stickiness
2. Do request durations vary a lot? yes -> least connections / requests
3. Are servers equal in capacity? no -> weights
4. Is traffic HTTP/2, gRPC or WebSockets? yes -> L7, per-request or connection-aware
5. How many balancers make decisions independently? many -> P2C, avoid global counters
6. Do we need canaries or blue-green? yes -> weighted routes
7. Where are users? global -> DNS/anycast + regional LBs
Factors and Trade-offs
Factor
Trade-off to consider
Simplicity
Round robin is predictable but load-blind
Load awareness
Dynamic methods need state and accurate signals
Affinity
Locality and correctness versus even load and failover
Layer
L7 features versus L4 speed and protocol neutrality
Managed service
Less operations work versus less control and cost per LCU or GB
Hashing
Stable mapping versus hot keys
Validating the Choice
Load-test with realistic traffic (mix of request types, connection lengths, burstiness).
Test failure scenarios: a server dying, a slow server, scaling events.
Revisit as traffic patterns change.
Pitfalls
Defaulting to round robin for long-lived connections.
Sticky sessions chosen as a quick fix that then blocks scaling.
Connection-based balancing for HTTP/2 and gRPC, which pins many requests to one server.
Over-engineering with complex latency-based algorithms that oscillate.
Interview Tips
List request profile, state, capacity, protocol, scale, availability, deployment, security and cost.
Map situations to methods with clear reasoning.
Emphasise validation with metrics and load tests.
18
Discuss how global server load balancing differs from local server load balancing.
KnownRevise
Local server load balancing (LSLB) distributes traffic among servers within one site: a data centre, cloud region or cluster. It works at L4 or L7, makes a decision per connection or per request, sees real-time server health and load, and reacts within seconds. Examples: nginx, HAProxy, F5 BIG-IP LTM, AWS ALB/NLB, Kubernetes Services. Global server load balancing (GSLB) distributes traffic across multiple sites or regions. It decides which site a user should reach, usually via DNS (geo, latency, weighted and failover answers) or anycast (the same IP advertised from many locations, with BGP routing users to the nearest one). Its goals are proximity and low latency, disaster recovery when a whole site fails, data residency and capacity spreading between regions. GSLB works at coarse granularity (per DNS lookup or network route), and DNS-based GSLB is limited by caching and TTLs. The two are complementary: GSLB picks the region or site, then LSLB inside it picks the server. Examples of GSLB: Route 53 routing policies, Azure Traffic Manager and Front Door, Google Cloud’s global external load balancer, Cloudflare Load Balancing, F5 BIG-IP DNS (GTM).
Read full answerHide full answer
Applies tomulti-region and multi-data-centre architectures
Comparison
Aspect
Local (LSLB)
Global (GSLB)
Scope
Servers within one site or region
Sites, regions or data centres
Mechanism
Reverse proxy or L4 forwarding
DNS answers, anycast, or a global edge proxy
Decision unit
Per connection or request
Per DNS resolution, per route, or per edge request
Inputs
Server health, connections, latency
Site health, user location, region latency, capacity, policy
Reaction time
Seconds
DNS: TTL plus resolver caching (minutes); anycast or edge proxy: seconds
Goal
Spread load, server failover
Proximity, regional failover, residency, DR
Examples
nginx, HAProxy, ALB, NLB, F5 LTM
Route 53, Traffic Manager, Front Door, Cloudflare LB, F5 GTM
Layered Architecture
user in Mumbai
|
v GSLB: DNS latency routing -> ap-south-1 (or anycast edge nearest Mumbai)
regional LB (ALB in ap-south-1)
|
v LSLB: least outstanding requests across healthy targets
app servers / pods in ap-south-1
If ap-south-1 fails its health checks, GSLB answers with the next-best region (for example eu-west-1). Local balancers there absorb the extra traffic.
GSLB Techniques
Technique
Notes
Geo-DNS
Region by resolver or client subnet (EDNS Client Subnet)
Latency-based DNS
Uses measured latency between networks and regions
Weighted DNS
Gradual migration between regions or providers
Failover DNS
Active-passive DR
Anycast
Same IP from many places; BGP picks the route; fast failover
Global proxy (edge)
One anycast IP terminates at the edge, then forwards to the best region (Google global LB, Front Door, Cloudflare)
Design Considerations for GSLB
Data: can every region serve every user? Multi-region data replication, conflict handling and residency rules decide this.
Session continuity: users moved between regions may lose sessions unless state is global.
Capacity: each region needs headroom to absorb a failed region’s load.
Health checks should reflect full regional health (dependencies, not just the load balancer).
TTL choices: short enough for failover, aware that some clients ignore them.
Pitfalls
Assuming GSLB failover is instant with DNS.
Active-active regions without replicated data, giving inconsistent user experiences.
Health checks that only test the regional LB, missing a broken database.
No capacity planning for failover, so the surviving region collapses.
Interview Tips
LSLB: servers within a site, fine-grained, real-time. GSLB: sites and regions, DNS or anycast, proximity and DR.
Show the layered model and name example products.
Discuss TTL limits, data replication and capacity headroom.
19
What metrics would you monitor to assess the performance of a load balancer?
KnownRevise
Monitor a load balancer from four angles: - Traffic: request rate (RPS), new and active connections, throughput (bytes in and out), TLS handshakes per second. - Latency: total request latency (p50, p95, p99), split into time spent in the load balancer and back-end response time, plus connect time to back ends and TLS handshake time. - Errors:5xx split by origin: generated by the LB itself (502 bad gateway, 503 no healthy back ends, 504 timeout) versus returned by back ends. Also 4xx rates, connection errors, resets, rejected connections, TLS failures and retry counts. - Saturation and health: LB CPU and memory (self-managed), connection table usage, queue lengths (requests waiting for a back-end slot), port exhaustion, healthy versus unhealthy back ends, health-check failures, ejections, and capacity units for managed LBs.
Also watch distribution quality: requests and connections per back end to detect imbalance from stickiness, hashing or weights. These map to the four golden signals: latency, traffic, errors and saturation. Alert on user-facing symptoms (error rate and p99 latency against SLOs), and use the other metrics for diagnosis.
A large gap between rt (total) and urt (upstream) points to slow clients or the proxy itself. A large uct points to back-end connection problems.
Diagnosing With Metrics
Symptom
Likely cause
LB 504s rising, back-end latency high
Slow back ends or timeouts set too low
LB 502s, back ends report no errors
Back ends closing keep-alive connections, crashes, protocol mismatch
503 “no healthy upstream”
All back ends failing checks
One back end at 3× the requests
Stickiness, hash hot key, weight misconfiguration
Rejected connections
LB connection limit or capacity exhausted
Pitfalls
Watching averages instead of percentiles.
Not separating LB-generated from back-end errors.
Missing saturation signals (queues), so problems surface only as errors.
Alerting on every metric rather than on SLO-related symptoms.
Interview Tips
Organise by golden signals: traffic, latency, errors and saturation, plus health and distribution.
Name concrete metrics (CloudWatch, HAProxy, Envoy).
Explain how metrics help diagnose issues and how to alert on SLOs.
20
Describe one method to handle SSL/TLS traffic in load balancing.
KnownRevise
One common method is TLS termination (SSL offloading) at the load balancer. The load balancer holds the certificate and private key, completes the TLS handshake with the client, decrypts the request, and forwards it to the back end either as plain HTTP on a private network or re-encrypted (TLS re-encryption, also called TLS bridging). Benefits: - Offloads CPU-heavy handshakes from application servers. - Centralises certificate management, including automated renewal (ACM, Let’s Encrypt, cert-manager). - Enables L7 features: routing by path, header or cookie, cookie stickiness, WAF inspection, compression, caching, HTTP/2 and HTTP/3 to clients. - Applies consistent TLS policy in one place (TLS 1.2+/1.3, strong ciphers, HSTS, OCSP stapling).
Trade-offs: traffic between the load balancer and back ends is unencrypted unless re-encrypted, which many compliance regimes and zero-trust designs disallow. The load balancer becomes a high-value target holding private keys. Back ends must learn the original client details from X-Forwarded-For and X-Forwarded-Proto headers. Alternatives are TLS passthrough (L4: the back end terminates TLS, routing by SNI only) and end-to-end re-encryption or mTLS to back ends (service meshes).
Trusting X-Forwarded-For from clients, allowing IP spoofing.
Interview Tips
Describe termination: the LB handles the handshake, decrypts, and forwards.
Give benefits (offload, central certificates, L7 features) and risks (plaintext back-end hop, key protection).
Mention re-encryption and passthrough as alternatives.
21
Explain the concept of direct server return and its significance in load balancing.
KnownRevise
Direct server return (DSR) is an L4 load balancing mode in which requests pass through the load balancer, but responses go from the back-end server directly to the client, bypassing the load balancer. How it works (the classic L2 DSR, or “direct routing”): 1. The client sends a packet to the virtual IP (VIP). 2. The load balancer picks a server and rewrites only the destination MAC address, forwarding the packet unchanged at L3. Tunnelled DSR variants (IP-in-IP, GRE, GUE) encapsulate the packet instead, so servers can be on other subnets. 3. Each server has the VIP configured on a loopback interface, with ARP replies for it suppressed, so it accepts packets addressed to the VIP. 4. The server replies directly to the client, using the VIP as its source address, so the client sees a normal conversation.
Significance: in most web workloads, responses are far larger than requests (videos, downloads, pages). With DSR the load balancer only handles the small inbound traffic, so it can serve much higher throughput with less bandwidth and CPU, and adds less latency on the response path. That is why DSR is used by CDNs, streaming, gaming and large L4 balancers: Linux IPVS -g (gatekeeper/direct routing) mode, Facebook’s Katran, GitHub’s GLB Director and F5 nPath.
Read full answerHide full answer
Limitations: the load balancer sees only one direction, so no TLS termination, HTTP inspection, response rewriting, cookie stickiness or L7 features. It needs network configuration on every server (loopback VIP, ARP settings, or tunnel decapsulation). Port translation is generally not possible. Troubleshooting and health tracking are harder, and asymmetric routing can trip firewalls and stateful devices.
# On the load balancer (director)
ipvsadm -A -t 198.51.100.10:443 -s wlc # virtual service, weighted least connections
ipvsadm -a -t 198.51.100.10:443 -r 10.0.0.21:443 -g -w 1 # -g = direct routing (DSR)
ipvsadm -a -t 198.51.100.10:443 -r 10.0.0.22:443 -g -w 1
# On each real server: accept the VIP, but never answer ARP for it
ip addr add 198.51.100.10/32 dev lo
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
With -g, the director and real servers must share an L2 segment. IPVS tunnelling mode (-i, IP-in-IP) removes that requirement.
DSR Versus Other Modes
Mode
Return path
LB load
L7 features
Server changes
Full proxy (L7)
Through LB
High (both directions, TLS)
Yes
None
NAT (L4)
Through LB
Medium
No
Default gateway is the LB
DSR
Direct to client
Low (inbound only)
No
VIP on loopback, ARP tuning or decapsulation
When to Use DSR
Good fit
Poor fit
High-bandwidth responses (video, downloads, game state)
Environments where server network config is restricted (many managed clouds)
A common pattern combines both: an L4 DSR tier spreads traffic across a fleet of L7 proxies, which terminate TLS.
Pitfalls
ARP problems: servers answering ARP for the VIP, causing traffic to bypass the LB unpredictably.
Firewalls dropping asymmetric flows.
Expecting L7 features in DSR mode.
MTU issues with tunnelled DSR (encapsulation overhead).
Interview Tips
Define DSR: requests via the LB, responses directly to the client.
Explain how (MAC rewrite or tunnelling, VIP on loopback, ARP suppression).
Give the significance (throughput for asymmetric traffic) and the limitations (no L7, network setup).
22
How can caching be integrated with load balancing to improve performance?
KnownRevise
Caching and load balancing combine at several layers: - Cache at the load balancer or reverse proxy: nginx proxy_cache, Varnish, HAProxy’s small object cache and Envoy’s cache filter store cacheable responses. Repeated requests are answered without reaching back ends, which reduces latency and back-end load. - CDN in front of the load balancer: edge caches serve static assets and cacheable pages close to users, so the load balancer and origins see only cache misses. - Cache-aware routing to back-end or cache servers:consistent hashing sends each key or URL to the same cache node, which maximises hit rates and avoids duplicating data across nodes, while minimising reshuffling on scaling. Examples: nginx hash $request_uri consistent in front of Varnish nodes, Envoy ring hash or Maglev. - Server-local caches: with stickiness or hashing, each app server keeps warm in-process caches for its users or keys. - Shared distributed caches (Redis, Memcached) behind stateless app servers: any server can use the cache, so the load balancer can route freely.
Supporting techniques: - Cache-Control headers, so every layer knows what to cache and for how long. - Request coalescing (collapsing concurrent misses for the same key) and stale-while-revalidate, to avoid thundering herds. - Vary-correct cache keys, so responses that differ by user, language or encoding aren’t mixed up.
users -> CDN edge cache (static, public pages)
-> L7 load balancer with micro-cache (1–10 s for hot API responses)
-> hash on URL -> cache tier (Varnish nodes) -- miss --> app servers
-> Redis (shared data cache) -> DB
upstream varnish_tier {
hash $request_uri consistent; # same URL -> same cache node; ~1/N keys move on scaling
server 10.0.2.11:6081;
server 10.0.2.12:6081;
server 10.0.2.13:6081;
}
With round robin, each cache node would hold its own copy of every popular object, giving lower hit rates and N× origin fetches on misses.
Integration Options
Layer
Caches
Benefit
Watch out for
CDN
Static assets, public pages, some APIs
Global latency, origin offload
Invalidation, personalised content
LB / reverse proxy
Hot responses, micro-caching
Fewer back-end calls
Cache keys, auth headers
Hash-routed cache tier
Large object sets
High hit rate, no duplication
Hot keys
App-local cache with affinity
Per-user or per-key data
Fastest access
Imbalance, loss on failover
Shared Redis/Memcached
Data and sessions
Stateless app tier
Network hop, cache stampedes
Pitfalls
Caching personalised responses (with Set-Cookie or Authorization), which leaks data between users.
Missing Vary handling (encoding, language).
Thundering herds when popular entries expire: use locks, coalescing or stale serving.
Round robin across cache nodes, which dilutes hit rates.
No invalidation strategy for content that must update quickly.
Cover Cache-Control, coalescing, stale-while-revalidate and personalisation safety.
23
Discuss the challenges of load balancing microservice architectures.
KnownRevise
Microservices turn load balancing from one edge problem into a many-to-many problem: hundreds of services, thousands of instances and east-west traffic between services. The main challenges: - Dynamic endpoints: instances start, stop, autoscale and move constantly, so load balancers need service discovery (Kubernetes endpoints, Consul, Eureka, DNS) and must keep up without sending traffic to dead pods. - Where to balance:centralised proxies (an extra hop, a bottleneck, a single point of failure), client-side libraries (language-specific: gRPC, older Netflix Ribbon), or sidecar or ambient service meshes (Istio, Linkerd, Envoy), which add operational complexity and resource overhead. - Long-lived, multiplexed connections:HTTP/2 and gRPC keep one connection open for many requests. Kubernetes Services balance at L4 per connection, so traffic can pile onto a few pods. You need L7, per-request balancing. - Many independent balancers: every client instance makes its own decisions with partial information. Power of two choices and least outstanding requests cope better than global counters. - Cascading failures and retries: retries at several layers multiply load (retry storms), and slow dependencies tie up resources. This calls for timeouts, retry budgets, circuit breakers, outlier detection and bulkheads. - Heterogeneous services: different protocols, latency profiles and scaling behaviours need different policies. - Deployments: canaries, blue-green and versioned routing across many services, with graceful draining during frequent rollouts. - Security:mTLS and authorisation between services, with certificate rotation. - Observability: distributed tracing, per-service metrics and correlation IDs are needed to understand where latency and errors come from. - Locality and cost: cross-zone traffic adds latency and data-transfer costs, so zone-aware routing helps.
Read full answerHide full answer
Applies toKubernetes, service meshes (Istio, Linkerd), Envoy, gRPC
Typical Layout
internet -> edge LB / API gateway (north-south: TLS, auth, rate limits)
-> service A pods --(sidecar or client LB: per-request, P2C, mTLS)--> service B pods
\--> service C pods
service discovery (Kubernetes API / xDS control plane) keeps endpoint lists current
Relying on kube-proxy for gRPC, so one pod receives most traffic.
Retries everywhere (client, mesh, gateway), multiplying traffic during incidents.
No timeouts, so threads pile up waiting on slow services.
Mesh adoption without the operational capacity to run it.
Ignoring draining, causing errors on every deployment.
Interview Tips
Highlight east-west traffic, dynamic discovery, gRPC/HTTP/2 imbalance and cascading failures.
Compare central proxies, client-side load balancing and service meshes.
Give mitigations: L7 per-request balancing, outlier detection, retry budgets, locality, mTLS and tracing.
24
How do you configure a health check on a load balancer?
KnownRevise
Configuring a health check means deciding what to probe, how often, and when to change a server’s state: 1. Choose the check type:TCP (port accepts connections), HTTP/HTTPS (expect a status code and optionally a body), gRPC health checking protocol, or a custom script. Prefer HTTP for web services, since a TCP check misses hung applications. 2. Expose a health endpoint in the application, such as /healthz (liveness) and /ready (readiness). It should be fast and cheap, check critical local readiness (started, warm, can reach essential dependencies), and not cascade: a shared database blip shouldn’t fail every server at once. 3. Set timing and thresholds: - interval (for example 5 s); - timeout (for example 2 s, shorter than the interval); - unhealthy threshold (fall, for example 3 consecutive failures); - healthy threshold (rise, for example 2 consecutive passes). These trade detection speed against flapping. 4. Define the success criteria: status codes (200 or a range), an optional body match, and the port and host header. 5. Add passive checks and outlier detection (real-traffic errors) to catch problems between probes. 6. Plan recovery:slow start for recovered servers, connection draining for removed ones, and fail-open behaviour if every server fails. 7. Secure and monitor: restrict the endpoint to the load balancer’s network, don’t expose sensitive details, and alert on unhealthy host counts.
# Health-check state machine with rise/fall thresholds (like HAProxy rise 2 fall 3)
class Backend:
def __init__(self, name, rise=2, fall=3):
self.name, self.rise, self.fall = name, rise, fall
self.up, self.streak = True, 0 # streak counts results that disagree with the state
def record(self, ok):
if ok == self.up:
self.streak = 0 # result agrees with the current state
return
self.streak += 1
if self.up and self.streak >= self.fall:
self.up, self.streak = False, 0
elif not self.up and self.streak >= self.rise:
self.up, self.streak = True, 0
b = Backend("app2")
probes = [1, 0, 1, 0, 0, 0, 1, 0, 1, 1] # 1 = probe passed, 0 = failed
for t, ok in enumerate(probes):
b.record(bool(ok))
print(f"probe {t}: {'pass' if ok else 'FAIL'} -> {'UP' if b.up else 'DOWN'}")
# probe 1: FAIL -> UP (one failure is ignored: no flapping)
# probe 5: FAIL -> DOWN (third consecutive failure)
# probe 9: pass -> UP (second consecutive pass)
Single failed probes are ignored, so a server isn’t removed for one blip. Recovery needs consecutive passes, so a flapping server stays out.
HAProxy
backend app
option httpchk
http-check send meth GET uri /ready ver HTTP/1.1 hdr Host app.internal
http-check expect status 200
default-server inter 5s fastinter 1s downinter 10s fall 3 rise 2 slowstart 30s
server app1 10.0.1.11:8080 check
server app2 10.0.1.12:8080 check
fastinter probes faster while a server is changing state; downinter probes less often while it is down.
Open-source nginx only has passive checks (max_fails, fail_timeout); active health_check is an NGINX Plus feature.
Choosing Settings
Setting
Lower value
Higher value
Interval
Faster detection, more probe traffic
Less load, slower detection
Timeout
Detects slow servers
Tolerates brief slowness
Unhealthy threshold
Quick removal, risk of flapping
Stable, slower removal
Healthy threshold
Quick return
Ensures real recovery
Worst-case detection time ≈ interval × unhealthy threshold (plus timeout): 5 s × 3 = about 15 s.
What the Endpoint Should Check
Check
Include?
Process up and serving HTTP
Yes
Warm-up done (caches, connections)
Yes (readiness)
Local disk or memory critically low
Yes
Shared database reachable
Carefully: often better as degraded mode than a failure
Third-party APIs
Usually no
Pitfalls
TCP-only checks for HTTP apps (miss hung processes).
Deep checks on shared dependencies, so a database blip removes every server: a self-inflicted outage.
Heavy health endpoints that add load.
Liveness probes that check dependencies, causing restart loops.
Thresholds too low, so servers flap.
Interview Tips
Walk through type, endpoint, interval, timeout, thresholds and success criteria.
Show a concrete config (HAProxy, ALB, Kubernetes).
Discuss readiness versus liveness, shallow versus deep checks, passive checks and slow start.
25
What is connection draining, and why is it used in load balancing?
KnownRevise
Connection draining (called deregistration delay in AWS, and graceful drain or “draining” mode in HAProxy, Envoy and F5) is how a load balancer removes a back end gracefully: - it stops sending new requests or connections to the server; - it lets existing in-flight requests and open connections finish until they complete or a drain timeout expires; - only then does it fully remove the server.
It is used so that routine operations don’t break user requests: - deployments and rolling updates (old instances drain while new ones take over); - autoscaling scale-in; - maintenance and patching; - instance replacement (spot interruption notices, host retirements).
Read full answerHide full answer
Without draining, removing a server would cut active requests (failed uploads, half-finished checkouts, dropped WebSockets), causing 5xx errors and resets on every deploy. The drain timeout should match typical request duration: short (10–30 s) for APIs, longer for uploads, while long-lived connections like WebSockets need application help, such as asking clients to reconnect. Draining works together with the application’s graceful shutdown: on SIGTERM it fails readiness, finishes work, then exits.
Applies to: AWS ELB, HAProxy, Envoy, nginx, Kubernetes
Lifecycle
t=0 deploy starts: instance i-2 deregistered state: draining
LB sends no new requests to i-2; 14 requests still in flight
t=0-8s in-flight requests complete normally
t=8s no active requests -> i-2 removed (or at deregistration delay of 30 s at the latest)
t=8s+ instance terminated
Configuration Examples
# AWS target group: wait up to 30 s for in-flight requests (default 300 s)
aws elbv2 modify-target-group-attributes --target-group-arn "$TG" \
--attributes Key=deregistration_delay.timeout_seconds,Value=30
# HAProxy runtime API: put app2 in drain mode (no new traffic, existing sessions continue)
echo "set server app/app2 state drain" | socat stdio /run/haproxy/admin.sock
echo "show servers state app" | socat stdio /run/haproxy/admin.sock # watch connections fall
echo "set server app/app2 state maint" | socat stdio /run/haproxy/admin.sock
# Kubernetes: delay shutdown so endpoints update, and allow time to finish work
spec:
terminationGracePeriodSeconds: 45
containers:
- name: api
lifecycle:
preStop:
exec:
command: ["sleep", "10"] # endpoint removal propagates before SIGTERM
Kubernetes removes a terminating pod from Service endpoints, but that update reaches kube-proxy, ingress controllers and cloud LBs asynchronously. The preStop sleep covers that gap.
Choosing the Drain Timeout
Workload
Typical timeout
Note
Fast APIs
10–30 s
Default 300 s in AWS slows deploys unnecessarily
File uploads, reports
Minutes
Or design for resumable uploads
WebSockets, streaming
Application-driven reconnect
Send a close or reconnect message
Batch workers
Until the current job finishes
Pull-based queues: stop taking jobs
Draining Versus Related Concepts
Concept
Meaning
Connection draining
No new traffic; finish existing
Health-check failure
Server removed because it failed probes (may cut connections)
Maintenance mode
Fully out of rotation
Slow start
Opposite direction: gradually add traffic to a new server
Pitfalls
Drain timeout too short, cutting long requests.
Drain timeout too long, so deployments and scale-in crawl.
App exits immediately on SIGTERM, ignoring the drain.
Sticky sessions that keep users on draining servers longer than expected.
Long-lived connections never closing, so the drain always hits its timeout.
Interview Tips
Define draining: stop new traffic, let existing traffic finish, then remove.
Explain why: zero-downtime deployments, scale-in and maintenance.
Mention timeouts, deregistration delay, preStop hooks and app graceful shutdown.
26
How do cloud load balancers like AWS ELB differ from on-premise solutions?
KnownRevise
Cloud load balancers (AWS Elastic Load Balancing: ALB, NLB, GWLB; Azure Load Balancer and Application Gateway; Google Cloud Load Balancing) are managed services: - the provider runs, patches and scales them automatically, with built-in high availability across zones; - they are configured through APIs, consoles and infrastructure as code; - they are billed pay-as-you-go (hours plus capacity units or data processed); - they are tightly integrated with autoscaling groups, certificate managers (ACM, Key Vault), WAF and DDoS protection (AWS WAF and Shield, Azure Front Door, Cloud Armor), IAM, monitoring and container platforms.
On-premise solutions are hardware appliances (F5 BIG-IP, Citrix NetScaler, A10) or software you run yourself (HAProxy, nginx, Envoy, Linux IPVS, Kemp). You own sizing, redundancy (active-passive pairs), upgrades, monitoring and capacity planning. In return you get full control over features and tuning, often richer or more customisable functionality (iRules scripting, advanced traffic manipulation, full TCP and TLS tuning), predictable costs at steady high volume, and the ability to meet data-locality or regulatory constraints.
Read full answerHide full answer
Trade-offs: - Cloud: less control, provider-specific limits and features, per-GB costs at scale, and lock-in. - On-premise: upfront capital cost, slower scaling, and operational burden.
Many organisations use both: cloud LBs at the edge, and self-managed software proxies (Envoy, HAProxy, nginx) inside clusters, or hybrid setups.
Applies to: AWS ELB, Azure Load Balancer and Application Gateway, Google Cloud LB versus F5, Citrix, HAProxy, nginx
Comparison
Aspect
Cloud managed LB
On-premise (appliance or self-hosted software)
Operation
Provider runs, patches and scales
You install, patch, monitor and scale
Scaling
Automatic and elastic (pre-warming rarely needed)
Capacity planned; buy or deploy more
HA
Built-in multi-zone
You build it (active-passive pairs, VRRP, clustering)
Cost model
Opex: hourly + LCU/data processed
Capex (hardware, licences) + staff; cheaper at steady high volume
Vendor-specific (appliances) or portable (open source)
Compliance
Shared responsibility model
Full control over location and data
AWS ELB Types
Type
Layer
Use
Application Load Balancer
L7 HTTP/HTTPS, gRPC
Path, host and header routing; WAF; Lambda targets
Network Load Balancer
L4 TCP/UDP/TLS
Very high throughput, static IPs, low latency
Gateway Load Balancer
L3 (GENEVE)
Inserting firewall and inspection appliances
Classic Load Balancer
L4/L7 legacy
Previous generation, being retired
Example: Cloud LB as Infrastructure as Code (Terraform)
resource "aws_lb" "web" {
name = "web-alb"
load_balancer_type = "application"
subnets = var.public_subnet_ids # spread across availability zones
security_groups = [aws_security_group.alb.id]
}
resource "aws_lb_target_group" "app" {
name = "app"
port = 8080
protocol = "HTTP"
vpc_id = var.vpc_id
health_check { path = "/ready" }
}
The equivalent on-premise setup means installing and configuring HAProxy or an F5 pair, configuring keepalived or VRRP for a floating IP, and planning capacity.
Choosing
Choose cloud LB when
Choose on-premise or self-managed when
Workloads run in that cloud
Workloads are in your data centre
Elastic, variable traffic
Steady, very high traffic where per-GB fees dominate
Small ops team
Need deep customisation or specific vendor features
Fast provisioning matters
Strict data-locality or air-gapped environments
Pitfalls
Assuming infinite instant scale: cloud LBs still have quotas, and sudden spikes may need provider coordination.
Ignoring data-processing costs at high volume.
Single on-premise appliance without an HA pair.
Lifting appliance-specific scripting into clouds that cannot replicate it.
Interview Tips
Contrast managed, elastic, integrated, opex with controlled, customisable, capex, self-operated.
Name ALB, NLB and GWLB and their layers.
Discuss cost, compliance and hybrid patterns.
27
Explain the role of load balancing in auto-scaling groups and clusters.
KnownRevise
Load balancing is the front door that makes auto-scaling useful. Auto-scaling changes the number of instances based on demand (CPU, request count, queue depth, schedules), and the load balancer: - Registers new instances automatically once they pass health checks, then spreads traffic across them, optionally with slow start so cold instances aren’t flooded. - Drains and deregisters instances during scale-in, so in-flight requests complete before termination. - Provides a stable endpoint (DNS name or VIP), so clients never need to know which instances exist. - Supplies health signals: in AWS, an Auto Scaling group can use ELB health checks to replace instances that the load balancer marks unhealthy, not just ones whose host failed. - Supplies scaling metrics: request count per target, latency or connection counts can drive scaling policies (AWS ALBRequestCountPerTarget, Kubernetes HPA with custom metrics). - Balances across zones, so scaling and failures in one zone don’t overload others.
In Kubernetes clusters, the same roles are split: the HPA scales pods, the cluster autoscaler or Karpenter scales nodes, and Services, Ingress or Gateway controllers route to ready pods only. Together they give elasticity without client-side changes.
Read full answerHide full answer
Applies toAWS Auto Scaling groups with ELB, Azure VM Scale Sets, GCP managed instance groups, Kubernetes
Lifecycle
scale-out: metric high -> ASG launches instance -> boots, app warms up
-> LB health check passes (healthy threshold) -> registered -> traffic (slow start)
scale-in: metric low -> ASG picks instance -> LB deregisters (draining, delay 30 s)
-> in-flight requests finish -> instance terminated
failure: LB health check fails -> ASG (ELB health type) replaces the instance
Example: AWS Target Tracking on Requests per Target
The ASG keeps about 800 requests per instance per minute. With ELB health checks, an instance that fails the load balancer’s checks is replaced. The grace period stops new instances being killed while they boot.
Responsibilities
Concern
Auto-scaler
Load balancer
How many instances
Yes
No (provides metrics)
Which instance gets a request
No
Yes
Adding new capacity safely
Launches
Health checks, slow start, registration
Removing capacity safely
Terminates
Draining / deregistration delay
Replacing broken instances
Yes (with LB health info)
Stops routing to them
Stable client endpoint
No
Yes
Tuning Tips
Make health-check grace periods and readiness reflect real start-up and warm-up time.
Keep scale-in protection or cooldowns to avoid thrashing.
Choose scaling metrics that reflect user load (requests per target, latency) rather than only CPU.
Use multiple zones and cross-zone balancing.
Pre-scale for known events, because autoscaling reacts with a delay.
Pitfalls
New instances receiving traffic before they are warm: high latency or errors on scale-out.
No deregistration delay, so scale-in drops requests.
Health checks that fail under load, causing replacement cascades.
Stateful instances in groups that scale in, losing sessions.
Interview Tips
Explain how the LB registers, health-checks, drains and spreads traffic as instances come and go.
Mention ELB health checks driving replacement and LB metrics driving scaling.
Cover the Kubernetes equivalents: HPA, cluster autoscaler, readiness and Services.
28
Describe the process of load balancing in a multi-cloud environment.
KnownRevise
In a multi-cloud environment, an application runs on two or more providers (for example AWS and Azure, or GCP plus Oracle Cloud). Load balancing works in layers: 1. Global traffic steering across clouds: a provider-neutral DNS/GSLB or edge service decides which cloud and region serves each user. Examples: Cloudflare Load Balancing, NS1, Akamai, F5 Distributed Cloud, or Route 53 used neutrally. It routes by health, geography, latency, weights or cost, and fails over if one cloud degrades. 2. Regional load balancing inside each cloud: each cloud’s native LB (ALB/NLB, Azure Application Gateway or Load Balancer, Google Cloud LB), or self-managed proxies, distributes traffic to instances, containers or functions there. 3. Service-to-service traffic: inside and between clouds, a service mesh (Istio multi-cluster, Consul, Cilium Cluster Mesh) or private connectivity (VPN, Interconnect, ExpressRoute, Direct Connect, or a cloud exchange) handles east-west traffic with mTLS, locality-aware routing and failover.
The process also needs: - consistent health checks that test the full stack in each cloud; - shared configuration as code (Terraform, Kubernetes manifests); - centralised observability; - a data strategy: replication across clouds, which consistency model, and which writes go where.
Read full answerHide full answer
Without the data strategy, active-active routing breaks consistency.
users -> global DNS/GSLB or anycast edge (Cloudflare LB: health checks against both clouds)
|-- 60% --> AWS us-east-1: ALB -> EKS services
|-- 40% --> Azure East US: Application Gateway -> AKS services
(failover: if AWS pool unhealthy, 100% to Azure)
east-west: Istio multi-cluster / private interconnect with mTLS
data: replicated database (e.g. CockroachDB, YugabyteDB) or a primary in one cloud with replicas elsewhere
Each provider has its own exact syntax (Cloudflare pools and monitors, NS1 filter chains), but the concepts are the same: pools per cloud, health monitors, a steering policy and a fallback.
Steps in Practice
Step
Details
Define goals
Resilience to a provider outage, latency, cost, regulation, avoiding lock-in
Choose a global layer
Neutral DNS/edge (not dependent on one cloud)
Standardise the platform
Kubernetes, containers, IaC across clouds
Configure regional LBs
Native or portable (Envoy, nginx, HAProxy) per cloud
IaC, GitOps, shared observability, practice drills
DNS caching slowing failover
Short TTLs; anycast edge proxies
Pitfalls
Global DNS hosted in one of the clouds you are trying to survive losing.
Active-active without data replication that supports it.
Untested failover, which fails when needed.
Unexpected egress bills from cross-cloud traffic.
Interview Tips
Describe the layers: global steering, per-cloud LBs, east-west mesh.
Stress neutral health-checked DNS/edge, consistent IaC and data strategy.
Mention costs, latency and failover testing.
29
How does load balancing work in a hybrid cloud with private and public resources?
KnownRevise
In a hybrid cloud, an application spans a private environment (on-premise data centre or private cloud) and a public cloud. Load balancing then has to steer traffic between environments, as well as within each one. Common patterns: - Global or DNS-level steering: a GSLB (F5 BIG-IP DNS, Cloudflare, NS1, Route 53) or an edge service directs users to on-premise or cloud endpoints by health, weights, geography or capacity. - Cloud bursting: the private data centre handles baseline load. When it nears capacity, overflow goes to the public cloud, using weights, priority pools or an on-premise LB with cloud back ends in its pool. - Active-passive DR: on-premise is primary and the cloud is a warm standby (or the reverse). Health-checked failover switches traffic. - Migration:weighted shifting (90/10 → 50/50 → 0/100) moves traffic gradually from on-premise to cloud. - Tiered split: the front end runs in the cloud near users, behind a cloud LB, while some back ends (mainframes, regulated data stores) stay on-premise, reached over private links (Direct Connect, ExpressRoute, Cloud Interconnect, VPN) through internal load balancers.
Key concerns: - network connectivity and latency between environments; - consistent security (TLS, identity, firewalls); - data locality and compliance; - health checks that see both sides; - unified observability.
backend web
balance leastconn
# Private servers take traffic first
server dc1 10.10.0.11:8080 check maxconn 500
server dc2 10.10.0.12:8080 check maxconn 500
# Cloud capacity, used only when all private servers are down
server aws1 10.200.1.21:8080 check backup
server aws2 10.200.1.22:8080 check backup
backup servers are used only when no primary server is available, which fits failover. For overflow while private servers are merely busy, keep separate private and cloud backends and switch with an ACL:
frontend fe
bind :80
acl dc_busy avg_queue(web_private) gt 20
use_backend web_cloud if dc_busy
default_backend web_private
Patterns Compared
Pattern
Traffic split
Benefit
Watch out for
Cloud bursting
Private first, overflow to cloud
Pay for cloud only at peaks
Data access latency from cloud to on-prem
Active-passive DR
All to primary; failover
Resilience
Standby freshness, failover tests
Active-active
Split by geo or weight
Capacity and resilience
Data replication and consistency
Migration
Gradual weight shift
Low-risk move
Two environments to operate
Tiered split
Front end in cloud, data on-prem
Compliance
Private link capacity and latency
Connectivity and Security
Private links with redundancy, for predictable latency and bandwidth.
Consistent IP planning: no overlapping CIDRs.
mTLS or IPsec between environments; unified identity (federated IAM, workload identity).
Firewall rules that allow health checks from both sides.
Pitfalls
Bursting stateless front ends that depend on on-premise data, so latency jumps and the link saturates.
Health checks that only see one side.
Single private link as a single point of failure.
Different configs drifting between environments.
Interview Tips
Describe GSLB steering, bursting, DR, migration and tiered splits.
Show concrete config (backup or overflow pools, weights).
Emphasise connectivity, data locality, security and consistent observability.
30
What is cross-region load balancing, and in what scenario would it be necessary?
KnownRevise
Cross-region load balancing distributes traffic across multiple geographic regions (for example us-east-1, eu-west-1 and ap-south-1) rather than only across zones or servers in one region. It works through: - DNS-based routing (latency, geo, weighted, failover policies); - anycast global load balancers that advertise one IP worldwide and route to the nearest healthy region (Google Cloud global external Application Load Balancer, AWS Global Accelerator, Azure Front Door, Cloudflare); - global edge proxies that terminate connections near users and forward to a chosen region.
Regional load balancers then distribute traffic to servers within each region. Scenarios where it is necessary: - Regional disaster recovery: surviving the outage of an entire region (cloud incidents, natural disasters), where business requirements set an RTO and RPO that one region cannot meet. - Global users and latency: serving users on different continents within latency targets, such as under 100 ms. - Data residency and regulation: EU users served and stored in the EU (GDPR), and in-country requirements in India, China and elsewhere. - Capacity: a single region’s quotas or capacity are not enough for peak traffic. - Follow-the-sun or cost strategies, and gradual regional rollouts of new versions.
Read full answerHide full answer
It brings complexity: data replication across regions (latency, consistency choices, conflict resolution), capacity headroom to absorb a failed region, cross-region data transfer costs and harder testing.
Applies to: AWS Global Accelerator and Route 53, Google Cloud global LB, Azure Front Door and Traffic Manager, Cloudflare
Architecture
anycast IP / latency DNS
users EU ---------> eu-west-1 ALB -> app -> DB replica (or regional primary)
users US ---------> us-east-1 ALB -> app -> DB primary
users APAC -------> ap-south-1 ALB -> app -> DB replica
region outage: health checks fail -> traffic shifts to the next nearest healthy region
Global Accelerator gives two static anycast IPs. Traffic enters AWS’s network at the nearest edge and goes to the closest healthy endpoint group. For ALB and NLB endpoints, Global Accelerator uses the load balancer’s own target health, so a region whose targets are unhealthy is skipped. The traffic dial reduces a region’s share (for example to 0% during maintenance), and failover doesn’t depend on DNS TTLs.
Active-Active Versus Active-Passive
Model
How
Pros
Cons
Active-active
All regions serve traffic
Low latency everywhere; capacity used
Multi-region data writes, conflicts
Active-passive
One primary; failover to standby
Simpler data (single writer)
Standby cost, failover time, untested standby
Read-local, write-global
Reads in every region, writes to one
Good compromise
Write latency for distant users
When It Is Necessary
Scenario
Necessary?
Regional SaaS with small user base, RTO of hours
No: multi-AZ is enough
Global consumer app with strict latency targets
Yes
Banking or healthcare with regional outage SLAs
Yes
EU and US customers with residency rules
Yes (often with per-region data partitioning)
Internal tool used in one country
Usually no
Pitfalls
Multi-region front ends with a single-region database, so a region outage still takes the app down.
No capacity headroom, so the failover region collapses.
Ignoring data transfer costs and replication lag.
Never running failover drills.
Interview Tips
Define it as distributing traffic across regions, via DNS, anycast or global proxies.
Give scenarios: region DR, global latency, data residency, capacity.
Discuss data replication, headroom, active-active versus active-passive and costs.
31
Explain how a load balancer can make routing decisions based on content type.
KnownRevise
An L7 (application-layer) load balancer terminates the connection, parses the HTTP request, and chooses a back-end pool based on its content. This is called content-based routing. Signals it can use: - the URL path or file extension (/static/*, *.jpg, /api/*, /video/*); - the Host header or SNI (virtual hosting: api.example.com versus www.example.com); - the request Content-Type (for example application/grpc, multipart/form-data uploads, application/json); - the Accept header (requests for text/html versus application/json or images); - other headers and cookies (API version, user tier, Upgrade: websocket, A/B cohort); - the HTTP method and query parameters.
This lets each kind of content go to the best-suited servers: - static files to a cache or object store; - images to an image-processing service; - uploads to servers with large body limits; - gRPC to HTTP/2-capable back ends; - WebSockets to connection-optimised servers; - APIs to application servers.
Read full answerHide full answer
Benefits: independent scaling, specialised tuning, caching and simpler services. Trade-offs: L7 processing cost (TLS termination and parsing), more complex configuration, and headers can be spoofed, so never use them alone for security decisions. L4 load balancers cannot do this because they don’t see HTTP content.
Applies to: nginx, HAProxy, Envoy, AWS ALB listener rules, Kubernetes Ingress and Gateway API
Example: HAProxy Content Routing
frontend web
bind :443 ssl crt /etc/haproxy/certs/site.pem alpn h2,http/1.1
acl is_static path_end -i .css .js .png .jpg .webp .svg .woff2
acl is_upload req.hdr(content-type) -m beg multipart/form-data
acl is_grpc req.hdr(content-type) -m beg application/grpc
acl is_ws req.hdr(upgrade) -i websocket
acl is_api path_beg /api/
use_backend static_cache if is_static
use_backend uploads if is_upload
use_backend grpc_services if is_grpc
use_backend websockets if is_ws
use_backend api if is_api
default_backend web_app
backend grpc_services
server g1 10.0.3.11:50051 check proto h2 # HTTP/2 to back end for gRPC
Relying on client-supplied headers for security (authorisation must happen in the service).
Rule ordering mistakes: a broad rule matches before a specific one.
Inconsistent caching when responses vary by Accept without Vary: Accept.
Too many complex rules, making config hard to reason about. Prefer clear path conventions.
Interview Tips
Explain L7 parsing and the signals: path, host, Content-Type, Accept, headers, method.
Give examples (static vs API, gRPC, uploads, WebSockets) with config.
Mention benefits (specialisation, scaling) and trade-offs (cost, complexity, spoofing).
32
How would you enable graceful shutdown of servers in a load-balanced environment?
KnownRevise
Graceful shutdown means removing a server without failing any requests. It needs the load balancer and the application to cooperate: 1. Stop new traffic first: deregister the server or put it in drain mode at the load balancer, or have the app fail its readiness check, so the LB marks it out after the unhealthy threshold. 2. Wait for propagation: load balancers, kube-proxy and ingress controllers learn about the change asynchronously, so keep serving for a short period (Kubernetes preStop sleep, or about interval × fall threshold). 3. Finish in-flight work: the app handles SIGTERM by stopping its listener from accepting new connections, completing active requests, closing keep-alive connections (respond with Connection: close), telling WebSocket clients to reconnect, and finishing or requeueing background jobs. 4. Respect timeouts: the LB’s drain or deregistration delay and the platform’s termination grace period must be longer than the app’s shutdown window, and longer than typical request duration. 5. Release resources: flush logs and metrics, close database connections and commit offsets. 6. Exit, and only then terminate the instance or container.
Also use rolling deployments with max-unavailable limits, health-checked new instances before old ones go, and slow start for replacements.
# Graceful shutdown: fail readiness, wait for the LB to notice, finish in-flight work, then exit
import signal, threading, time, urllib.request, urllib.error
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
draining = threading.Event()
inflight = 0
lock = threading.Lock()
class Handler(BaseHTTPRequestHandler):
def log_message(self, *args):
pass
def do_GET(self):
global inflight
if self.path == "/ready": # the LB health check
code = 503 if draining.is_set() else 200
self.send_response(code); self.end_headers()
return
with lock:
inflight += 1
time.sleep(0.5) # simulated slow request
self.send_response(200); self.end_headers(); self.wfile.write(b"done")
with lock:
inflight -= 1
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()
def shutdown(signum=None, frame=None):
draining.set() # 1. readiness now fails
time.sleep(0.3) # 2. wait for LB to stop sending (fall x interval)
while True: # 3. finish in-flight requests
with lock:
if inflight == 0:
break
time.sleep(0.05)
server.shutdown() # 4. stop accepting and exit
print("shutdown complete, in-flight requests finished")
signal.signal(signal.SIGTERM, shutdown) # Kubernetes and systemd send SIGTERM
def get(path):
try:
with urllib.request.urlopen(f"http://127.0.0.1:{port}{path}") as r:
return r.status, r.read().decode()
except urllib.error.HTTPError as e:
return e.code, ""
results = []
slow = threading.Thread(target=lambda: results.append(get("/work")))
slow.start()
time.sleep(0.1) # request is now in flight
print("ready before:", get("/ready")[0]) # 200
t = threading.Thread(target=shutdown) # same as receiving SIGTERM
t.start()
time.sleep(0.1)
print("ready while draining:", get("/ready")[0]) # 503
slow.join(); t.join()
print("in-flight request result:", results[0]) # (200, 'done')
The in-flight request started before shutdown completes successfully, and the readiness endpoint returns 503, so the load balancer stops sending new requests.
Sequence: the pod is marked terminating → removed from endpoints (asynchronously) → preStop sleep → SIGTERM → app drains → exits, or receives SIGKILL after 60 s.
Server and Framework Settings
Server
Graceful option
nginx
nginx -s quit (SIGQUIT): workers finish current requests
HAProxy
set server ... state drain; soft-stop on SIGUSR1 (with hard-stop-after)
Puma
SIGTERM stops accepting and finishes requests (persistent_timeout, worker shutdown timeout)
Gunicorn
SIGTERM with --graceful-timeout
Node.js
server.close(), plus closing idle keep-alive connections (server.closeIdleConnections())
Go
http.Server.Shutdown(ctx)
Spring Boot
server.shutdown=graceful with spring.lifecycle.timeout-per-shutdown-phase
Timeout Alignment
Timer
Example
Rule
Health check detection (interval × fall)
5 s × 1
Or deregister explicitly
preStop delay
15 s
≥ endpoint propagation time
App drain timeout
30 s
≥ typical long request
LB deregistration delay
45 s
≥ preStop + app drain
Termination grace period
60 s
> everything above
Pitfalls
App exits immediately on SIGTERM, so in-flight requests fail with 502.
SIGTERM arrives before the LB stops routing, so new requests hit a closing server: add the preStop delay.
PID 1 issues in containers: shell wrappers that don’t forward signals (use exec or an init such as tini).
Grace periods shorter than drain timeouts, so SIGKILL cuts work.
Long-lived connections never closed, so every shutdown hits the timeout.
Interview Tips
Give the ordered steps: stop new traffic, propagate, drain in-flight work, release resources, exit.
Show app signal handling and readiness failure.
Align timeouts across LB, platform and app; mention preStop hooks and PID 1 signal handling.
33
Discuss the role of API gateways in conjunction with load balancers.
KnownRevise
An API gateway and a load balancer often sit together in the request path, but they do different jobs. A load balancer focuses on distributing traffic across healthy instances: L4 or L7, health checks, TLS termination, high availability and scaling. An API gateway is an API management layer (Kong, AWS API Gateway, Apigee, Azure API Management, Tyk, Envoy-based gateways) that handles cross-cutting API concerns: - authentication and authorisation (API keys, OAuth 2.0 and JWT validation); - rate limiting and quotas per client or plan; - request routing to microservices by path, method or version; - request and response transformation, protocol translation (REST to gRPC), and aggregation; - caching, API versioning, a developer portal, usage analytics and monetisation.
They combine in layers. A typical layout is: edge load balancer or CDN (TLS, DDoS and WAF, spreading load across gateway instances) → API gateway (auth, limits, routing) → internal load balancing to service instances, done by the gateway itself, Kubernetes Services or a mesh. The load balancer makes the gateway highly available and scalable, and the gateway turns many services into one coherent API. Some products blur the lines: an ALB can do path routing and OIDC authentication, and a gateway such as Kong or Envoy also load-balances upstreams. The decision depends on which features are needed and where.
Read full answerHide full answer
Applies toKong, AWS API Gateway, Apigee, Azure API Management, Envoy Gateway, NGINX
Gateway as a single point of failure: run several instances behind a load balancer.
Business logic in the gateway, making it a bottleneck “monolith”.
Duplicated auth and limits at many layers, inconsistently configured.
Extra latency from too many hops (CDN → LB → gateway → mesh → service).
Interview Tips
Distinguish traffic distribution and availability (LB) from API management (gateway).
Draw the layered path and explain how the LB makes the gateway highly available.
Mention overlaps and avoiding single points of failure.
34
What is load balancer affinity, and when might you enforce it?
KnownRevise
Load balancer affinity is a rule that binds a client, session or key to a specific back end, so related requests keep going to the same server instead of being freely distributed. Forms: - Session (client) affinity / sticky sessions: bound by a cookie (LB-generated or application cookie), or by source IP (ClientIP affinity in Kubernetes Services, source-IP hash). - Key-based affinity:hashing a header, user ID, tenant ID or URL (consistent hashing) to choose the server. - Connection affinity: inherent in long-lived connections (WebSockets, gRPC streams), where all messages on a connection go to one server. - Locality affinity: preferring servers in the same zone or region as the client or caller (topology-aware routing), which reduces latency and cost.
When to enforce it: - Server-local session state (legacy apps storing sessions in memory or on local disk). - Multi-step flows (checkout, file uploads in chunks, SAML or OAuth flows that keep state locally). - Cache locality: per-user or per-key data warm on one server (recommendation models, per-tenant caches). - Stateful real-time systems: game rooms, chat rooms or collaborative documents held in one server’s memory. - Protocol requirements: Socket.IO polling fallback, some NTLM or Kerberos setups that authenticate per connection. - Experiments: keeping a user in the same A/B or canary cohort.
Read full answerHide full answer
Avoid affinity when services are stateless: it unbalances load, causes session loss on failover, slows scale-in and deployments, and IP affinity breaks behind NAT and proxies. Prefer externalised state, using affinity as a performance optimisation, not a correctness crutch.
Applies to: HAProxy, nginx, AWS ALB stickiness, Kubernetes Services and Ingress, Envoy
Configuration Examples
backend app
balance roundrobin
cookie SRV insert indirect nocache httponly secure # LB-inserted affinity cookie
option redispatch # rebind if the server is down
server app1 10.0.1.11:8080 check cookie a1
server app2 10.0.1.12:8080 check cookie a2
Redispatch on failure, and accept that state on a failed server is lost.
Drain servers before removal, so pinned users finish.
Monitor per-server load for skew.
Plan to remove the need for affinity (Redis sessions, JWTs).
Pitfalls
IP affinity for mobile and corporate users.
Infinite stickiness, so new servers get no existing users after scaling out.
Treating affinity as reliability.
Cookie affinity behind caches that share cookies between users.
Interview Tips
Define affinity as binding a client or key to the same back end.
Explain the mechanisms: cookie, IP, hash, locality.
Give when to enforce (local state, flows, cache locality, real-time rooms) and its costs; recommend stateless designs.
35
How does load balancing work in containerized environments like Docker and Kubernetes?
KnownRevise
In containerised environments, load balancing has to keep up with containers that start, stop and move constantly and get new IP addresses. Platforms solve this with service discovery plus virtual addresses. - Docker Compose: on a user-defined network, containers reach each other by service name, and Docker’s embedded DNS returns the IPs of all replicas. Spreading is basic (clients may pick the first IP), so you often add a proxy (nginx, Traefik, HAProxy). - Docker Swarm: each service gets a virtual IP (VIP), and IPVS spreads connections across tasks. The routing mesh publishes ports on every node and forwards to any task. - Kubernetes works in layers: - Service (ClusterIP): a stable virtual IP and DNS name (orders.prod.svc.cluster.local). kube-proxy (iptables, IPVS or nftables mode) or Cilium’s eBPF data plane spreads connections across ready pod endpoints, tracked by EndpointSlices. - Readiness probes decide whether a pod receives traffic. - NodePort and LoadBalancer Services expose Services outside the cluster; a cloud controller provisions an external LB. - Ingress or the Gateway API, implemented by controllers such as NGINX, Envoy Gateway, Traefik or cloud LB controllers, provide L7 routing (host, path, TLS, weighted canaries). - Service meshes (Istio, Linkerd) add per-request L7 balancing, retries, outlier detection and mTLS between services. - Headless Services (clusterIP: None) return pod IPs directly for client-side load balancing (gRPC, databases).
Key detail: kube-proxy balances per connection at L4, so long-lived HTTP/2 or gRPC connections can stick to a few pods. Use a mesh, an L7 ingress or client-side balancing for those.
Read full answerHide full answer
Applies toDocker Compose and Swarm, Kubernetes 1.30+
Kubernetes Traffic Path
internet -> cloud LB (Service type LoadBalancer or Ingress/Gateway controller)
-> ingress pods (L7: host/path rules, TLS)
-> Service ClusterIP (kube-proxy / eBPF picks a ready pod per connection)
-> pod (readinessProbe passed)
kubectl get endpointslices -l kubernetes.io/service-name=orders # pods currently receiving traffic
kubectl scale deploy/orders --replicas=6 # new pods join once ready
Example: Docker Compose With Replicas Behind nginx
In nginx.conf, proxy_pass http://api:8080; resolves the service name through Docker DNS. nginx resolves it at startup, so if replicas change you need a resolver 127.0.0.11 with a variable in proxy_pass, or a reload.
Layers Compared
Layer
Balances
Level
Docker DNS
Name to IPs
DNS (client picks)
Swarm VIP / IPVS
Connections to tasks
L4
Kubernetes Service (kube-proxy, eBPF)
Connections to ready pods
L4
Ingress / Gateway controller
HTTP requests by host, path, weight
L7
Service mesh sidecar or ambient proxy
Each request, with retries and outlier detection
L7
Client-side (gRPC with headless Service)
Each request or stream
L7 in the client
Pitfalls
gRPC through ClusterIP: one connection per client pins traffic to one pod.
No readiness probes: pods receive traffic before they are ready.
Deployments dropping requests without preStop delays and graceful shutdown.
externalTrafficPolicy: Cluster losing client source IPs, or Local causing uneven spread.
Walk through Service → kube-proxy → ready pods, plus Ingress/Gateway for L7.
Mention gRPC long-lived connection imbalance, headless Services and meshes.
36
What is a reverse proxy, and how does it relate to load balancing?
KnownRevise
A reverse proxy is a server that sits in front of one or more back-end servers and receives client requests on their behalf. Clients connect to the proxy, which forwards each request to a back end and returns the response, so clients never talk to the origin directly. (A forward proxy is the opposite: it sits in front of clients and makes requests to the internet for them.) A reverse proxy commonly provides: - TLS termination and HTTP/2 or HTTP/3 to clients; - routing by host or path to different applications; - caching and compression; - security: hiding origin servers, WAF integration, rate limiting, request size limits, header sanitising; - buffering slow clients, so application workers aren’t tied up; - authentication offload, and logging and metrics.
Relationship: a load balancer distributes traffic across multiple servers. Most L7 load balancers are reverse proxies with a pool of back ends plus balancing algorithms and health checks: nginx, HAProxy, Envoy, Traefik, AWS ALB. A reverse proxy in front of one server is still useful (TLS, caching, security) but does no balancing. Conversely, some load balancers are not reverse proxies: L4 balancers (IPVS, AWS NLB in pass-through mode, DSR setups) forward packets or connections without terminating HTTP, and DNS-based balancing just returns addresses.
CDN (a globally distributed reverse proxy with caching)
-> cloud L4/L7 load balancer
-> ingress reverse proxy (nginx/Envoy) with routing + load balancing to pods
Pitfalls
Losing client information: set and trust X-Forwarded-* or the PROXY protocol only from your proxies.
Single reverse proxy in front of many servers: a single point of failure unless made redundant.
Buffering or timeouts that break streaming, WebSockets or large uploads if not tuned.
Caching personalised responses at the proxy.
Interview Tips
Define a reverse proxy as an intermediary in front of servers, and contrast it with a forward proxy.
Explain that L7 load balancers are reverse proxies with pools and algorithms.
Note that L4 and DNS load balancing are not reverse proxies, and list reverse proxy features (TLS, caching, security).
37
Explain how load balancing works with WebSockets and other persistent connection protocols.
KnownRevise
WebSockets, Server-Sent Events, gRPC streams, MQTT and long polling keep a connection open for minutes to hours. Load balancing them differs from short HTTP requests: - The balancing decision happens once per connection: the WebSocket starts as an HTTP request with Upgrade: websocket. The load balancer picks a back end, the handshake completes (101 Switching Protocols), and all later messages stay on that server for the connection’s lifetime. Balance improves only as connections churn. - Use connection-aware algorithms:least connections suits long-lived connections far better than round robin, which can pile connections onto some servers over time. - Configure timeouts and upgrades: the proxy must forward the Upgrade and Connection headers and use long idle timeouts, plus ping/pong heartbeats (application or protocol level), so idle connections aren’t cut. Examples: nginx proxy_read_timeout, HAProxy timeout tunnel, ALB idle timeout. - Plan capacity per connection: each connection holds memory and file descriptors. Monitor concurrent connections per server, and tune OS limits and ephemeral ports. - Share state across servers: users in the same chat room may be on different servers, so broadcasts go through pub/sub (Redis, NATS, Kafka), or rooms are routed by consistent hashing to one server. - Use stickiness for fallbacks: libraries like Socket.IO with HTTP long-polling fallback need sticky sessions, because a session’s polling requests must hit the same server. - Make deployments and scaling graceful: draining can take hours, so servers send close or “reconnect” messages, and clients reconnect with jittered exponential backoff (avoiding reconnect storms) and land on new servers. New servers start empty, so rebalancing happens as clients reconnect.
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream ws_backend {
least_conn;
server 10.0.1.11:9000;
server 10.0.1.12:9000;
}
server {
listen 443 ssl;
location /ws/ {
proxy_pass http://ws_backend;
proxy_http_version 1.1; # required for Upgrade
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_read_timeout 1h; # default 60 s would cut idle sockets
proxy_send_timeout 1h;
}
}
HAProxy
backend websockets
balance leastconn
timeout tunnel 1h # applies after the upgrade
timeout server 30s
server ws1 10.0.1.11:9000 check
server ws2 10.0.1.12:9000 check
Round robin with long-lived connections, creating long-term imbalance.
Default 60 s proxy timeouts silently closing idle sockets.
Missing Connection: upgrade forwarding.
Reconnect storms without jitter after a deploy or outage.
Interview Tips
Explain that the decision is per connection, not per message.
Cover upgrade headers, timeouts, least connections, heartbeats, pub/sub and stickiness for fallbacks.
Discuss draining, reconnection with backoff, and rebalancing after scaling.
38
Discuss how machine learning can be applied to improve load balancing decisions.
KnownRevise
Classic algorithms react to current, simple signals (connection counts, health). Machine learning can use richer data and prediction: - Latency and load prediction: models estimate each back end’s response time for a request from request features (endpoint, payload size, user tier) and server state (CPU, queue depth, recent latencies), then route to the lowest predicted latency or the best expected SLO compliance. - Online learning and bandits:multi-armed bandit or reinforcement learning policies try back ends, observe outcomes and adapt continuously, balancing exploration (keeping estimates fresh) against exploitation (sending traffic to the best server). EWMA-based “peak EWMA” balancing, as in Finagle and Linkerd, is a simple form. - Traffic forecasting for capacity: time-series models predict demand (daily cycles, events) to pre-scale instances before spikes, as predictive autoscaling does in AWS and Google Cloud. - Anomaly detection: detecting degraded or “grey failure” servers (slow but passing health checks) and abnormal traffic (DDoS, bots) faster than fixed thresholds. - Smarter global steering: choosing regions or CDN paths from measured network performance (real user monitoring). - Cost and energy-aware routing: balancing latency against cost, carbon intensity or spot-instance availability.
Challenges: - ML decisions must be fast (microseconds on the data path), so models are often trained offline with only lightweight scoring inline, or they adjust weights periodically rather than per request. - Feedback loops and herding: everyone chasing the “best” server overloads it, so add randomisation (power of two choices) and load terms. - Explainability, safety and fallback to simple algorithms are needed when models fail. - It needs good telemetry.
Read full answerHide full answer
In practice, ML works best alongside robust baselines (least requests, P2C, outlier detection), not replacing them.
Applies to: adaptive load balancing, predictive autoscaling (Python 3.12 simulation)
Code Example: Online-Learned Latency Routing
# Learned latency model: route to the server with the lowest *predicted* latency (an EWMA
# updated online from every response), versus round robin. Server C silently becomes
# 4x slower halfway through, and B is always a little slower.
import random, statistics
random.seed(7)
BASE = {"A": 20.0, "B": 25.0, "C": 20.0} # true mean latency (ms)
def true_latency(s, t):
mean = BASE[s] * (4 if s == "C" and t > 2000 else 1)
return random.expovariate(1 / mean)
def run(policy, n=4000, alpha=0.1, explore=0.05):
est = {s: 20.0 for s in BASE} # learned estimate per server
lat = []
for t in range(n):
if policy == "round_robin":
s = "ABC"[t % 3]
elif random.random() < explore: # small exploration keeps estimates fresh
s = random.choice("ABC")
else:
s = min(est, key=est.get)
x = true_latency(s, t)
est[s] = (1 - alpha) * est[s] + alpha * x # online learning from each response
lat.append(x)
lat.sort()
return round(statistics.mean(lat), 1), round(lat[int(0.99 * n)], 1)
print("round robin (avg, p99 ms):", run("round_robin")) # (31.3, 203.3)
print("learned latency (avg, p99 ms):", run("learned")) # (21.8, 105.0)
The learned policy notices that C degraded and that B is slower, and shifts traffic away, cutting average and tail latency. Note that a real system must also model load: routing everything to the currently fastest server would overload it. That’s why production designs combine estimates with P2C and concurrency terms.
Keep the data path simple: score cheaply, or push weights from an offline or nearline controller.
Guardrails: cap weight changes, keep minimum traffic to every healthy server, and add automatic fallback to static algorithms.
Avoid herding: randomise between top candidates, and include queue length or in-flight requests.
Evaluate safely: shadow mode, canaries and A/B comparison against baseline p99 and error rates.
Monitor model drift as traffic patterns change.
Pitfalls
Oscillation: all traffic moves to one server, it slows, and traffic moves back.
Stale or noisy features causing bad decisions.
Complexity without measurable gains over P2C or least requests.
Opaque decisions that make incidents harder to debug.
Interview Tips
Mention latency prediction, bandits, forecasting for predictive scaling and anomaly detection.
Discuss real-time constraints, herding, guardrails and fallback.
Position ML as augmenting proven algorithms, validated by experiments.
39
What strategies might you employ to handle sudden spikes in traffic using load balancing?
KnownRevise
Handling sudden spikes (“flash crowds”, launches, viral posts, retries after an outage) needs several strategies together: - Absorb at the edge:CDN caching of static and cacheable content, and micro-caching hot API responses for a few seconds at the load balancer, so most spike traffic never reaches the application. - Spread efficiently: use least outstanding requests or power of two choices, not round robin, so slow servers don’t pile up. Enable cross-zone balancing. Use anycast or GSLB to spread across regions. - Scale fast:autoscaling on request rate or latency with pre-warmed capacity or warm pools. Schedule scaling for known events, and use predictive scaling for recurring patterns. Managed LBs may need pre-warming for extreme, instant spikes. - Control admission: apply rate limiting (token buckets per client, plus a global limit), connection limits and queues with bounded size. Return 429 or 503 with Retry-After instead of letting everything time out. Use a virtual waiting room for ticket sales and launches. - Shed and degrade gracefully:prioritise critical traffic (checkout over recommendations), turn off expensive features with feature flags, and serve stale cache or static fallbacks. - Protect dependencies:circuit breakers, timeouts, retry budgets and bulkheads prevent the spike from cascading into databases and third parties. - Prepare:load-test to known limits, keep capacity headroom, and practise runbooks.
# Token bucket at the load balancer: absorb short bursts, shed sustained overload with 429s
class TokenBucket:
def __init__(self, rate, burst):
self.rate, self.capacity, self.tokens = rate, burst, burst
def tick(self): # called once per second
self.tokens = min(self.capacity, self.tokens + self.rate)
def take(self, n): # returns (accepted, rejected)
ok = min(n, int(self.tokens))
self.tokens -= ok
return ok, n - ok
bucket = TokenBucket(rate=100, burst=300) # 100 req/s sustained, bursts up to 300
traffic = [80, 90, 350, 120, 100, 400, 400, 400, 90, 80]
for sec, n in enumerate(traffic):
ok, shed = bucket.take(n)
print(f"t={sec}s offered={n:3d} served={ok:3d} rejected_429={shed:3d}")
bucket.tick()
# t=2s offered=350 served=300 rejected_429= 50 (burst capacity absorbs most of the spike)
# t=5s offered=400 served=100 rejected_429=300 (sustained overload: only the steady rate is served)
Short bursts are absorbed by the bucket’s burst capacity. During a sustained overload, only the sustainable rate is admitted and the rest get fast 429 responses, instead of every request slowing down and timing out.
nginx: Rate Limits With Burst and Queueing Limits
limit_req_zone $binary_remote_addr zone=perclient:20m rate=10r/s;
limit_req_zone $server_name zone=global:1m rate=2000r/s;
upstream app {
least_conn;
server 10.0.1.11:8080 max_conns=200; # cap concurrency per server
server 10.0.1.12:8080 max_conns=200;
}
server {
location /api/ {
limit_req zone=perclient burst=20 nodelay;
limit_req zone=global burst=500;
limit_req_status 429;
proxy_pass http://app;
}
location /api/catalog/ {
proxy_cache api; proxy_cache_valid 200 5s; # micro-cache hot reads
proxy_cache_lock on;
proxy_cache_use_stale updating error timeout;
proxy_pass http://app;
}
}
max_conns in open-source nginx applies per worker process unless the upstream has a zone. Add zone app 64k; so the limit is shared across workers.
Relying only on autoscaling, which is too slow for second-scale spikes.
Unbounded queues: requests wait until they time out anyway.
Retries at every layer, amplifying the spike.
Scaling app servers into a database connection limit.
Interview Tips
Layer the answer: cache, spread, scale, admit, shed, protect, prepare.
Show rate limiting with bursts and 429/Retry-After.
Mention waiting rooms, predictive and scheduled scaling and load testing.
40
How do load balancers contribute to a system’s security posture?
KnownRevise
Load balancers sit at the entry point of a system, which makes them a natural security control layer: - Reduced attack surface: back-end servers live in private subnets with no public IPs. Only the load balancer is exposed, on specific ports (usually 443), and security groups allow back-end traffic only from the LB. - TLS enforcement: centralised TLS termination with modern protocols and ciphers, HTTP-to-HTTPS redirects, HSTS, automated certificate management, and optional re-encryption or mTLS to back ends. - DDoS resilience: absorbing SYN floods and connection exhaustion, connection limits and timeouts against slow attacks (Slowloris), integration with AWS Shield, Azure DDoS Protection, Cloud Armor and Cloudflare, and scaling capacity. - Application-layer filtering:WAF integration (AWS WAF, ModSecurity / Coraza, Cloud Armor rules) against injection, XSS and known exploits; bot management; geo and IP allow/deny lists. - Rate limiting against brute-force login attempts, scraping and API abuse. - Authentication offload: OIDC or JWT validation at the edge (ALB OIDC authentication, Envoy JWT filter), and mTLS client certificates for partner APIs. - Request hygiene: limits on header and body size, rejection of malformed HTTP (preventing some request smuggling), and removal of internal headers such as Server, X-Powered-By and debug headers. - Visibility:access logs and metrics feeding SIEM systems for detection and forensics. - Availability as security: health checks and failover keep services available, which is the “A” in the CIA triad.
The load balancer is itself a critical asset: it holds private keys, so harden it, patch it and restrict admin access. It is one layer in defence in depth, not a replacement for secure application code.
# Back ends accept traffic only from the ALB's security group, not from the internet
aws ec2 authorize-security-group-ingress --group-id "$APP_SG" \
--protocol tcp --port 8080 --source-group "$ALB_SG"
Security Functions
Function
Threat addressed
Private back ends, single exposed port
Direct server attacks, scanning
TLS termination and policies
Eavesdropping, downgrade attacks
WAF rules
SQL injection, XSS, known CVE exploits
Rate limits, bot management
Credential stuffing, scraping, abuse
Connection limits and timeouts
Slowloris, connection exhaustion
mTLS / OIDC at the edge
Unauthenticated access
Header and size limits, strict HTTP parsing
Smuggling, buffer abuse
Logging
Detection, incident response
Hardening the Load Balancer Itself
Patch promptly (proxy CVEs such as HTTP/2 Rapid Reset, CVE-2023-44487).
Restrict management interfaces and APIs; use IAM least privilege.
Store keys in KMS or HSMs, or use managed certificates.
Monitor configuration changes, and keep config in version control.
Pitfalls
Origins still publicly reachable, bypassing the LB and WAF.
Trusting client-supplied X-Forwarded-For for IP-based controls.
Stress defence in depth and hardening the LB itself.
Mention security groups that allow traffic only from the LB.
41
Can load balancers protect against SQL injection or XSS attacks? If so, how?
KnownRevise
Partly, and only with a WAF. A plain load balancer distributes traffic and does not understand application attacks. An L7 load balancer integrated with a Web Application Firewall (WAF) can inspect HTTP requests (URL, query string, headers, cookies, body) and block or flag those that match attack patterns. This works through: - managed rule sets such as the OWASP Core Rule Set (CRS), AWS Managed Rules (AWSManagedRulesSQLiRuleSet, AWSManagedRulesCommonRuleSet), Cloud Armor preconfigured rules, and Azure’s default rule set; - normalisation before matching: URL decoding, HTML entity decoding, case folding and whitespace compression, to defeat encoding tricks; - anomaly scoring, where several weak signals add up to a block; - rate limiting and bot management, which slow automated injection scanners; - virtual patching to block a known exploit quickly while the code fix ships.
The limits are important. WAFs work on patterns, not application semantics: - they can be bypassed with novel encodings, JSON or GraphQL payloads, or stored XSS delivered through channels they don’t see; - they produce false positives that need tuning; - they cannot see DOM-based XSS in browser JavaScript, or second-order injection.
Read full answerHide full answer
The real defences are in the application: - parameterised queries or ORMs for SQL; - context-aware output encoding and safe templating for XSS; - input validation and Content Security Policy.
A WAF at the load balancer is a valuable extra layer (defence in depth), not a substitute.
Applies to: AWS WAF on ALB, Azure Application Gateway WAF, Google Cloud Armor, ModSecurity/Coraza with nginx, HAProxy or Envoy (Python 3.12 illustration)
Code Example: Why Normalisation Matters
# Toy WAF rule check: why WAFs normalise (decode) input before matching signatures
import re
from urllib.parse import unquote_plus
RULES = {
"sqli": re.compile(r"('|%27)\s*(or|and)\s+\d+\s*=\s*\d+|union\s+select|;\s*drop\s+table", re.I),
"xss": re.compile(r"<\s*script|on\w+\s*=|javascript:", re.I),
}
def inspect(raw, normalise=True):
value = raw
if normalise:
for _ in range(3): # repeated decoding catches double-encoding
decoded = unquote_plus(value)
if decoded == value:
break
value = decoded
return [name for name, rx in RULES.items() if rx.search(value)] or ["allow"]
requests = [
"q=laptops",
"id=1' OR 1=1--",
"id=1%2527%2520OR%25201%253D1--", # double URL-encoded SQL injection
"name=<script>alert(1)</script>",
"name=%3Cimg%20src%3Dx%20onerror%3Dalert(1)%3E",
"comment=Use the select tool, then union them", # benign text (no false positive here)
]
for r in requests:
print(f"{r[:40]:42} raw={inspect(r, normalise=False)} normalised={inspect(r)}")
# id=1%2527%2520OR%25201%253D1-- raw=['allow'] normalised=['sqli'] (encoding bypasses naive rules)
# name=%3Cimg%20src%3Dx%20onerror%3D raw=['allow'] normalised=['xss']
A rule matching only raw text misses double-encoded payloads. Real WAFs apply transformation chains before matching, and are still bypassable with techniques the rules don’t model.
Example: AWS WAF Managed SQL Injection Rules on an ALB
# Parameterised SQL and escaped HTML output: the payloads become harmless data
import html, sqlite3
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE users (email TEXT, name TEXT)")
db.execute("INSERT INTO users VALUES ('a@example.com', 'Asha')")
email = "x' OR 1=1--" # injection attempt
rows = db.execute("SELECT name FROM users WHERE email = ?", (email,)).fetchall()
print("rows returned:", rows) # [] (treated as a literal string)
name = "<script>alert(1)</script>"
print("rendered:", f"<p>Hello, {html.escape(name)}</p>") # <p>Hello, <script>...</p>
A Content-Security-Policy header such as default-src 'self'; script-src 'self' limits what injected scripts can do. The load balancer can add it to every response.
What a WAF at the Load Balancer Can and Cannot Do
Can
Cannot
Block common SQLi/XSS payloads in requests
Understand your queries or templates
Normalise encodings before matching
Catch every novel bypass
Virtually patch known CVEs fast
Stop DOM-based XSS in the browser
Rate-limit scanners and bots
Fix stored or second-order injection reliably
Add security headers (CSP, HSTS)
Replace secure coding and testing
Deployment Tips
Start in count or detection mode, review logs, tune exclusions, then switch to blocking.
Keep rules updated, and alert on blocked-request spikes.
Make sure origins can’t be reached directly, bypassing the WAF.
Pitfalls
WAF as the only defence against injection.
Leaving it in detection mode forever.
Body inspection limits silently skipping large payloads.
False positives blocking legitimate users, such as code snippets in forums.
Interview Tips
Answer “yes, with a WAF, partially”: signatures, normalisation, managed rules.
Explain limits (bypasses, false positives, DOM XSS).
Stress parameterised queries, output encoding and CSP as the primary defences.
42
How does a Web Application Firewall (WAF) integrate with load balancing solutions?
KnownRevise
A WAF inspects HTTP(S) traffic for attacks (injection, XSS, known exploits, bots, abuse), so it must sit where traffic is decrypted and parsed, which is usually at or next to the L7 load balancer. Integration models: - Built into the load balancer: the WAF is a feature attached to the LB. - AWS WAF web ACLs on ALB, CloudFront and API Gateway. - Azure Application Gateway WAF and Front Door WAF. - Google Cloud Armor security policies on the external Application Load Balancer. - F5 Advanced WAF on BIG-IP. - NGINX App Protect. - ModSecurity or Coraza modules in nginx, HAProxy (SPOE) and Envoy (proxy-wasm or ext_authz).
The LB terminates TLS, the WAF evaluates the request, then it is blocked (403 or custom page), challenged (CAPTCHA) or forwarded to the pool. - Edge or CDN WAF in front of the load balancer: Cloudflare, Akamai, Fastly and CloudFront with AWS WAF filter at the edge. The origin LB then accepts traffic only from the CDN, via IP allowlists, authenticated origin pulls (mTLS) or secret headers. - Inline appliance or sandwich: a separate WAF tier between two load balancers, or behind a Gateway Load Balancer (AWS GWLB with GENEVE), so WAF instances scale and fail over independently. - Sidecar or in-mesh WAF for east-west traffic in Kubernetes.
Read full answerHide full answer
Operational integration includes: - shared TLS certificates; - health and failover of the WAF tier (fail-open versus fail-closed policy); - passing the real client IP (X-Forwarded-For, PROXY protocol) so IP rules work; - logging to SIEM; - rule tuning (count mode, then block).
Example: ModSecurity (OWASP CRS) in nginx as a Load Balancer
load_module modules/ngx_http_modsecurity_module.so;
upstream app { least_conn; server 10.0.1.11:8080; server 10.0.1.12:8080; }
server {
listen 443 ssl;
modsecurity on;
modsecurity_rules_file /etc/nginx/modsec/main.conf; # includes CRS + local exclusions
location / {
proxy_pass http://app;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
# /etc/nginx/modsec/main.conf (excerpt)
Include /etc/nginx/modsec/modsecurity.conf # SecRuleEngine DetectionOnly -> On after tuning
Include /usr/share/modsecurity-crs/crs-setup.conf
Include /usr/share/modsecurity-crs/rules/*.conf
SecRuleRemoveById 942100 # example tuned exclusion for a false positive
ModSecurity is now an OWASP project. OWASP Coraza is a Go-based, CRS-compatible alternative that is often used with Envoy and Caddy.
Example: Cloud Armor Rule on a Google Cloud Load Balancer
WAF needs decrypted traffic: integrated LBs or re-encryption to a WAF tier
Client IP
Preserve it for geo and IP rules (XFF, PROXY protocol)
Latency
Rule evaluation adds latency; managed WAFs typically add a few ms or less
Availability
Scale WAF tiers horizontally; decide fail-open or fail-closed
Bypass
Lock origins to the LB or CDN only
Tuning
Count mode, exclusions, then block; review regularly
Logging
Ship WAF logs to SIEM; alert on spikes
Pitfalls
Origins reachable directly, bypassing the WAF.
Wrong client IP, so rate and IP rules hit the CDN or LB address.
Body inspection limits that let large payloads through uninspected.
Untuned rules that block real users, or that never move to blocking mode.
Interview Tips
Explain that a WAF needs decrypted HTTP, so it sits at the L7 load balancer or edge.
Describe integrated, edge, WAF-tier and GWLB models.
Cover client IP preservation, origin lockdown, tuning and availability.
43
Describe the importance of properly configuring HTTPS certificates on load balancers.
KnownRevise
When a load balancer terminates TLS, its certificate configuration decides whether every user can connect securely. Getting it right matters for several reasons: - Trust and availability: browsers and API clients reject connections with an expired, untrusted, mismatched or incomplete-chain certificate. A single mistake therefore causes a site-wide outage and frightening warnings. Expired certificates are one of the most common causes of preventable outages. - Correct names: the certificate must cover every hostname served (SANs, wildcards). With many domains, SNI lets one listener select the right certificate per hostname. - Complete chain: serve the leaf plus intermediate certificates. Some browsers can fetch missing intermediates, but many clients cannot (mobile apps, IoT, Java and curl), so an incomplete chain causes failures for only some users. - Strong security:key size or type (RSA 2048+, or ECDSA P-256), a modern TLS policy (TLS 1.2 minimum, TLS 1.3 preferred, strong ciphers, forward secrecy), HSTS, and OCSP stapling for faster, more private revocation checks. - Key protection: the private key at the LB can decrypt traffic for that domain. Store it in managed certificate services, KMS or HSMs, limit access, and rotate on compromise. - Automation:automatic renewal (ACM, Let’s Encrypt via ACME, cert-manager, Azure Key Vault integration) plus expiry monitoring, since certificate lifetimes are getting shorter: the CA/Browser Forum has approved a phased reduction to 47 days by 2029. - Compliance: standards like PCI DSS require strong TLS and secure key handling. - Consistency across instances: every LB node, region and listener must serve the same valid certificate.
Read full answerHide full answer
Applies toAWS ALB/NLB with ACM, Azure Application Gateway, Google Cloud LB, nginx, HAProxy, Kubernetes Ingress with cert-manager
Wildcard certificates shared widely, increasing the blast radius.
Not testing non-browser clients.
Interview Tips
Explain trust, availability, security and compliance impacts.
Cover SANs and SNI, full chains, TLS policy, HSTS, OCSP stapling and key protection.
Emphasise automation and expiry monitoring.
44
Discuss the security implications of SSL termination at the load balancer.
KnownRevise
SSL/TLS termination at the load balancer means the LB decrypts client traffic, so traffic between the LB and back ends may travel unencrypted unless re-encrypted. The security implications cut both ways.
Benefits: - central, consistent TLS policy (TLS 1.2/1.3, strong ciphers, HSTS); - one place to manage certificates and renewals; - visibility for security controls, since a WAF, bot detection, rate limiting and request validation all need decrypted traffic; - reduced attack surface on back ends, which hold no public certificates.
Read full answerHide full answer
Risks and considerations: - Plaintext in the internal network: anyone with access to that network segment (a compromised host, a misconfigured mirror port, a malicious insider) could read or alter traffic, including credentials, tokens and personal data. Zero-trust and many compliance regimes (PCI DSS, HIPAA guidance) expect encryption in transit everywhere, so re-encrypt (TLS bridging) or use mTLS / a service mesh to back ends. - The LB holds private keys: it becomes a high-value target. Protect keys with managed certificate services, KMS or HSMs, harden and patch the LB, and restrict admin access. - Trust in forwarded headers: back ends rely on X-Forwarded-For and X-Forwarded-Proto. If clients can reach back ends directly, or if headers aren’t overwritten, attackers can spoof IPs or trick apps into thinking a connection was HTTPS. - Back ends lose TLS context: client certificate details (for mTLS) must be passed securely in headers, and apps must build correct https:// URLs and secure cookies. - Logging exposure: decrypted requests may land in LB or WAF logs with sensitive data. Scrub or limit what is logged. - Concentration: a vulnerability in the TLS stack of the LB affects every service behind it, so patch quickly (Heartbleed in the past; HTTP/2 Rapid Reset).
The usual resolution: terminate at the LB for inspection, then re-encrypt to back ends (end-to-end encryption with an inspection point), or use TLS passthrough where no intermediary may decrypt.
Applies to: nginx, HAProxy, Envoy, AWS ALB, Azure Application Gateway, Google Cloud LB
Explain re-encryption and passthrough as alternatives.
Mention compliance and zero-trust expectations.
45
Explain how rate limiting works in the context of load balancing.
KnownRevise
Rate limiting caps how many requests (or connections) a client, key or route may make in a given time. Load balancers and gateways are a natural place to enforce it, because they see all traffic before it reaches back ends. How it works: 1. Identify the client or key: IP address (the real client IP, using X-Forwarded-For or the PROXY protocol from trusted proxies), API key, user ID from a JWT, tenant, path or a combination. 2. Count with an algorithm: - Fixed window: simple, but allows bursts at window edges. - Sliding window log or counter: smoother. - Token bucket: a steady refill rate plus burst capacity. - Leaky bucket: smooths the output rate by queueing. 3. Decide: allow, delay or queue (shaping), or reject with HTTP 429 Too Many Requests, ideally with Retry-After and rate-limit headers. 4. Store counters: - Locally per LB instance: fast, but the effective limit is the limit times the number of instances, and it is uneven. - Centrally: Redis, or a rate-limit service such as Envoy’s global rate limit service. Accurate, but adds latency and a dependency. - Hybrid: local plus periodic sync.
Purposes: protecting back ends from overload, fairness between tenants, abuse prevention (brute force, scraping, credential stuffing), cost control, and API product tiers. Rate limiting complements connection limits, concurrency limits, WAF and DDoS protection. Applications should still enforce business limits, such as login attempts per account.
# Sliding-window counter rate limiter (as used by many gateways): smooths the edges of
# fixed windows by weighting the previous window's count
import math
class SlidingWindow:
def __init__(self, limit, window=60):
self.limit, self.window = limit, window
self.counts = {} # (client, window index) -> count
def allow(self, client, now):
idx = math.floor(now / self.window)
elapsed = (now % self.window) / self.window
prev = self.counts.get((client, idx - 1), 0)
cur = self.counts.get((client, idx), 0)
estimate = prev * (1 - elapsed) + cur # requests in the last `window` seconds
if estimate >= self.limit:
return False
self.counts[(client, idx)] = cur + 1
return True
rl = SlidingWindow(limit=100, window=60)
# 100 requests at t=59 s, then 100 more at t=61 s: a fixed window would allow all 200
first = sum(rl.allow("ip-1", 59.0) for _ in range(100))
second = sum(rl.allow("ip-1", 61.0) for _ in range(100))
later = sum(rl.allow("ip-1", 110.0) for _ in range(100))
other = sum(rl.allow("ip-2", 61.0) for _ in range(100))
print("t=59s allowed:", first) # 100
print("t=61s allowed:", second) # 2 (previous window still weighs 100 x 59/60 = 98.3)
print("t=110s allowed:", later) # 82 (previous window now weighs 100 x 10/60 = 16.7; 2 already counted)
print("other client allowed:", other) # 100 (limits are per client)
A fixed 60-second window would allow 200 requests in 2 seconds across the boundary. The sliding estimate prevents that, and each client has its own budget.
nginx
limit_req_zone $binary_remote_addr zone=ip:10m rate=10r/s; # per client IP
limit_req_zone $http_x_api_key zone=key:10m rate=50r/s; # per API key
server {
location /api/ {
limit_req zone=ip burst=20 nodelay; # token-bucket-like burst allowance
limit_req zone=key burst=100;
limit_req_status 429;
proxy_pass http://api;
}
location = /login {
limit_req zone=ip burst=3; # stricter on authentication
proxy_pass http://api;
}
}
Each nginx instance keeps its own counters (shared across workers through the zone), so with three instances a client may get up to 3× the limit.
HAProxy (Stick Tables)
frontend fe
bind :443 ssl crt /etc/haproxy/certs/site.pem
stick-table type ip size 1m expire 10m store http_req_rate(10s)
http-request track-sc0 src
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }
default_backend app
Stick tables can be synchronised between HAProxy peers for a shared view.
Algorithms Compared
Algorithm
Behaviour
Notes
Fixed window
Count per minute bucket
Simple; 2× bursts at boundaries
Sliding log
Exact timestamps
Accurate; memory-heavy
Sliding window counter
Weighted previous + current window
Good accuracy, cheap
Token bucket
Refill rate + burst size
Allows bursts, common in proxies
Leaky bucket
Queue drained at a fixed rate
Smooths output; adds latency
Concurrency limit
Max in-flight requests
Protects against slow requests better than rates
Design Choices
Choice
Options
Key
IP, API key, user, tenant, route, or combined
Scope
Per instance, global (Redis or a rate-limit service), or edge (CDN, WAF)
Response
429 + Retry-After, delay, CAPTCHA challenge
Tiers
Different limits per plan or endpoint
Pitfalls
Limiting by the proxy’s IP instead of the real client IP.
NAT and corporate users sharing one IP and hitting limits together.
No Retry-After, so clients retry immediately and make things worse.
Central store outages: decide whether to fail open or closed.
Interview Tips
Explain identify, count, decide, store.
Compare fixed window, sliding window, token and leaky bucket.
Discuss local versus global counters, real client IPs and 429 with Retry-After.
46
How would you troubleshoot a situation where a load balancer is incorrectly routing traffic?
KnownRevise
“Incorrect routing” can mean requests reaching the wrong back end or pool, the wrong version or region, uneven distribution, broken stickiness, or traffic still going to removed servers. Troubleshoot systematically from the client inward: 1. Define the symptom precisely: which hosts, paths, headers or clients are affected, what was expected and what actually happened, and since when. Correlate with recent changes (deployments, config, DNS, certificates, autoscaling events). Most routing bugs follow a change. 2. Check that traffic reaches the intended load balancer:DNS (dig; stale records, TTL caching, split-horizon DNS), CDN configuration, and whether clients hit the correct listener, port and IP. 3. Inspect the routing rules: - rule order and priority (a broad rule matching first); - host or path match types (prefix versus exact, trailing slashes, case); - header conditions, regexes, default actions; - SNI and certificate selection for HTTPS. 4. Check the target pools: registered targets, health status, ports and weights (a canary left at 50%), and stale endpoints (Kubernetes EndpointSlices, service discovery lag). 5. Check stickiness and hashing: affinity cookies pinning users to old servers, hash keys that changed, consistent-hash rings after scaling. 6. Reproduce with controlled requests:curl with explicit Host, headers and --resolve. Have back ends return an identifying header (server ID, version). 7. Read the LB logs: which rule matched, which target was chosen, the response code. Use tracing across hops. 8. Fix carefully: correct the rule or pool, roll back recent changes if needed, then verify with tests and metrics. Add config tests and change reviews to prevent recurrence.
Read full answerHide full answer
Applies tonginx, HAProxy, Envoy/Istio, AWS ALB, Kubernetes Ingress and Gateway API
Useful Commands
# Which IPs do clients resolve? Compare resolvers.
dig +short shop.example.com @1.1.1.1 ; dig +short shop.example.com @8.8.8.8
# Bypass DNS and send a request to a specific LB IP with the right Host/SNI
curl -sv --resolve shop.example.com:443:203.0.113.10 https://shop.example.com/api/orders \
-H "X-Debug: 1" -o /dev/null 2>&1 | grep -iE "< HTTP|< x-served-by|< x-version"
# Repeat to see the distribution across back ends
for i in $(seq 1 20); do curl -s -o /dev/null -w "%header{x-served-by}\n" https://shop.example.com/; done | sort | uniq -c
%header{...} in -w needs curl 7.84+.
Platform-Specific Checks
Platform
What to inspect
nginx
nginx -T (full effective config), location match order (exact =, ^~, regex, prefix), $upstream_addr in logs
HAProxy
use_backend order and ACLs, show servers state, logs with %b/%s (backend/server)
Listener rule priorities and conditions, target group health, access logs (target:port, matched_rule_priority)
Kubernetes
kubectl describe ingress/httproute, kubectl get endpointslices, Service selectors and labels
Common Causes
Symptom
Likely cause
Requests hit the default backend
Host header mismatch, rule order, path type (Exact vs Prefix)
Traffic to old version
Canary weight not reset, stale endpoints, sticky cookies
One server gets most traffic
Hash on a low-cardinality key, IP affinity behind NAT, HTTP/2 long-lived connections at L4
Some users routed wrongly
Split DNS, CDN cache keyed incorrectly, geo rules
Traffic to removed servers
Long TTLs, deregistration not done, service discovery lag
Wrong certificate or 421 errors
SNI and Host mismatch, connection coalescing in HTTP/2
A Kubernetes Service selector that matches pods from two Deployments (for example app: web on both v1 and v2) sends traffic to both; check labels carefully.
Pitfalls
Testing from a machine with cached DNS or hosts-file overrides.
Changing several things at once, losing track of the cause.
Ignoring rule precedence semantics, which differ by product.
No back-end identifying headers, making routing invisible.
Interview Tips
Use a systematic path: symptom → recent changes → DNS → listener → rules → pools/health → stickiness → logs.
What would you do if a load balancer marks a healthy backend as down?
KnownRevise
If the load balancer marks a back end down while the server seems to be running, the health check is failing from the load balancer’s point of view. Reproduce the check exactly as the LB performs it, and work through the possible causes: 1. Read the LB’s reason: health-check status and reason codes (ALB Target.ResponseCodeMismatch, Target.Timeout, Target.FailedHealthChecks), HAProxy logs (L4TOUT, L7STS, L4CON), Envoy /clusters health flags. 2. Run the same check from the LB’s network: same IP, port, path, Host header, protocol and timeout: curl -v http://10.0.1.12:8080/ready from an LB host or a debug pod in the same subnet. “Running” on localhost proves little. 3. Check network reachability:security groups, NACLs and firewalls (allow health checks from the LB’s subnets or security group), NetworkPolicies in Kubernetes, routes, and whether the app listens on 0.0.0.0 rather than 127.0.0.1. 4. Check the health check configuration: - wrong port or path (/health versus /healthz); - expected status (the app returns 301 to HTTPS or 401, but the check expects 200); - a Host header the app rejects; - HTTP versus HTTPS mismatch; - timeouts shorter than the endpoint’s response time; - thresholds too strict. 5. Check the health endpoint’s logic: it may fail because of a dependency (database, cache) that is slow or down, or because readiness is deliberately failing (warm-up, draining, maintenance flag). 6. Check resources and timing: CPU throttling, GC pauses, thread or connection pool exhaustion making the endpoint slow under load, and new instances still in grace or start-up periods. 7. Fix the root cause (config, network rules, endpoint logic or capacity), watch the server pass the healthy threshold, and add monitoring and alerts on health-check failures.
# AWS: why is the target unhealthy?
aws elbv2 describe-target-health --target-group-arn "$TG" \
--query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State,TargetHealth.Reason,TargetHealth.Description]'
# From a host in the LB's subnet: replicate the probe exactly
curl -sv -m 5 -H "Host: app.internal" http://10.0.1.12:8080/ready
# On the server: is it listening on the right interface and port?
ss -ltnp | grep 8080 # 127.0.0.1:8080 means unreachable from the LB
# Kubernetes: readiness probe failures
kubectl describe pod api-7d9f -n prod | sed -n '/Readiness/,+3p;/Events/,$p'
Common Causes
Cause
Clue
Fix
Firewall or security group blocks LB
Target.Timeout, L4TOUT
Allow the LB’s SG or subnet on the health port
App bound to 127.0.0.1
Works locally only
Bind to 0.0.0.0
Wrong path or port
404 or connection refused
Fix health check config
HTTP→HTTPS redirect on the health path
301/302 mismatch
Exempt health path or accept 200–399
Auth required
401/403
Make health path unauthenticated (restricted by network)
Host-based routing in the app
404 or 400 for unknown Host
Set the Host header in the check
Slow endpoint (deep dependency checks)
Timeouts under load
Shallow readiness checks, longer timeout
Grace period too short
Fails only at start-up
Increase grace period or initial delay
Prevention
Keep health endpoints cheap, unauthenticated (but network-restricted) and fast.
Document exact health check settings with the service, and test them in CI or staging.
Alert on unhealthy host count, and include reason codes in dashboards.
Pitfalls
Declaring the server “fine” because curl localhost works.
Lowering thresholds or disabling checks to make alarms go away.
Deep dependency checks taking all servers out when a shared database blips.
Fixing one instance manually when the cause is shared configuration.
Interview Tips
Emphasise reproducing the check from the LB’s perspective.
Walk through network, config, endpoint logic and resources.
Mention reason codes, tooling and prevention.
48
Describe how you might identify and resolve performance bottlenecks in a load-balanced system.
KnownRevise
Performance work in a load-balanced system means finding which hop or resource limits throughput or latency, then fixing that, re-measuring and repeating. Approach: 1. Define the problem and baseline: which SLO is violated (p99 latency, throughput, errors), on which endpoints and when. Compare against normal behaviour. 2. Break latency down per hop: client → CDN → LB → back end → dependencies. Use LB metrics (LB processing time versus target response time), distributed tracing (OpenTelemetry) and timing fields in logs ($request_time versus $upstream_response_time). 3. Check each layer with the USE method (utilisation, saturation, errors): - Load balancer: CPU (TLS), connections, queues, bandwidth, capacity units, connection limits, ephemeral ports. - Distribution: per-server request counts and latency. Look for imbalance caused by stickiness, hashing, long-lived connections or poor algorithms. - Back ends: CPU, memory, GC, thread and worker pools, connection pools. - Dependencies: database (slow queries, locks, connection limits), caches (hit rate), external APIs. - Network: cross-zone hops, packet loss, MTU, DNS lookup times. 4. Look for queueing: latency rising sharply as utilisation nears 100% is the classic sign of a saturated stage. 5. Form a hypothesis, test it with load tests or targeted experiments, and fix: - scale out the saturated tier; - switch algorithms (least requests or P2C); - enable keep-alive and connection reuse; - cache; - tune pools and timeouts; - optimise queries; - fix imbalance (remove IP affinity, use L7 balancing for gRPC). 6. Verify and prevent: re-measure, add dashboards and alerts, and load-test regularly.
# Split total request time into "upstream" (back end) and "proxy/client" time from nginx-style
# logs, per upstream server, to see where latency comes from
import re, statistics
from collections import defaultdict
LOG = """\
10.1.1.5 "GET /api/items" 200 rt=0.212 uct=0.001 urt=0.205 ua="10.0.1.11:8080"
10.1.1.6 "GET /api/items" 200 rt=0.198 uct=0.001 urt=0.190 ua="10.0.1.12:8080"
10.1.1.7 "GET /api/items" 200 rt=1.402 uct=0.001 urt=1.380 ua="10.0.1.13:8080"
10.1.1.8 "POST /upload" 200 rt=3.950 uct=0.002 urt=0.310 ua="10.0.1.11:8080"
10.1.1.9 "GET /api/items" 200 rt=1.250 uct=0.002 urt=1.231 ua="10.0.1.13:8080"
10.1.1.5 "GET /api/items" 502 rt=1.003 uct=1.000 urt=1.000 ua="10.0.1.14:8080"
10.1.1.6 "GET /api/items" 200 rt=0.220 uct=0.001 urt=0.211 ua="10.0.1.12:8080"
"""
pat = re.compile(r'"(\S+) (\S+)" (\d+) rt=([\d.]+) uct=([\d.]+) urt=([\d.]+) ua="([^"]+)"')
by_server = defaultdict(list)
for line in LOG.splitlines():
m = pat.search(line)
method, path, status, rt, uct, urt, ua = m.groups()
rt, uct, urt = float(rt), float(uct), float(urt)
by_server[ua].append((int(status), rt, uct, urt))
if rt - urt > 1.0:
print(f"slow outside the back end: {method} {path} total={rt}s upstream={urt}s")
if uct >= 1.0:
print(f"connect timeout to {ua}: check back-end health, security groups, listen backlog")
for ua, rows in sorted(by_server.items()):
med = statistics.median(r[3] for r in rows)
errors = sum(r[0] >= 500 for r in rows)
print(f"{ua}: requests={len(rows)} median_upstream={med:.3f}s errors={errors}")
# slow outside the back end: POST /upload total=3.95s upstream=0.31s (slow client upload)
# connect timeout to 10.0.1.14:8080: ... (dead or unreachable)
# 10.0.1.13:8080: requests=2 median_upstream=1.305s errors=0 (one slow back end)
Three different bottlenecks appear in a few lines: - a slow client upload: time outside the back end, so consider buffering, upload limits or direct-to-storage uploads; - an unreachable back end: connect timeouts; - one slow server: an outlier among identical instances, so check that host, or eject it with outlier detection.
Signals and Likely Bottlenecks
Signal
Likely bottleneck
Fix
LB time high, target time normal
LB itself (TLS CPU, limits) or slow clients
Scale or upgrade the LB, session resumption, buffering
One server slow, others fine
Bad host, noisy neighbour, imbalance
Outlier detection, replace host, fix affinity
All servers slow, CPU high
Back-end capacity
Scale out, optimise code
All servers slow, CPU low
Waiting on dependencies or pools
Tune pools, fix slow queries, add caching
Latency jumps at a traffic threshold
Queueing at a saturated stage
Add capacity at that stage
Many new connections per second
No keep-alive to back ends
Enable upstream keep-alive
502/504 under load
Timeouts, worker exhaustion
Right-size workers and timeouts, add capacity
Example: Enable Upstream Keep-Alive in nginx
upstream app {
least_conn;
server 10.0.1.11:8080;
server 10.0.1.12:8080;
keepalive 128; # pool of idle connections per worker
}
location / {
proxy_pass http://app;
proxy_http_version 1.1;
proxy_set_header Connection ""; # don't send "close"
}
Without this, nginx opens a new TCP connection per request to back ends, adding latency and risking ephemeral port exhaustion.
Load Testing
# k6: ramp to find the knee of the latency curve
k6 run --vus 50 --duration 5m script.js
# Watch p95/p99 and per-tier utilisation together during the test
Pitfalls
Averages hiding tail latency.
Scaling the wrong tier (adding app servers when the database is saturated).
Testing with unrealistic traffic (no TLS, no keep-alive, uniform requests).
Fixing symptoms (longer timeouts) instead of causes.
Interview Tips
Describe per-hop latency breakdown, the USE method, tracing and queueing signs.
Give concrete bottleneck patterns and fixes (imbalance, keep-alive, pools, caching).
Emphasise hypothesis → test → fix → verify.
49
How would you handle a situation where the load balancer becomes the performance bottleneck?
KnownRevise
First confirm that the load balancer really is the bottleneck: LB processing time rises while back-end response times stay normal; LB CPU (often TLS), connection counts, bandwidth or queues are saturated; you see rejected connections, surge-queue overflows or 5xx errors generated by the LB itself; or you hit limits such as file descriptors, conntrack tables, ephemeral ports or capacity units. Then address it at several levels: - Tune the existing balancer: - worker processes and threads matching cores; - higher file descriptor and connection limits; - keep-alive to clients and upstream connection reuse; - TLS efficiency: session resumption, TLS 1.3, ECDSA certificates (cheaper than RSA), hardware acceleration; - avoiding expensive features on hot paths, such as complex regexes, large Lua scripts, or logging synchronously to slow disks. - Scale up: larger instances with more cores and network bandwidth. - Scale out the LB tier: run several LB instances behind DNS round robin, anycast, ECMP routing, or an L4 balancer in front of L7 proxies (the “L4 then L7” pattern: IPVS, Katran or NLB spreading traffic across HAProxy, Envoy or nginx). Managed LBs scale automatically, but may need pre-warming for sudden very large spikes, or a switch to a higher-capacity type such as NLB for raw TCP. - Offload work: move caching and static content to a CDN, let the CDN or edge terminate TLS, and serve large downloads from object storage. Use DSR for response-heavy L4 traffic. - Reduce work per request: fewer rewrites, simpler rule sets, compression at the right place, and HTTP/2 or HTTP/3 multiplexing to cut connection overhead. - Split by function: separate LBs for different services or traffic types (API, WebSockets, static), or client-side load balancing and service meshes for east-west traffic, so a central LB isn’t in every internal call.
Afterwards, load-test the new design, and monitor LB saturation metrics with alerts before limits are hit.
Read full answerHide full answer
Applies tonginx, HAProxy, Envoy, Linux kernel tuning, AWS ALB/NLB, CDNs
Then give tune, scale up, scale out (L4 → L7 tiers, anycast), offload (CDN, DSR) and reduce work.
Finish with load testing and saturation alerts.
50
Suppose a load balancer is causing increased latency—what potential root causes would you investigate?
KnownRevise
When latency rises at the load balancer, compare total latency with back-end response time. The gap is time spent in or around the LB. Root causes to investigate: - TLS overhead: full handshakes for every connection (no session resumption, keep-alive disabled), RSA 4096 keys, OCSP fetching without stapling, CPU saturation from handshakes. - Connection handling: no upstream keep-alive (a new TCP plus TLS connection to the back end per request), connection pool exhaustion, ephemeral port or conntrack exhaustion, low maxconn causing queueing. - Queueing and saturation: LB CPU, memory or bandwidth limits; worker or thread limits; managed LB scaling lag during spikes. - Algorithm and distribution: requests sent to slow or overloaded servers (round robin ignoring load, sticky sessions, hash hot spots, HTTP/2 or gRPC at L4), slow-start missing for cold instances, or health checks too slow to remove a degraded server. - Retries and timeouts:automatic retries hiding failures but adding latency; long connect timeouts to dead servers. - Buffering behaviour: request or response buffering adding delay for streaming (SSE, gRPC), or disabled buffering making slow clients hold back-end connections. - Expensive processing: complex regex rules, WAF inspection of large bodies, Lua or Wasm filters, compression at high levels, header rewriting, or synchronous logging to slow storage. - DNS issues: the LB resolving back-end names on every request, or slow DNS resolvers. - Network path:cross-zone or cross-region hops, MTU or fragmentation issues, packet loss, a geo-DNS mistake sending users to a distant region, or Nagle’s algorithm and delayed ACK interactions. - Recent changes: new rules, version upgrades, certificate swaps, configuration reloads.
Investigate with LB metrics, logs that split times (connect, TLS, upstream, total), tracing, and packet captures if needed. Then fix the specific cause and re-measure.
# Client-side timing breakdown: DNS, TCP connect, TLS, time to first byte
curl -s -o /dev/null https://shop.example.com/api/items -w \
'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n'
# LB-side: total vs upstream connect/header/response times
log_format t '$request rt=$request_time uct=$upstream_connect_time '
'uht=$upstream_header_time urt=$upstream_response_time ua=$upstream_addr cs=$upstream_cache_status';
A high Tw in HAProxy means requests are queued waiting for a server slot. A high Tc means slow back-end connects. A high Tr means the back end is slow, not the LB.