What is a microservice and how does it differ from a monolithic architecture?
KnownRevise
A microservice is a small, independently deployable service that owns one business capability (orders, payments, search) together with its own data, and talks to other services over the network through well-defined APIs or events. A monolith builds all capabilities into one deployable unit, usually with one shared database. The core difference is the unit of deployment and ownership. In a microservices system, each team can change, deploy and scale its service on its own, at the cost of distributed-system complexity: network calls, partial failures, data consistency across services, and much more operational work.
Read full answerHide full answer
Applies tosoftware architecture (vendor-neutral)
Side-by-Side Comparison
Aspect
Monolith
Microservices
Deployment
One artifact; every change redeploys everything
Each service deploys independently
Scaling
Scale the whole application
Scale only the services under load
Data
Usually one shared database; ACID transactions across modules
Database per service; consistency through events and sagas
Communication
In-process function calls (nanoseconds)
Network calls (milliseconds), which can fail
Technology
One main stack
Each service can choose its stack, within team standards
Team structure
Teams share one codebase
Teams own services end to end
Failure
A memory leak or crash can take down everything
Failures can be isolated, if timeouts, retries and circuit breakers are in place
Testing
Simpler end-to-end tests
Contract tests, more environments
Operations
One thing to monitor
Distributed tracing, centralised logs, service discovery, many pipelines
Starting cost
Low
High: platform, CI/CD, observability
A Concrete Example
An online shop as a monolith: one application contains catalog, cart, orders, payments and shipping modules, and they all use one PostgreSQL database. Placing an order is a single database transaction.
Placing an order is now a distributed workflow. The order service records the order, the payment and shipping services react to events, and failures are handled by compensating actions rather than a single rollback.
When Each Is the Right Choice
Choose a monolith (or modular monolith) when
Choose microservices when
The team is small (a few engineers)
Many teams need to release independently
The domain is still being discovered
Domain boundaries are stable and well understood
You need to move fast with little operational overhead
Parts of the system have very different scaling or reliability needs
Strong transactional consistency is central
Teams can invest in a platform: CI/CD, observability, on-call
A modular monolith, one deployable unit with strict module boundaries and separate schemas per module, is often the best starting point. Well-separated modules can later be extracted into services when there is a clear reason.
Common Misconceptions
“Microservices are always more scalable.” A well-built monolith scales horizontally behind a load balancer. Microservices help when parts need to scale differently.
“Small means a few hundred lines.” Size is about one business capability and one team, not lines of code.
“Splitting the code is enough.” Services that share one database, or must deploy together, form a distributed monolith: the costs of both styles and the benefits of neither.
Pitfalls
Splitting too early, before boundaries are clear, which leads to constant cross-service changes.
Chatty synchronous chains, where one user request makes ten network calls in series.
No ownership: services without a responsible team decay quickly.
Interview Tips
Define a microservice by independent deployability, business capability and data ownership.
Present it as a trade-off: team autonomy and independent scaling in exchange for distributed complexity.
Mention the modular monolith and the distributed monolith anti-pattern.
2
What are the principles behind the microservices architecture?
KnownRevise
Microservices architecture rests on a few connected principles: - Single responsibility around a business capability: each service does one business job. - Independent deployability: a service can be released without coordinating with others. - Decentralised data management: each service owns its data, and others access it only through its API or events. - Loose coupling with high cohesion. - Smart endpoints and dumb pipes: business logic lives in the services, not in the messaging layer. - Design for failure. - Automation of infrastructure and delivery. - Decentralised governance: teams choose their tools within shared standards. - Observability.
Most of these principles were set out by James Lewis and Martin Fowler in 2014. They exist so that teams can change parts of a system independently and safely.
Read full answerHide full answer
Applies tosoftware architecture (vendor-neutral)
The Principles in Practice
Principle
What it means
Signs you follow it
Business capability focus
Services map to capabilities (billing, catalog), not to technical layers (a “database service”)
A feature usually changes one service
Single team ownership
“You build it, you run it”: a team owns code, deployment and on-call
Clear owner for every service
Independent deployability
Release one service without releasing others
No coordinated “release trains” across services
Decentralised data
Database per service; no reaching into another service’s tables
Data shared via APIs or events only
Loose coupling, high cohesion
Few dependencies between services; related logic stays together
Changes rarely ripple across services
Explicit contracts
Versioned APIs and event schemas; backward compatibility
Contract tests in CI
Smart endpoints, dumb pipes
Business rules in services; brokers and HTTP just carry messages
No business logic in the message bus or gateway
Design for failure
Assume every call can fail or be slow
Timeouts, retries with backoff, circuit breakers, fallbacks
Automation
CI/CD, infrastructure as code, automated tests
A service can go from commit to production without manual steps
Observability
Logs, metrics and traces correlated across services
Any request can be traced end to end
Decentralised governance
Teams choose suitable tools within guardrails
A paved-road platform, with exceptions justified
Evolutionary design
Expect services to be split, merged or replaced
Strangler-style migrations are routine
How the Principles Support Each Other
Data ownership makes independent deployment possible: if two services share tables, a schema change forces them to deploy together.
Explicit, backward-compatible contracts let consumers upgrade on their own schedule.
Design for failure is needed because network calls replace in-process calls.
Automation and observability are what make running dozens of services manageable.
Related Laws and Ideas
Idea
Relevance
Conway’s law
Systems mirror the communication structure of the organisation that builds them; align service boundaries with teams
Bounded contexts (Domain-Driven Design)
A good way to find service boundaries: one consistent model and language per context
The fallacies of distributed computing
The network is not reliable, latency is not zero, bandwidth is not infinite, and so on
Twelve-factor apps
Configuration in the environment, stateless processes, disposability, logs as event streams
Anti-Patterns That Break the Principles
Anti-pattern
Principle violated
Shared database across services
Decentralised data, independent deployability
Services that must be deployed together
Independent deployability
“Entity services” (a CustomerService that is only CRUD) with logic spread elsewhere
Business capability focus, high cohesion
Long synchronous call chains
Design for failure, loose coupling
An enterprise service bus with routing and business rules
Smart endpoints, dumb pipes
No owner after the project ends
Team ownership
Interview Tips
Lead with business capability, independent deployability and data ownership; most other principles follow from these.
Mention Conway’s law and bounded contexts for finding boundaries.
Give one anti-pattern (the shared database) to show you know what violating the principles looks like.
3
What are the main benefits of using microservices?
KnownRevise
The main benefits of microservices come from independence: - Faster, safer releases: each service deploys on its own, so changes are smaller. - Team autonomy: a team owns a service end to end and is not blocked by others. - Independent scaling: only the busy parts get more resources. - Fault isolation: one failing service need not take down the whole system, if resilience patterns are in place. - Technology flexibility: a service can use the language or database that fits its job. - Easier evolution: a service can be rewritten or replaced without touching the rest.
These benefits are not automatic. They appear only when boundaries are well chosen and the organisation invests in automation, observability and clear ownership.
Read full answerHide full answer
Applies tosoftware architecture (vendor-neutral)
Benefits and What They Depend On
Benefit
What it looks like
Requires
Independent deployment
Teams ship many times a day; a fix to search does not redeploy payments
CI/CD per service, backward-compatible APIs
Team autonomy and scaling the organisation
Teams work in parallel without merge conflicts or release coordination
Services aligned with teams (Conway’s law)
Independent scaling
Scale the checkout service ×10 on sale days, but not the admin service
Stateless services, autoscaling, separate resource limits
Fault isolation
A slow recommendations service degrades one widget instead of the whole page
Timeouts, circuit breakers, bulkheads, fallbacks
Technology fit
Search uses Elasticsearch; payments use PostgreSQL; ML scoring runs in Python
Platform standards so variety stays manageable
Replaceability
Rewrite a service behind the same API
Stable contracts, contract tests
Smaller codebases
New engineers understand one service quickly
Clear boundaries, good documentation
Targeted security and compliance
Only the payment service handles card data, which shrinks the scope of a PCI DSS audit
Network segmentation, least-privilege access
Measuring Whether You Get the Benefits
The four DORA metrics are a common way to check:
Metric
Expected trend with good microservices
Deployment frequency
Up: many small deployments
Lead time for changes
Down: commit to production faster
Change failure rate
Same or down: smaller changes are less risky
Time to restore service
Down: failures are isolated and easier to roll back
If deployment frequency stays the same and teams still coordinate releases, the system may be a distributed monolith.
The Costs That Come With the Benefits
Cost
Why
Operational complexity
Many services to deploy, monitor and secure
Network latency and failures
In-process calls become remote calls
Data consistency
No cross-service ACID transactions; sagas and events instead
Testing complexity
Integration across services, contract tests
Debugging
Requires distributed tracing and correlated logs
Duplicated effort
Each service needs CI, monitoring, security patches
When the Benefits Outweigh the Costs
Large or growing organisations, where team coordination is the bottleneck.
Systems with very different load or reliability needs across features.
Products that must release frequently with low risk.
Organisations with a platform team or mature cloud tooling (Kubernetes, managed databases, observability).
For small teams and early products, a modular monolith usually delivers most of the code-organisation benefits with far less operational cost.
Interview Tips
Name the benefits, then state the conditions each one needs.
Mention DORA metrics as a way to confirm the benefits.
Show balance by listing the costs; interviewers look for that judgement.
4
What are some of the challenges you might face when designing a microservices architecture?
KnownRevise
The hardest microservices challenges are: - Choosing service boundaries: wrong boundaries create constant cross-service changes. - Data consistency: each service has its own database, so there are no simple cross-service transactions. - Distributed-systems failures: latency, timeouts, partial failures and cascading outages. - Operational complexity: deployment, observability, security and configuration across many services. - Testing and debugging across service boundaries. - API and schema evolution without breaking consumers. - Organisational alignment: teams must own services, and boundaries should match how teams communicate.
Most of these do not exist in a monolith; they are the price of independent deployment.
Read full answerHide full answer
Applies tosoftware architecture (vendor-neutral)
Challenges and Mitigations
Challenge
Symptom
Mitigation
Wrong boundaries
Every feature touches several services; services deploy together
Use bounded contexts; start with a modular monolith; merge services that always change together
Data consistency
Orders paid but not reserved; duplicate charges
Sagas with compensations, transactional outbox, idempotent consumers
Distributed data queries
“Show orders with customer names” needs joins across services
API composition, CQRS read models, data replicated through events
Network failures and latency
Slow pages, timeouts, retry storms
Timeouts, retries with backoff and jitter, circuit breakers, bulkheads, caching
Cascading failures
One slow dependency exhausts threads everywhere
Timeouts shorter than callers’ timeouts, load shedding, fallbacks
Observability
Cannot tell which service caused an error
Distributed tracing (OpenTelemetry), correlation IDs, centralised logs and metrics
Deployment complexity
Many pipelines, environments and versions
Standard CI/CD templates, containers, Kubernetes, GitOps
What are the bounded contexts, and which team owns each?
Which data does each service own, and how do others get it?
Synchronous or asynchronous? Which interactions need an immediate answer, and which can be events?
How are failures handled? Timeouts, retries, idempotency and fallbacks for each dependency.
How is a request traced across services?
How do APIs evolve without breaking consumers?
What is the platform? Deployment, service discovery, secrets, logging and metrics.
The Distributed Monolith Trap
Sign
Why it is bad
Services share database tables
Schema changes need coordinated deployments
A change requires releasing several services together
Independent deployment is lost
Long chains of synchronous calls
Latency adds up; availability multiplies down. Five services at 99.9% each give about 99.5% in series
Shared libraries containing business logic
Upgrading the library forces everyone to redeploy
Organisational Challenges
Conway’s law: boundaries that cut across team lines create constant coordination.
Skills: teams need operations, security and distributed-systems knowledge.
Platform investment: without shared tooling, each team reinvents CI, logging and deployment.
Interview Tips
Group your answer: boundaries, data, failures, operations, organisation.
Pair each challenge with a concrete mitigation, such as sagas, OpenTelemetry or contract tests.
Mention the distributed monolith as the most common failure mode.
5
How do microservices communicate with each other?
KnownRevise
Microservices communicate in two main styles. In synchronous request/response, a caller sends a request and waits for the answer, usually over HTTP/REST or gRPC, and sometimes GraphQL at the edge. In asynchronous messaging, a service publishes a message or event to a broker such as Kafka, RabbitMQ, NATS or a cloud queue, and other services consume it later. Synchronous calls are simple and give immediate answers but couple availability and latency: if the callee is down, the caller fails too. Asynchronous messaging decouples services in time and absorbs load spikes, at the cost of eventual consistency and harder debugging. Most systems use both, with service discovery, timeouts and retries, and often a service mesh for routing and security.
Read full answerHide full answer
Applies todistributed system architecture (vendor-neutral)
1. Client --HTTP--> API gateway --gRPC--> order-service (synchronous: user waits for an order ID)
2. order-service writes the order, publishes OrderPlaced --> Kafka topic "orders"
3. payment-service, inventory-service, email-service consume OrderPlaced (asynchronous)
4. payment-service publishes PaymentCompleted --> order-service updates the order status
The user gets a quick response, and the slower steps happen in the background.
Supporting Infrastructure
Concern
Solutions
Service discovery
Kubernetes DNS and Services, Consul, cloud load balancers
Load balancing
Client-side (gRPC) or server-side (Kubernetes Service, Envoy)
Resilience
Timeouts, retries with exponential backoff and jitter, circuit breakers
Security
mTLS between services, JWT or OAuth2 tokens, network policies
Contracts
OpenAPI for REST, .proto files for gRPC, schema registries (Avro, Protobuf, JSON Schema) for events
Service mesh
Istio, Linkerd: mTLS, retries and traffic routing without changing code
Tracing
OpenTelemetry context propagated in HTTP headers and message metadata
Best Practices
Prefer asynchronous events between bounded contexts, and synchronous calls for queries that need an answer now.
Avoid long synchronous chains; each hop adds latency and failure risk.
Make message consumers idempotent, since most brokers deliver at least once.
Use the transactional outbox pattern, so a database write and its event cannot diverge.
Version APIs and event schemas, and keep changes backward compatible.
Set timeouts on every call; never wait indefinitely.
Common Pitfalls
Using a message broker as a remote procedure call system with blocking waits, which brings back synchronous coupling.
Chatty APIs, with many small calls per user action, instead of coarser operations.
Assuming exactly-once delivery without idempotency.
Interview Tips
Name both styles, with examples (REST, gRPC; Kafka, RabbitMQ).
Explain when to use each, with the coupling and consistency trade-off.
Mention discovery, resilience patterns, outbox and idempotency to show production experience.
6
What is Domain-Driven Design (DDD) and how is it related to microservices?
KnownRevise
Domain-Driven Design (DDD), introduced by Eric Evans in his 2003 book, is an approach to software that puts the business domain and its language at the centre of the design. Developers and domain experts build a shared ubiquitous language, and the system is divided into bounded contexts: areas where one model and one vocabulary apply consistently. DDD relates to microservices because bounded contexts are the best-known guide for drawing service boundaries. A service that matches a bounded context owns a coherent model and its data, changes mostly on its own, and maps well to one team. DDD’s tactical patterns (aggregates, entities, value objects, domain events) then shape the code inside each service.
Read full answerHide full answer
Applies tosoftware architecture and domain modelling (vendor-neutral)
Strategic DDD: Finding Boundaries
Concept
Meaning
Microservices relevance
Domain / subdomain
The business area, and its parts: core (competitive advantage), supporting and generic
Invest most in core subdomains; buy or reuse generic ones (authentication, email)
Bounded context
A boundary inside which a model and its terms have one meaning
Usually one service or a small group of services
Ubiquitous language
Shared terms used in conversation and in code
API names and event names use the same terms
Context map
How contexts relate: customer/supplier, conformist, shared kernel, anti-corruption layer, open host service, published language
Defines integration styles between services
Example: “Customer” means different things in different contexts:
Context
“Customer” model
Sales
Leads, contact history, discounts
Billing
Payment methods, invoices, tax ID
Shipping
Delivery addresses, delivery preferences
Forcing one shared Customer model on all three couples them. DDD gives each context its own model, connected by IDs and events.
Tactical DDD: Inside a Service
Pattern
Purpose
Entity
Object with an identity that persists over time (Order #123)
Value object
Immutable, compared by value (Money, Address)
Aggregate
A cluster of objects changed together, with one aggregate root that enforces its rules; the unit of a transaction
Repository
Loads and saves aggregates
Domain event
Something meaningful that happened (OrderPlaced), often published to other services
Domain service
Logic that does not belong to a single entity
Aggregates and microservices: keep each transaction inside one aggregate, and therefore inside one service. Changes that span aggregates or services happen through domain events and eventual consistency.
How DDD Shapes a Microservices Design
Event storming: workshops with domain experts that map commands, events and policies on a timeline. They reveal natural boundaries.
Group the results into bounded contexts.
Draw a context map and choose integration patterns, such as an anti-corruption layer in front of a legacy system.
Make each context a service (or a module in a modular monolith), owned by one team.
Publish domain events for other contexts; avoid sharing internal models.
Benefits and Cautions
Benefits
Cautions
Boundaries follow the business, so they are more stable
Requires access to domain experts and time for modelling
Less accidental coupling between services
One bounded context is not always one service; a context can contain several services, and sometimes a monolith module is better
Clear language in code and APIs
Tactical patterns can be over-engineering for simple CRUD areas
Interview Tips
Define bounded context and ubiquitous language; these are the core of the connection.
Use the “Customer means different things” example.
Mention aggregates as transaction boundaries and domain events for cross-service communication.
7
How would you decompose a monolithic application into microservices?
KnownRevise
Decompose a monolith incrementally, not with a big-bang rewrite. First understand the domain and the code: identify bounded contexts and business capabilities, and measure which modules change together and which data they use. Next, modularise inside the monolith, giving modules clear interfaces and separate schemas. Then extract one service at a time using the strangler fig pattern: put a routing layer in front of the monolith, build the new service, move traffic to it gradually, and remove the old code. Begin with a capability that has clear boundaries and real value (independent scaling or fast change). Move its data with it, and keep the two systems consistent during the move with events or change-data-capture.
Read full answerHide full answer
Applies tosoftware architecture and migration (vendor-neutral)
Step by Step
Step
Activities
Output
1. Set goals
Why decompose? Faster releases, scaling a hotspot, team autonomy
Measurable targets (for example, deployment frequency)
2. Map the domain
Event storming, capability mapping, interviews with domain experts
Candidate bounded contexts
3. Analyse the code
Dependency graphs, change coupling from version-control history, database table usage
Modules and the data they touch
4. Modularise in place
Enforce module boundaries, give each module its own schema, add internal APIs
A modular monolith
5. Choose the first service
Loosely coupled, valuable, low risk; not the most entangled core
One extraction candidate
6. Strangle
Routing layer (API gateway, reverse proxy) sends some requests to the new service
Traffic split
7. Move the data
Dual writes through events, change-data-capture (Debezium), backfill, then cut over
Service owns its data
8. Remove the old code
Delete the monolith’s module once traffic and data are migrated
Routing is controlled per endpoint, and often per percentage of traffic, so it can be rolled back instantly.
Decomposition Strategies
Strategy
Idea
Watch out for
By business capability
Services match what the business does (billing, catalog)
Needs business input
By subdomain (DDD)
Core, supporting and generic subdomains
Takes modelling effort
By volatility
Extract parts that change most often
May not match team boundaries
By scaling needs
Extract the hotspot (search, image processing)
Do not split only for technical reasons if boundaries suffer
Anti-corruption layer
A translation layer between new services and the legacy model
Extra code to maintain during the migration
Branch by abstraction
Put an interface in front of a module, build the new implementation behind it, switch
Works inside the codebase, before extraction
Handling the Data
The database is usually the hardest part:
Find the tables each capability owns, and the ones it only reads.
Replace cross-module joins with API calls or replicated read models.
Give the new service its own schema or database. Synchronise it with change-data-capture or events during the transition.
Switch writes to the new service, then stop the old writes.
Remove foreign keys that cross the boundary; enforce those rules in code.
Pitfalls
Big-bang rewrites, which often run late while the monolith keeps changing.
Splitting by technical layer (a UI service, a logic service, a data service), which creates tight coupling.
Leaving a shared database, which produces a distributed monolith.
Too many tiny services at once, outpacing the platform’s maturity.
No observability before migrating, so problems in the new service go unnoticed.
Interview Tips
Stress an incremental strangler fig migration and modularising first.
Explain how to pick the first service, and how to migrate data with change-data-capture.
Mention measuring success, for example with DORA metrics, after each extraction.
8
What strategies can be employed to manage transactions across multiple microservices?
KnownRevise
Each microservice owns its database, so a business operation that spans services, such as “reserve stock, charge payment, create shipment”, cannot use one ACID transaction. The main strategies are: - Sagas: a sequence of local transactions, each with a compensating action that semantically undoes it if a later step fails. Sagas are coordinated either by orchestration (a central coordinator) or by choreography (services react to each other’s events). - The transactional outbox, which makes “update my database and publish an event” atomic. - Idempotent operations, so retries are safe. - Two-phase commit (2PC/XA), which exists but is usually avoided, because it blocks, couples availability and is poorly supported by modern brokers and cloud databases.
Read full answerHide full answer
Applies todistributed system architecture (example in Python 3.12)
The second order fails at payment, so the saga runs the compensation for every completed step in reverse order. Here that releases the reserved stock. In production, each step is a call to another service, and the saga’s progress is persisted, so it can resume after a crash.
Strategies Compared
Strategy
Consistency
Availability
Complexity
Use when
Saga, orchestration
Eventual; explicit compensations
High
Coordinator to build or adopt (Temporal, AWS Step Functions, Camunda)
Multi-step workflows with clear business order
Saga, choreography
Eventual
High
Logic spread across services; harder to follow
Few steps, loosely coupled reactions
Transactional outbox
Atomic local write + event
High
Outbox table and a relay process (or CDC such as Debezium)
Any service that publishes events after a state change
Two-phase commit (2PC/XA)
Atomic across participants
Lower: blocks if the coordinator fails
High; limited support
Rare, within one trusted environment that supports XA
Avoid the distributed transaction
Strong within one service
High
Redesign
When one service can own the whole invariant
Designing Compensations
Step
Compensation
Notes
Reserve stock
Release stock
Easy to undo
Charge card
Refund
Visible to the customer; may have fees
Send email
Send a correction email
Cannot truly undo
Ship parcel
Cannot undo: this is the pivot step
Put irreversible steps last
Order steps so that the retryable and reversible steps come first and the irreversible step (the pivot) comes after everything that can fail.
Making Sagas Reliable
Persist saga state (step, status) so a crash resumes instead of losing track.
Make every action and compensation idempotent, for example keyed by order ID, because retries will happen.
Use timeouts and retries with backoff, then escalate to manual handling.
Handle isolation anomalies: other requests can see intermediate states. Use semantic locks (an order status of PENDING), commutative updates, or rereading values before acting.
Publish events through a transactional outbox, so the database change and the message are never out of sync.
Choreography Example
OrderService: OrderCreated --> InventoryService reserves --> StockReserved
PaymentService: on StockReserved, charges --> PaymentFailed
InventoryService: on PaymentFailed, releases stock
OrderService: on PaymentFailed, marks the order CANCELLED
Choreography has no central coordinator, but the flow is implicit. Distributed tracing and clear event documentation become essential.
Interview Tips
Say clearly: “no distributed ACID; use sagas with compensations, the outbox and idempotency”.
Contrast orchestration and choreography, and name tools such as Temporal or Step Functions.
Explain why 2PC is avoided: blocking, coupling and limited broker support.
9
Explain the concept of ‘Bounded Context’ in the microservices architecture.
KnownRevise
A bounded context is a boundary, inside which one domain model and one vocabulary apply consistently. The term comes from Domain-Driven Design. Inside the boundary, a word such as “Order” or “Customer” has exactly one meaning, with its own rules and data. Outside it, other contexts may use the same word differently. In microservices, bounded contexts are the main tool for drawing service boundaries: a service, or a small group of services owned by one team, implements one context, owns its data, and integrates with other contexts through explicit APIs and events rather than a shared model or a shared database.
Read full answerHide full answer
Applies tosoftware architecture and Domain-Driven Design (vendor-neutral)
Example: One Word, Several Models
Context
What “Product” means
Data it owns
Catalog
Name, description, images, categories
Marketing content
Pricing
Price lists, discounts, currency rules
Prices and promotions
Inventory
Stock per warehouse, reservations
Quantities
Shipping
Weight, dimensions, hazardous flags
Logistics attributes
All four share a product ID, but none shares the others’ model. If the catalog team changes how descriptions work, the inventory service is not affected.
How to Recognise Context Boundaries
The language changes: domain experts use the same word with different meanings, or different words for the same thing.
The rules change: for example, “an order is complete” means paid to billing and delivered to shipping.
Different teams or departments own the processes.
Different rates of change or different consistency needs.
Event storming workshops show clusters of commands and events with few links between them.
Integrating Contexts: Context Mapping
Relationship
Meaning
Microservices form
Customer / supplier
The upstream context serves the downstream context’s needs
API designed with the consumer’s input
Conformist
The downstream context accepts the upstream model as it is
Consuming a third-party API directly
Anti-corruption layer (ACL)
The downstream context translates the upstream model into its own
An adapter or translation service in front of a legacy system
Open host service + published language
The upstream context offers a stable, documented protocol
Versioned public API or event schema
Shared kernel
Two contexts share a small, jointly owned model
A shared library; use sparingly
Separate ways
No integration
Duplicate a little logic instead of coupling
Bounded Contexts and Services
Guideline
Why
One context per service is a good default
Clear ownership and model
A context may contain several services (for scaling or deployment reasons)
Context is a model boundary, service is a deployment unit
Never split one aggregate across services
Its invariants need one transaction
Each context owns its data store
Prevents hidden coupling through shared tables
Contexts communicate with IDs and events, not shared objects
Keeps models independent
Example of Integration
Ordering context publishes: OrderPlaced { orderId, customerId, lines: [{productId, qty}] }
Inventory context consumes it and translates it into its own model:
Reservation { reservationId, warehouseId, sku, quantity }
The inventory team never imports the ordering team’s Order class.
Pitfalls
One enterprise-wide model (a “canonical data model”) that every service must use, which couples everything.
Contexts that are too small, so a single business change touches several services.
Leaking internal models in events. Publish integration events designed for consumers, not internal database rows.
Ignoring the organisation: boundaries that cross team lines create constant coordination (Conway’s law).
Interview Tips
Define it as “a boundary within which a model and its language are consistent”.
Use a multi-meaning word example (Product, Customer, Order).
Mention context mapping, especially the anti-corruption layer, and the link to data ownership.
10
How do you handle failure in a microservice?
KnownRevise
In microservices, failure is normal: networks drop packets, dependencies slow down, pods restart and deployments go wrong. Handle it in layers: - Detect failures quickly, with timeouts on every call and health checks. - Recover from transient errors with bounded retries, using exponential backoff and jitter, and only for idempotent operations. - Contain damage with circuit breakers, bulkheads (separate resource pools) and load shedding. - Degrade gracefully with fallbacks such as cached data or default responses. - Make asynchronous work durable with persistent queues and dead-letter queues.
Then learn from failures through observability, SLOs and chaos testing.
Read full answerHide full answer
Applies todistributed system design (example in Python 3.12)
Code Example: Retries with Backoff and Jitter
import random
def retry_delays(attempts: int, base: float = 0.1, cap: float = 5.0, rng=random.random) -> list[float]:
"""Exponential backoff with 'full jitter': sleep a random time in [0, min(cap, base * 2**n)]."""
return [rng() * min(cap, base * 2 ** n) for n in range(attempts)]
def call_with_retry(func, attempts: int = 4, retryable=(TimeoutError, ConnectionError), sleep=None, rng=random.random):
"""Retry only transient errors, a bounded number of times, with jittered backoff."""
delays = retry_delays(attempts - 1, rng=rng)
for attempt in range(attempts):
try:
return func()
except retryable:
if attempt == attempts - 1:
raise
(sleep or (lambda s: None))(delays[attempt])
random.seed(1)
print([round(d, 3) for d in retry_delays(5)]) # [0.013, 0.169, 0.306, 0.204, 0.793]
outcomes = iter([TimeoutError(), ConnectionError(), "inventory: 12 left"])
def flaky():
result = next(outcomes)
if isinstance(result, Exception):
raise result
return result
slept = []
print(call_with_retry(flaky, sleep=slept.append, rng=lambda: 1.0)) # inventory: 12 left
print(slept) # [0.1, 0.2]
try:
call_with_retry(lambda: 1 / 0) # not retryable: fails at once
except ZeroDivisionError:
print("ZeroDivisionError raised without retrying")
Jitter spreads retries out over time, so thousands of clients do not retry at the same instant after an outage (a “retry storm”). Errors that are not transient, such as a validation error or a 4xx response, are not retried.
Failure-Handling Toolkit
Technique
Protects against
Notes
Timeouts
Waiting forever on a slow dependency
Set per call; a caller’s timeout should be longer than the total of its callee’s retries
Retries with backoff and jitter
Transient network errors, brief overload
Retry only idempotent requests, or use idempotency keys
Idempotency keys
Duplicate side effects from retries
An Idempotency-Key header lets the server store and replays the first result
Circuit breaker
Repeatedly calling a failing dependency
Fail fast, recover with half-open trials
Bulkheads
One dependency exhausting all threads or connections
Distributed tracing (OpenTelemetry) shows which hop failed.
SLOs and error budgets decide how much failure is acceptable and when to slow releases.
Chaos engineering (Chaos Mesh, LitmusChaos, AWS Fault Injection Service) tests these defences on purpose.
Blameless postmortems turn incidents into fixes.
Pitfalls
Retrying non-idempotent operations, causing duplicate charges or orders.
Retries at every layer: three layers with 3 retries each can multiply load by up to 27×.
No timeouts, or timeouts longer than the caller’s.
Health checks that call dependencies, so one database outage restarts every pod.
Interview Tips
Walk through detect, recover, contain, degrade, learn.
Mention timeouts, retries with jitter, idempotency, circuit breakers and bulkheads by name.
Warn about retry amplification.
11
What design patterns are commonly used in microservices architectures?
KnownRevise
Microservices patterns fall into a few families: - Decomposition: split by business capability or subdomain; the strangler fig for migration. - Integration and communication: API gateway, backend for frontend, asynchronous messaging, service discovery. - Data management: database per service, saga, CQRS, event sourcing, transactional outbox, API composition. - Resilience: timeout, retry, circuit breaker, bulkhead, rate limiter, fallback. - Observability: log aggregation, distributed tracing, health check API, metrics. - Deployment and infrastructure: service per container, sidecar, service mesh, blue-green and canary releases, externalised configuration.
Chris Richardson’s microservices pattern language catalogues most of these.
Read full answerHide full answer
Applies tosoftware architecture (vendor-neutral)
Decomposition
Pattern
Problem it solves
Decompose by business capability
Services aligned with what the business does
Decompose by subdomain (DDD)
Boundaries follow bounded contexts
Strangler fig
Migrate a monolith incrementally behind a routing layer
Anti-corruption layer
Keep a legacy model from leaking into new services
Integration and Communication
Pattern
Problem it solves
API gateway
One entry point for clients: routing, authentication, rate limiting
Backend for frontend (BFF)
A separate gateway per client type (web, mobile) with tailored APIs
API composition / aggregator
Combine data from several services into one response
Asynchronous messaging, publish/subscribe
Decouple services in time
Service discovery (client-side or server-side)
Find service instances whose addresses change
Request/reply over messaging
Long-running work with a later response
Data Management
Pattern
Problem it solves
Database per service
Independent schemas and deployments
Saga (orchestration or choreography)
Business transactions across services
Transactional outbox
Atomically update the database and publish an event
CQRS
Separate write and read models; read models built from events
Event sourcing
Store state as an append-only sequence of events
Change-data-capture
Stream database changes to other services (Debezium)
Idempotent consumer
Safe handling of duplicate messages
Resilience
Pattern
Problem it solves
Timeout
Never wait indefinitely
Retry with backoff and jitter
Transient failures
Circuit breaker
Stop calling a failing dependency
Bulkhead
Isolate resources per dependency
Rate limiter / load shedding
Protect services from overload
Fallback
Degrade gracefully
Observability
Pattern
Problem it solves
Log aggregation
Search logs from all services in one place
Distributed tracing
Follow a request across services (OpenTelemetry)
Correlation ID
Link logs, metrics and traces for one request
Health check API
Liveness and readiness for orchestrators and load balancers
Application metrics
Latency, traffic, errors, saturation
Deployment and Infrastructure
Pattern
Problem it solves
Service per container
Isolated, reproducible deployments
Sidecar
Add proxies, log shippers or agents without changing the service
Service mesh
mTLS, retries and traffic shifting in the infrastructure layer
Externalised configuration
Same build in every environment; configuration from the environment
Blue-green, canary, feature flags
Low-risk releases
How They Fit Together in One Flow
A mobile client calls a BFF. The BFF routes through the API gateway to the order service, applying timeouts and a circuit breaker. The order service writes to its own database and an outbox table. A relay publishes OrderPlaced, and a saga coordinates payment and inventory. Consumers are idempotent, a CQRS read model powers order history, and tracing follows the whole request.
Pitfalls
Applying every pattern from day one. Event sourcing and CQRS in particular add significant complexity; use them where they solve a real problem.
Treating the gateway as a place for business logic.
Implementing resilience inconsistently across services; shared libraries or a mesh help.
Interview Tips
Organise the answer by family, and give one sentence on the problem each pattern solves.
Show judgement: say which patterns are default (database per service, timeouts, tracing) and which are situational (event sourcing, CQRS).
12
What is the API Gateway pattern, and what are its benefits?
KnownRevise
The API gateway pattern places one entry point between clients and the microservices behind it. Clients call the gateway, and the gateway routes each request to the right service. It also handles cross-cutting concerns in one place: - authentication and authorisation, rate limiting, TLS termination; - request and response transformation, protocol translation (for example REST to gRPC), caching; - aggregation of several service calls into one response; - logging, metrics and tracing.
Clients get a simple, stable API, and internal services can change, split or move without clients noticing. A common variant is the backend for frontend (BFF), one gateway per client type.
Read full answerHide full answer
Applies todistributed system architecture; products such as Kong, Envoy Gateway, NGINX, AWS API Gateway, Azure API Management, Apigee and Spring Cloud Gateway
Architecture
Web app ----\ /--> user-service
Mobile app ---> API gateway (auth, ---> order-service
Partners ---/ rate limits, routing, \--> catalog-service
TLS, tracing) \-> payment-service
Validate a JWT or OAuth2 access token once at the edge
Authorisation (coarse)
Check scopes such as orders:read; detailed rules stay in services
Rate limiting and quotas
100 requests per minute per API key
TLS termination
HTTPS at the edge; mTLS or a mesh inside
Aggregation
GET /home combines profile, recommendations and cart
Protocol translation
Public REST/JSON to internal gRPC
Caching
Cache catalog responses for 60 seconds
Observability
Access logs, metrics, trace context injected into requests
Versioning and routing rules
/v1 and /v2 to different backends; canary traffic splitting
Benefits
Benefit
Why it matters
Simpler clients
One host and one API style instead of many services
Encapsulation
Internal services can be refactored without breaking clients
Consistent security
Authentication, rate limits and TLS applied uniformly
Fewer round trips
Aggregation reduces chattiness, which matters on mobile networks
Central policy and observability
One place to see and control external traffic
Safer releases
Traffic shifting for canary and blue-green releases
Drawbacks and Mitigations
Drawback
Mitigation
Single point of failure
Run several instances across zones, behind a load balancer
Extra network hop and latency
Keep gateway logic light; co-locate it with the services
Bottleneck for teams (one team owns all routes)
Self-service configuration, routes defined by each service team (GitOps)
Business logic creeping in
Keep only cross-cutting concerns in the gateway; business rules belong in services
“One size fits all” API
Use BFFs for clients with different needs
API Gateway Versus Related Components
Component
Focus
API gateway
North-south traffic: external clients to services
Service mesh
East-west traffic: service to service, with mTLS and retries
Load balancer
Distributes connections; little API awareness
BFF
A gateway shaped for one client (web, iOS, partner)
Kubernetes Gateway API
A standard Kubernetes API for configuring gateways and routes (Gateway, HTTPRoute)
Interview Tips
Define it as one entry point for clients, handling routing and cross-cutting concerns.
List the benefits and the drawbacks with mitigations, especially the risk of a single point of failure and of business logic creeping in.
Mention BFF and the difference between a gateway and a service mesh.
13
Explain the ‘Circuit Breaker’ pattern. Why is it important in a microservices ecosystem?
KnownRevise
A circuit breaker wraps calls to a dependency and stops sending requests when the dependency keeps failing, much like an electrical breaker that trips to protect a circuit. It has three states: - Closed: calls pass through, and failures are counted. - Open: after too many failures, calls fail fast or return a fallback without contacting the dependency. - Half-open: after a cool-down period, a few trial calls test whether the dependency has recovered; success closes the circuit, and failure opens it again.
It matters in microservices because a slow or failing service can tie up threads and connections in every caller, which spreads the outage (a cascading failure). Failing fast protects callers, gives the struggling service room to recover, and keeps the user experience degraded rather than broken.
Read full answerHide full answer
Applies todistributed system design (example in Python 3.12)
Code Example
import time
from enum import Enum
class State(Enum):
CLOSED = "closed" # calls flow normally; failures are counted
OPEN = "open" # calls fail fast without touching the dependency
HALF_OPEN = "half_open" # one trial call decides whether to close again
class CircuitOpenError(Exception):
pass
class CircuitBreaker:
def __init__(self, failure_threshold: int = 3, reset_timeout: float = 30.0, clock=time.monotonic) -> None:
self.failure_threshold, self.reset_timeout, self.clock = failure_threshold, reset_timeout, clock
self.state, self.failures, self.opened_at = State.CLOSED, 0, 0.0
def call(self, func, *args, fallback=None):
if self.state is State.OPEN:
if self.clock() - self.opened_at >= self.reset_timeout:
self.state = State.HALF_OPEN # allow one trial request
else:
if fallback is not None:
return fallback()
raise CircuitOpenError("dependency unavailable; failing fast")
try:
result = func(*args)
except Exception:
self._record_failure()
if fallback is not None:
return fallback()
raise
self.state, self.failures = State.CLOSED, 0 # success closes the circuit
return result
def _record_failure(self) -> None:
self.failures += 1
if self.state is State.HALF_OPEN or self.failures >= self.failure_threshold:
self.state, self.opened_at = State.OPEN, self.clock()
now = [0.0] # fake clock so the example runs instantly
breaker = CircuitBreaker(failure_threshold=3, reset_timeout=30, clock=lambda: now[0])
healthy = [False]
calls = []
def get_recommendations(user_id: int) -> list[str]:
calls.append(user_id)
if not healthy[0]:
raise TimeoutError("recommendation service timed out")
return ["book-17", "book-42"]
fallback = lambda: ["bestseller-1"] # degrade gracefully instead of failing the page
for _ in range(5):
print(breaker.call(get_recommendations, 7, fallback=fallback), breaker.state.value)
# ['bestseller-1'] closed failure 1
# ['bestseller-1'] closed failure 2
# ['bestseller-1'] open failure 3 opens the circuit
# ['bestseller-1'] open fallback served without calling the service
# ['bestseller-1'] open
print(len(calls)) # 3: the last two calls never reached the service
now[0], healthy[0] = 31.0, True # time passes; the service recovers
print(breaker.call(get_recommendations, 7, fallback=fallback), breaker.state.value)
# ['book-17', 'book-42'] closed (the half-open trial succeeded)
After the third failure, the circuit opens. The next two calls return the fallback without contacting the service, so calls stays at 3. After the reset timeout, a half-open trial succeeds and the circuit closes.
State Machine
From
Event
To
Closed
Failures reach the threshold (count or error rate in a window)
Open
Open
Reset timeout elapses
Half-open
Half-open
Trial call succeeds
Closed
Half-open
Trial call fails
Open (timer restarts)
Why It Is Important
Without a breaker
With a breaker
Every request waits for a timeout against a dead service
Requests fail in microseconds or use a fallback
Thread pools and connection pools fill up, so unrelated endpoints fail too
Resources stay available for healthy work
Retries pile more load onto an overloaded service
Load drops, so the service can recover
Users see errors or very slow pages
Users see a degraded but working page
Configuration Choices
Setting
Typical value
Consideration
Failure threshold
50% errors over at least 20 calls in a window
Counts or rates; avoid tripping on low traffic
What counts as failure
Timeouts, connection errors, 5xx
Do not count 4xx client errors
Open duration
10–60 s
Long enough for recovery; short enough to notice it
Half-open trial calls
1–10
Several trials give a more reliable signal
Slow-call threshold
Calls over 2 s count as failures
Catches dependencies that hang rather than error
Implementations
Layer
Examples
Libraries
Resilience4j (Java), Polly (.NET), gobreaker (Go), opossum (Node.js), pybreaker (Python)
Service mesh and proxies
Envoy and Istio outlier detection (ejects unhealthy hosts) and connection-pool limits
Framework integrations
Spring Cloud Circuit Breaker
Netflix Hystrix popularised the pattern but has been in maintenance mode since 2018; Resilience4j is the usual Java replacement.
Combining with Other Patterns
Timeouts define what counts as a slow failure.
Retries go inside the breaker, with a small budget, so repeated failures still trip it.
Bulkheads limit concurrent calls per dependency.
Fallbacks provide the degraded response.
Metrics and alerts on breaker state changes show outages early.
Pitfalls
One breaker shared by all endpoints of a service, so one bad endpoint blocks the good ones; scope breakers per dependency or operation.
Fallbacks that call another fragile dependency.
Thresholds that trip on a few failures at low traffic; require a minimum number of calls.
Hiding outages: alert when breakers open, or failures go unnoticed.
Interview Tips
Draw the three states and their transitions.
Explain cascading failures and resource exhaustion as the reason it exists.
Mention Resilience4j or Envoy outlier detection, and combining it with timeouts, retries and fallbacks.
14
What is a ‘Service Mesh’? How does it aid in managing microservices?
KnownRevise
A service mesh is an infrastructure layer that handles service-to-service communication for you. Proxies run next to each service, either as sidecars (for example Envoy in Istio, or Linkerd’s proxy) or as shared node-level proxies in “ambient” or sidecarless modes. They intercept traffic and provide mutual TLS encryption and identity, retries, timeouts and circuit breaking, traffic routing (canary releases, traffic mirroring, fault injection), authorisation policies and uniform metrics and traces, all without changing application code. A control plane (such as Istio’s istiod) configures the proxies from declarative policies.
Read full answerHide full answer
Applies toKubernetes service meshes (Istio 1.22+ with the `v1` APIs, Linkerd, Cilium Service Mesh, Consul)
Architecture
control plane (policies, certificates, service discovery)
| configures
+-----------------+------------------+
| Pod A Pod B |
| [app] <-> [proxy] ==mTLS== [proxy] <-> [app]
+------------------------------------+
data plane: every request passes through the proxies
What a Mesh Provides
Capability
Example
mTLS and workload identity
Encrypts traffic and authenticates services by identity (SPIFFE IDs), with automatic certificate rotation
Authorisation
“Only order-service may call payment-service on POST /charges”
10% of traffic goes to v2. Failing instances are ejected for 30 seconds. Every workload in the mesh must use mTLS.
Benefits and Costs
Benefits
Costs
Consistent security and resilience across languages
Extra latency per hop (typically around a millisecond or less per proxy) and CPU and memory for proxies
No library changes in each service
Operational complexity: upgrades, debugging the proxy layer
Central traffic control for safe releases
Another system to learn and secure
Uniform telemetry
Can hide problems if teams do not understand the mesh’s behaviour
Ambient mode in Istio (generally available since Istio 1.24) removes per-pod sidecars. A per-node proxy (ztunnel) handles mTLS, and optional waypoint proxies handle Layer 7 features. This lowers resource use.
When to Use a Mesh
Good fit
Probably not needed
Many services in several languages
A handful of services in one language
Zero-trust requirements (mTLS everywhere)
Network security already handled another way
Frequent canary releases and traffic shifting
Simple deployment needs
Platform team available to run it
No capacity to operate another layer
Pitfalls
Double retries, in both the mesh and the application, multiplying load.
Assuming the mesh fixes application bugs: it handles transport, not business logic or data consistency.
Rolling out to everything at once; adopt it namespace by namespace.
Interview Tips
Define it as “an infrastructure layer of proxies, plus a control plane, for service-to-service traffic”.
Name the main capabilities: mTLS, traffic routing, resilience, observability.
Compare it with an API gateway (north-south traffic), and mention the sidecar versus ambient models.
15
How do you ensure data consistency across microservices?
KnownRevise
With a database per service, no single transaction covers several services, so consistency across services is eventual, and it is built from a few reliable techniques: - Clear data ownership: each fact has one owner, and others hold read-only copies. - The transactional outbox: a service updates its data and records the event in the same local transaction. - At-least-once delivery with idempotent consumers. - Sagas with compensating actions for multi-step business operations. - Ordering per entity, for example partitioning messages by order ID. - Reconciliation jobs that detect and repair drift.
Where an invariant truly needs strong consistency, keep that data inside one service.
Read full answerHide full answer
Applies todistributed system design (example in Python 3.12 with SQLite)
Code Example: Transactional Outbox with an Idempotent Consumer
import json
import sqlite3
import uuid
db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE orders (id TEXT PRIMARY KEY, customer_id INTEGER, total_cents INTEGER, status TEXT);
CREATE TABLE outbox (id TEXT PRIMARY KEY, topic TEXT, payload TEXT, published INTEGER DEFAULT 0);
""")
def place_order(customer_id: int, total_cents: int) -> str:
"""The state change and its event commit together, or neither does."""
order_id = str(uuid.uuid4())
with db: # one local ACID transaction
db.execute("INSERT INTO orders VALUES (?, ?, ?, 'PLACED')", (order_id, customer_id, total_cents))
db.execute("INSERT INTO outbox (id, topic, payload) VALUES (?, ?, ?)",
(str(uuid.uuid4()), "orders.placed",
json.dumps({"order_id": order_id, "customer_id": customer_id, "total_cents": total_cents})))
return order_id
def relay(publish) -> int:
"""Background relay: publish unsent events, then mark them. Delivery is at least once."""
rows = db.execute("SELECT id, topic, payload FROM outbox WHERE published = 0 ORDER BY rowid").fetchall()
for event_id, topic, payload in rows:
publish(topic, event_id, json.loads(payload)) # a crash here means a resend later
with db:
db.execute("UPDATE outbox SET published = 1 WHERE id = ?", (event_id,))
return len(rows)
class InventoryConsumer:
"""Idempotent consumer: remembers processed event IDs, so duplicates are harmless."""
def __init__(self) -> None:
self.seen: set[str] = set()
self.reserved = 0
def handle(self, topic: str, event_id: str, event: dict) -> None:
if event_id in self.seen:
return
self.seen.add(event_id)
self.reserved += 1
inventory = InventoryConsumer()
place_order(42, 2599)
place_order(7, 1250)
try:
with db:
db.execute("INSERT INTO orders VALUES ('dup', 1, 1, 'PLACED')")
db.execute("INSERT INTO orders VALUES ('dup', 1, 1, 'PLACED')") # fails: rolls back both
except sqlite3.IntegrityError:
pass
print(relay(inventory.handle)) # 2
print(relay(inventory.handle)) # 0 (nothing left to send)
event_id, topic, payload = db.execute("SELECT id, topic, payload FROM outbox LIMIT 1").fetchone()
inventory.handle(topic, event_id, json.loads(payload)) # broker redelivers a duplicate
print(inventory.reserved, db.execute("SELECT COUNT(*) FROM orders").fetchone()[0]) # 2 2
The order row and its event are written atomically. The relay can crash between publishing and marking an event as sent, which causes a resend. The consumer’s processed-ID check makes that harmless. The failed transaction (the duplicate key) leaves neither an order nor an event.
Techniques
Technique
Guarantees
Notes
Single owner per data item
One source of truth
Other services keep copies updated by events
Transactional outbox
Database change and event never diverge
Relay by polling, or change-data-capture (Debezium) reading the write-ahead log
Idempotent consumers
Duplicates have no extra effect
Store processed event IDs, or use upserts and conditional updates
Saga
Multi-step workflows reach a consistent end state
Compensations for failures
Per-key ordering
Events for one entity are processed in order
Kafka partition key = entity ID; version numbers to reject stale updates
Optimistic concurrency
No lost updates within a service
Version column, UPDATE … WHERE version = ?
Reconciliation
Detects drift from bugs or lost messages
Periodic comparison jobs, alerts
Keep strong invariants in one service
ACID where it matters
For example, account balance and its ledger in one database
Consistency Models in Practice
Model
Meaning
Example
Strong (within a service)
Reads see the latest committed write
Balance checks during payment
Eventual (across services)
Copies converge after a delay
Search index updated seconds after a product change
Read-your-writes (per user)
A user sees their own changes immediately
Read from the owning service right after a write, or show the client’s local state
Causal
Events are seen in cause-then-effect order
OrderPlaced before OrderShipped, using the same partition key
Avoid These Approaches
Approach
Problem
Dual writes (write to the database, then publish to the broker)
A crash between the two loses the event, or publishes an event for a rolled-back change
Shared database
Couples schemas and deployments
Distributed transactions (2PC)
Blocking, poor availability, limited broker support
Synchronous calls to keep copies updated
Couples availability; misses updates if a call fails
Edge Cases
Out-of-order events: include versions or timestamps and ignore older updates.
Event schema changes: version events; consumers must tolerate unknown fields.
Replays: consumers rebuilding state from history must be idempotent too.
Deletion and privacy requests: propagate them as events, and make sure every copy honours them.
Interview Tips
Say “eventual consistency by design”, then explain the outbox + idempotent consumer combination.
Explain why dual writes are unsafe.
Mention sagas for workflows, per-key ordering, and reconciliation as a safety net.
16
Discuss the strategies you would use for testing microservices.
KnownRevise
Testing microservices follows a test pyramid adapted for distribution: - many fast unit tests of business logic; - component tests of each service in isolation, with real infrastructure in containers and test doubles for other services; - contract tests (consumer-driven, for example with Pact), which verify that services agree on APIs and events without deploying them together; - a small number of end-to-end tests for critical user journeys.
Around the pyramid sit non-functional tests (load, resilience and chaos, security) and testing in production with canaries, feature flags and synthetic monitoring. The aim is confidence that each service can be deployed independently.
Read full answerHide full answer
Applies tomicroservices testing strategy (vendor-neutral; tools named as examples)
The Layers
Layer
Scope
Tools
How many
Unit
Functions and classes, no I/O
JUnit, pytest, Go testing, Jest
Many; run in milliseconds
Integration (inside a service)
Repository against a real database, message serialisation
Testcontainers (PostgreSQL, Kafka, Redis in Docker)
Moderate
Component
Whole service through its API, with dependencies stubbed
The consumer (for example the order service) writes tests that describe what it needs from the provider: request, response fields and types.
These tests produce a contract file, which is published to a broker (for example Pact Broker or PactFlow).
The provider’s CI verifies its real implementation against every consumer’s contract.
A deployment check (“can I deploy?”) blocks a release that would break a consumer.
This catches breaking changes before deployment and replaces many slow end-to-end tests.
What to Test in Each Service
Concern
Test
Business rules
Unit tests
Database queries and migrations
Integration tests with Testcontainers; run migrations in CI
Message handling
Publish a test event, assert state; test duplicates and out-of-order delivery
API behaviour
Component tests: status codes, validation, error formats
Timeouts, retries, circuit breakers
Fault-injection tests with Toxiproxy or stubbed slow responses
Backward compatibility
Contract tests; schema compatibility checks
Testing in Production (Safely)
Canary releases with automated rollback on error-rate or latency regressions.
Feature flags: deploy dark, enable for a small group, then expand.
Synthetic monitoring: scripted user journeys run continuously against production.
Traffic mirroring (shadowing): copy real traffic to a new version and discard its responses.
Pitfalls
An “ice cream cone” of many end-to-end tests and few unit tests: slow and flaky.
Shared, long-lived staging environments that are always broken; prefer on-demand, per-branch environments.
Mocks that drift from reality; contract tests keep them honest.
Ignoring asynchronous behaviour; test idempotency, ordering and dead-letter handling.
Test data coupling between services’ tests.
Interview Tips
Present the pyramid and name contract testing as the key microservices-specific layer.
Mention Testcontainers for realistic integration tests.
Add resilience testing and testing in production with canaries and feature flags.
17
How can you prevent configuration drift in a microservices environment?
KnownRevise
Configuration drift happens when the actual state of environments, clusters or services slowly stops matching the intended state. Typical causes are manual hotfixes, one-off kubectl edit commands, settings that differ between staging and production, and unpatched servers. You prevent it by making Git the single source of truth and letting automation apply and continuously reconcile that state: - infrastructure as code (Terraform or OpenTofu, Pulumi, CloudFormation); - declarative Kubernetes manifests with GitOps controllers (Argo CD, Flux) that detect drift and self-heal; - immutable artifacts (the same container image promoted across environments); - externalised, versioned configuration and secrets management; - policy checks in CI and at admission; - restricted manual access, with drift detection and alerts.
Read full answerHide full answer
Applies toKubernetes and cloud infrastructure (Argo CD, Flux, Terraform/OpenTofu)
Example: GitOps with Automatic Self-Healing
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: orders-prod
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/example/deploy-config.git
targetRevision: main
path: services/orders/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: orders
syncPolicy:
automated:
prune: true # delete resources that were removed from Git
selfHeal: true # revert manual changes made in the cluster
syncOptions:
- CreateNamespace=true
If someone runs kubectl scale deploy/orders --replicas=1 in production, Argo CD notices the difference from Git and restores the declared value. The only lasting way to change production is a reviewed pull request.
Prevention Techniques
Technique
How it prevents drift
Infrastructure as code
Networks, databases and queues are defined in code and reviewed
GitOps reconciliation
Controllers compare live state with Git continuously and fix differences
Immutable infrastructure
Replace servers and containers instead of patching them in place
Build once, promote everywhere
The same image digest moves from staging to production; only configuration differs
Environment overlays (Kustomize, Helm values)
Shared base plus small, explicit per-environment differences
OPA Gatekeeper or Kyverno reject non-compliant resources (no latest tags, resource limits required)
Least-privilege access
Humans get read-only production access; changes go through pipelines
Scheduled drift detection
terraform plan in CI on a schedule; alert when it shows changes
Detecting Drift
Layer
Tool or command
Kubernetes workloads
Argo CD or Flux “OutOfSync” status; kubectl diff -f manifests/
Cloud resources
terraform plan (or tofu plan) showing changes nobody made in code; AWS Config rules
Container images
Admission policies requiring signed images from approved registries
Application config
Version or hash of the config exposed in a /info endpoint and compared across environments
OS and packages
Image scanning; rebuild base images regularly instead of patching live hosts
Handling Emergencies Without Drift
Make the emergency change through the normal pipeline, with a fast-track review if needed.
If a manual change is unavoidable, pause auto-sync for that application, apply the change, and commit the same change to Git immediately.
Re-enable reconciliation, and record the incident.
Pitfalls
Settings defined in several places (Helm values, environment variables, a config service) with unclear precedence.
Long-lived feature branches for environments; use directories or overlays instead.
Secrets committed to Git in plain text; use sealed secrets or an external secret store.
Self-heal disabled “temporarily” and never re-enabled.
Interview Tips
Define drift, then answer with IaC + GitOps + immutable artifacts.
Mention self-healing reconciliation (Argo CD selfHeal) and policy as code.
Explain how you would detect drift, not only prevent it.
18
When should you use synchronous vs. asynchronous communication in microservices?
KnownRevise
Use synchronous communication (REST or gRPC request/response) when the caller needs an answer immediately to continue: a query a user is waiting on, an authorisation check, a price quote or input validation. Use asynchronous communication (events or messages through a broker) when the caller does not need the result right away, when several services must react to something that happened, when work is slow or bursty, or when services should stay available even if others are down. A useful default is synchronous for queries and asynchronous for side effects, with long synchronous call chains avoided.
Read full answerHide full answer
Applies todistributed system design (vendor-neutral)
Decision Guide
Question
If yes, lean toward
Does the user or caller need the result to continue?
Synchronous
Is the operation a read or query?
Synchronous (or a local read model built from events)
Should several services react to the same fact?
Asynchronous publish/subscribe
Can the work take seconds or minutes?
Asynchronous, with status polling or notification
Must the caller keep working when the callee is down?
Asynchronous
Are there traffic spikes the callee cannot absorb?
Asynchronous queue to buffer and level the load
Is strict, immediate consistency required?
Synchronous within one service, or redesign so one service owns the invariant
Trade-offs
Aspect
Synchronous
Asynchronous
Coupling
Temporal: both must be up
Decoupled in time
Latency for the caller
Sum of the call chain
Fast acknowledgement; results arrive later
Availability
Multiplies: 99.9% × 99.9% × 99.9% ≈ 99.7% for a chain of three
Each service fails independently
Consistency
Immediate
Eventual
Error handling
Timeouts, retries, circuit breakers
Retries, dead-letter queues, idempotency
Debugging
Simpler
Needs tracing through messages
Back-pressure
Callers get errors under load
Queues absorb bursts; monitor queue lag
Examples
Use case
Choice
Why
Show a product page
Synchronous calls to catalog and pricing (plus caching)
The user is waiting
Place an order
Synchronous to the order service, then asynchronous OrderPlaced
The user needs an order ID; payment, email and analytics follow
Send a confirmation email
Asynchronous
Not needed immediately; retries are fine
Fraud scoring before payment
Synchronous with a strict timeout and fallback
The decision is needed now
Video transcoding
Asynchronous job queue
Slow; report progress
Update a search index
Asynchronous events
Eventual consistency is acceptable
Hybrid Patterns
Asynchronous request/response: return 202 Accepted with a status URL, and complete the work in the background.
Local read models: subscribe to events and keep a local copy of data you would otherwise query synchronously (for example, customer names in the order service).
Synchronous edge, asynchronous core: an API gateway answers the client quickly while internal steps run through events.
Pitfalls
Long synchronous chains (A → B → C → D), where latency and failure probability add up.
Request/reply over a queue with a blocking wait, which keeps the coupling and adds broker complexity.
Asynchronous events for queries, where a user waits for an eventual result; use a synchronous call or a read model instead.
Forgetting idempotency on asynchronous consumers.
No visibility into queue lag, so slow consumers go unnoticed.
Interview Tips
Give the rule of thumb: queries synchronous, side effects asynchronous.
Quantify the availability cost of synchronous chains.
Mention the 202 Accepted pattern and local read models as hybrids.
19
What role does containerization play in microservices?
KnownRevise
Containers package a service with its runtime, libraries and configuration defaults into one immutable image that runs the same on a laptop, in CI and in production. That makes them the standard deployment unit for microservices. Each service gets an isolated, reproducible environment. Images are built once and promoted through environments. Containers start in seconds, which makes scaling and rolling updates fast. Different services can use different languages and versions without conflicts. Orchestrators such as Kubernetes then schedule, scale, heal and connect containers across a cluster, providing service discovery, health checks and rolling deployments.
Read full answerHide full answer
Applies toDocker and OCI containers, Kubernetes
Example: A Small, Secure Image
# syntax=docker/dockerfile:1
FROM golang:1.25 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download # cached layer while dependencies are unchanged
COPY . .
RUN CGO_ENABLED=0 go build -o /out/orders ./cmd/orders
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/orders /orders
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/orders"]
The multi-stage build keeps compilers out of the final image. The distroless base has no shell or package manager, which reduces the attack surface. The service runs as a non-root user.
What Containers Give Microservices
Need
How containers help
Independent deployment
Each service has its own image and version
Consistency across environments
The same image digest runs in dev, staging and production
Polyglot services
Java 21, Python 3.12 and Go side by side, without conflicts on the host
Fast scaling
Start new replicas in seconds
Resource isolation
CPU and memory limits per container (cgroups); namespaces isolate processes and the network
Density
Many containers per host; less overhead than virtual machines
Immutable releases and rollbacks
Roll back by redeploying the previous image tag or digest
Local development
Docker Compose runs a service with its dependencies
Weaker by default; strengthen with seccomp, non-root users, gVisor or Kata Containers
Stronger
Best Practices
One process or concern per container; sidecars for helpers.
Pin base images and rebuild regularly to pick up security patches; scan images (Trivy, Grype).
Stateless containers; keep state in databases, object storage or volumes.
Configuration through environment variables or mounted files, not baked into the image.
Handle SIGTERM for graceful shutdown, so in-flight requests finish during rolling updates.
Set resource requests and limits.
Deploy by immutable tag or digest, never latest.
Pitfalls
Large images with build tools and secrets left inside.
Running as root without need.
Writing important data to the container filesystem, which disappears when the container is replaced.
Treating containers as small virtual machines and patching them in place, which causes drift.
Interview Tips
Say: containers are the packaging and deployment unit; orchestrators run them at scale.
Mention immutability, build once, promote, and multi-stage, non-root images.
Contrast containers with VMs, and note the security hardening options.
20
What are the challenges of deploying microservices?
KnownRevise
Deploying microservices is harder than deploying one application, because there are many independently versioned services. Each needs its own build and release pipeline, and every release must stay compatible with the versions of other services already running. The main challenges are: - pipeline sprawl and consistency across teams; - version compatibility of APIs, events and database schemas; - zero-downtime releases without breaking in-flight requests; - configuration and secrets across environments; - dependencies and ordering between services; - observability, to tell whether a release is healthy; - fast, safe rollback; - environment parity and cost.
The answers are standardised pipelines, backward-compatible changes, progressive delivery (rolling, blue-green, canary), GitOps and automated health checks with rollback.
Read full answerHide full answer
Applies toCI/CD for microservices (Kubernetes and cloud platforms)
Challenges and Mitigations
Challenge
What goes wrong
Mitigation
Many pipelines
Each team builds its own; quality varies
Shared pipeline templates (a “paved road”), reusable workflows
Version labels on metrics and traces; dashboards comparing versions; SLO-based alerts
Environment cost and parity
Full copies of every service per test environment are expensive
Ephemeral preview environments for changed services only, plus shared dependencies or stubs
Coordination across teams
Release trains creep back
Independent deployability as a rule; avoid “deploy together” requirements
A Safe Deployment Flow
Build once: create an image tagged with the commit SHA; scan it and sign it.
Test: unit, component and contract tests; verify the provider against consumer contracts.
Deploy to staging through GitOps; run smoke tests.
Progressive production rollout: a 5% canary, automated comparison of error rate and latency, then 25%, 50%, 100%.
Automatic rollback if SLO checks fail.
Post-deploy monitoring with alerts tied to the new version.
Expand/Contract Migrations
Phase
Database
Code
Expand
Add the new column (nullable)
Old code ignores it
Migrate
Backfill data
New code writes both columns and reads the new one
Contract
Drop the old column after all versions stop using it
Remove the old code paths
Each phase is safe to deploy and roll back independently.
Pitfalls
Breaking changes without versioning.
Deploying many services at once for one feature, which suggests a distributed monolith.
No graceful shutdown, so rolling updates drop connections.
Rollbacks that are never tested.
Interview Tips
Group the challenges into compatibility, release safety, operations and organisation.
Mention expand/contract migrations, contract tests and canary releases with automated rollback.
Stress that independent deployability must be protected deliberately.
21
Describe blue-green deployment and how it applies to microservices.
KnownRevise
Blue-green deployment runs two production environments side by side. Blue serves live traffic, and green has the new version deployed and tested. A release switches the router (load balancer, DNS, Kubernetes Service selector or gateway route) from blue to green in one step. Rollback is just switching back, while blue is still running. In microservices, blue-green is usually applied per service: two versions of one service run side by side behind a stable address, so a team can release and roll back that service on its own. The hardest parts are databases and message consumers, which both versions share.
Before: clients -> router -> blue (v1, live) green (v2, deployed, being tested)
Switch: clients -> router -> green (v2, live) blue (v1, idle, kept for rollback)
Later: blue is updated to the next version and becomes the new standby
Kubernetes Example: Switching a Service Selector
apiVersion: v1
kind: Service
metadata:
name: orders
spec:
selector:
app: orders
track: blue # change to "green" to switch all traffic
ports:
- port: 80
targetPort: 8080
Two Deployments, orders-blue and orders-green, carry track: blue and track: green labels. After green passes its smoke tests on a separate test Service, the switch is one command:
kubectl patch service orders -p '{"spec":{"selector":{"app":"orders","track":"green"}}}'
Rolling back is the same command with blue. Argo Rollouts automates this with its blueGreen strategy, using an active Service and a preview Service.
Benefits and Costs
Benefits
Costs
Instant rollback by switching back
Double capacity during the release
Full testing of the new version in production infrastructure before it gets traffic
All users switch at once, so problems affect everyone (unlike a canary)
No mixed versions serving users (simpler than a rolling update)
Shared state (database, queues) must work with both versions
Predictable, quick cutover
Long-lived connections (WebSockets) need draining
Handling Shared State
Concern
Approach
Database schema
Expand/contract migrations, so blue and green both work with the schema during the switch
Message consumers
Only one colour should consume, or consumers must be idempotent and version-tolerant; pause green’s consumers until the switch
Caches
Version cache keys if the data format changes
Sessions
Store them externally (Redis) or use stateless tokens, so the switch does not log users out
Scheduled jobs
Run them from one colour only, to avoid double execution
Blue-Green Versus Other Strategies
Strategy
Traffic shift
Rollback
Extra capacity
Best for
Rolling update
Gradual, pod by pod
Roll forward or back gradually
Small
Routine, low-risk changes
Blue-green
All at once
Instant
100% during the release
Releases that need fast, clean rollback
Canary
Small percentage first
Fast; few users affected
Small
Risky changes; metrics-driven decisions
Feature flags
Per feature and per user
Toggle off
None
Decoupling deployment from release
Pitfalls
Incompatible database migrations, which make rollback impossible.
Both colours consuming the same queue and processing messages differently.
Forgetting to test green under realistic conditions before switching.
DNS-based switches delayed by client-side caching; prefer load balancer or Service switches.
Interview Tips
Explain two environments, one switch, instant rollback.
Address database compatibility with expand/contract, and message consumers.
Compare it with canary releases: blue-green is all-or-nothing, a canary limits the blast radius.
22
How does canary releasing work, and how is it beneficial for microservices deployments?
KnownRevise
In a canary release, a new version of a service gets a small share of real traffic first, for example 5%, while the stable version serves the rest. The system compares the canary’s metrics (error rate, latency, saturation, business KPIs) with the baseline. If they look healthy, traffic increases step by step to 100%. If not, traffic is rolled back automatically. For microservices, canaries limit the blast radius of each release to a few users and to one service, and they test the change against real production traffic and dependencies, which staging rarely matches. This lets teams deploy often with less risk.
Read full answerHide full answer
Applies toprogressive delivery on Kubernetes (Argo Rollouts, Flagger, service meshes, Gateway API)
Changing the image starts the rollout. An AnalysisTemplate can query Prometheus at each step and abort the rollout automatically if, for example, the error rate rises above 1%.
How Traffic Is Split
Mechanism
Precision
Notes
Replica ratio (1 canary pod of 10)
Coarse
Works without a mesh; depends on pod counts
Service mesh (Istio, Linkerd)
Exact percentages, header-based routing
Can send only internal users or one region to the canary
Gateway API HTTPRoute weights, ingress controllers
Exact weights at the edge
Good for public endpoints
Feature flags
Per user, per account
Deploy dark, then enable for chosen users
What to Measure
Signal
Example check
Error rate
Canary 5xx rate is not more than baseline + 0.5 percentage points
Latency
Canary p99 latency is within 10% of baseline
Saturation
CPU and memory not trending toward limits
Business metrics
Checkout conversion, payment success rate
Logs
No new error signatures
Compare the canary with the baseline running at the same time, not with yesterday’s numbers, to cancel out traffic patterns.
Benefits for Microservices
Benefit
Why
Limited blast radius
A bug reaches 5% of requests to one service, not everyone
Real-world validation
Real traffic, data and dependencies
Automated, objective decisions
Metric analysis replaces manual judgement
Fast rollback
Shift traffic back in seconds
Frequent releases
Lower risk per release encourages small, frequent deployments
Version-aware observability
Metrics labelled by version show regressions quickly
Requirements and Pitfalls
Backward compatibility: both versions run at once, so APIs, events and database schemas must work with both.
Enough traffic: at low volume, 5% may be too few requests to judge; use longer steps or synthetic traffic.
Sticky sessions: keep each user on one version if a mixed experience would confuse them.
Asynchronous consumers: a canary consumer receives a share of messages depending on how the partitions are assigned, not exact traffic weights.
Downstream effects: the canary can still harm shared resources; watch the dependencies too.
Canary Versus Blue-Green
Aspect
Canary
Blue-green
Traffic shift
Gradual
All at once
Users affected by a bad release
Few
Everyone, until rollback
Extra capacity
Small
Full duplicate
Complexity
Needs traffic splitting and metric analysis
Simpler switch
Interview Tips
Describe the loop: small share → measure against baseline → increase or roll back.
Name tools: Argo Rollouts, Flagger, Istio, Gateway API weights.
Mention automated analysis and the backward-compatibility requirement.
23
Explain the concept of ‘Infrastructure as Code’ and how it benefits microservices management.
KnownRevise
Infrastructure as Code (IaC) means defining infrastructure (networks, clusters, databases, queues, DNS, permissions) in version-controlled code files instead of creating it by hand in consoles. Tools such as Terraform or OpenTofu, Pulumi, AWS CloudFormation or CDK, and Crossplane read these definitions and create or update the real resources to match. For microservices, where each service may need its own database, queue, bucket, IAM role and DNS entry, IaC makes environments reproducible, reviewable and consistent. It lets teams create new services and environments quickly from templates, and prevents configuration drift.
terraform plan -var environment=staging shows exactly what will change, and reviewers approve it in a pull request. terraform apply makes the change. The same code creates identical staging and production queues.
Benefits for Microservices
Benefit
Why it matters with many services
Reproducibility
Every environment is built from the same code
Review and audit
Pull requests show who changed what and why; Git history is the audit trail
Self-service templates
A “new service” module creates a repository, pipeline, database, queue and alerts in minutes
Configures servers; less central with immutable containers
Practices
Practice
Detail
Remote state with locking
For example S3 with state locking, or Terraform Cloud and similar; never local state for shared environments
Modules
Reusable building blocks (service database, queue, alerting)
Environments as variables or workspaces
Small, explicit differences between staging and production
CI for infrastructure
fmt, validate, plan on pull requests; security scanning (Checkov, tfsec/Trivy)
Policy as code
OPA, Sentinel or Kyverno rules (no public buckets, encryption required)
Secrets outside code
Reference a secret store; never commit secret values
Small blast radius
Separate state per service or per layer, so one change cannot affect everything
Pitfalls
Manual console changes that the code then overwrites or that cause drift.
One huge state file for the whole company: slow plans and risky applies.
Secrets in state files: state can contain sensitive values, so encrypt it and restrict access.
Unpinned provider versions, which cause surprise changes.
Interview Tips
Define IaC as infrastructure in version-controlled, declarative code, applied by tools.
Link it to microservices: many similar services → templates, consistency, self-service.
Mention plan/review/apply, remote state, modules and policy as code.
24
Describe what Continuous Integration/Continuous Deployment (CI/CD) pipelines look like for microservices.
KnownRevise
In microservices, each service has its own CI/CD pipeline, usually built from a shared template, so that it can be tested, built and deployed independently. A typical pipeline: 1. On every change, runs unit, component and contract tests. 2. Builds one immutable container image, tagged with the commit SHA, then scans and signs it. 3. Pushes the image to a registry. 4. Deploys to staging, often by updating a GitOps repository that Argo CD or Flux applies. 5. Runs smoke tests, then promotes the same image to production with progressive delivery (canary or blue-green), and rolls back automatically if health checks fail.
In monorepos, path filters make sure only the services that changed are built.
Read full answerHide full answer
Applies toCI/CD for containerised microservices (GitHub Actions example; same ideas in GitLab CI, Jenkins, Tekton)
After the image is pushed, a later step, or a tool such as Argo CD Image Updater, commits the new tag to the deployment repository. Argo CD then syncs staging, and production follows after checks pass.
Pipeline Stages
Stage
Activities
Gate
Commit / pull request
Lint, unit tests, static analysis, dependency scan
Must pass to merge
Build
Build the image once; tag it with the commit SHA; SBOM generation
Reproducible artifact
Verify
Component tests with Testcontainers, contract verification (Pact “can I deploy?”), image scan (Trivy, Grype), signing (Sigstore cosign)
No critical vulnerabilities; contracts satisfied
Deploy to staging
GitOps commit; database migrations (expand phase)
Smoke tests pass
Production rollout
Canary or blue-green with automated metric analysis
SLOs met at each step
Post-deploy
Monitoring, alerts per version
Automatic rollback on regression
Design Choices
Decision
Options
Repository layout
One repository per service; or a monorepo with path-based triggers and build caching (Bazel, Nx, Turborepo)
Deployment model
Push (pipeline runs kubectl or helm) or pull/GitOps (Argo CD or Flux syncs from Git)
Templates
Reusable workflows or shared pipeline libraries, so every service gets the same security and quality gates
Environments
Long-lived staging plus ephemeral preview environments per pull request
Promotion
Same image digest promoted; configuration differs by environment
What Makes It “Microservices-Ready”
Independence: a service’s pipeline never waits for other services’ pipelines.
Compatibility checks: contract tests and schema registry checks replace large integration test phases.
Speed: pipelines take minutes, using test parallelism and layer caching.
Safety: progressive delivery and automated rollback.
Traceability: each running version maps to a commit, an image digest and a change record.
Pitfalls
Rebuilding images per environment, so production runs something that was never tested.
Shared end-to-end test stages that block every team.
Secrets in pipeline logs; use OIDC federation to the cloud and masked variables.
Interview Tips
Describe one pipeline per service from a shared template.
Emphasise build once, promote the same image, contract tests, and GitOps with canary releases.
Mention path filters in monorepos and supply-chain security (SBOMs, signing).
25
How do you monitor health and performance of microservices?
KnownRevise
Monitor microservices with three connected signal types: metrics (numbers over time: request rate, errors, latency, saturation), logs (structured events with context) and traces (the path of one request across services), all tagged with the same service, version and trace IDs. Each service exposes health endpoints for the orchestrator (liveness and readiness), and teams define SLOs (for example, “99.9% of checkout requests succeed in under 300 ms”) with alerts on symptoms users feel, not on every CPU spike. OpenTelemetry is the common standard for collecting all three, sent to backends such as Prometheus and Grafana, Loki or Elasticsearch, Jaeger or Tempo, or a commercial platform.
OpenTelemetry SDKs and auto-instrumentation; Prometheus client libraries
Collection
OpenTelemetry Collector, Prometheus, Fluent Bit
Metrics storage
Prometheus with Thanos or Mimir for long-term storage
Logs
Loki, Elasticsearch/OpenSearch
Traces
Jaeger, Grafana Tempo, Zipkin
Dashboards and alerts
Grafana, Alertmanager; commercial platforms such as Datadog, New Relic, Dynatrace
Practices
Dashboards per service with RED metrics, plus a service map showing dependencies.
Version labels on metrics, so a new release can be compared with the old one.
SLOs and error budgets, with burn-rate alerts instead of fixed thresholds.
Correlation: put the trace ID in every log line so you can jump from a slow trace to its logs.
Synthetic checks that run real user journeys from outside.
Runbooks linked from every alert.
Pitfalls
Alert fatigue from paging on causes (CPU 80%) instead of symptoms (users failing).
High-cardinality labels (user IDs in metric labels) that overload Prometheus.
Liveness probes that call databases, restarting every pod during a database outage.
Unstructured logs that cannot be searched or correlated.
Interview Tips
Name metrics, logs and traces, with RED/USE as frameworks.
Explain liveness versus readiness and SLO-based alerting.
Mention OpenTelemetry as the vendor-neutral standard.
26
How do you handle database schema changes in a microservice architecture?
KnownRevise
Each microservice owns its database schema, so only that service changes it, through versioned migration scripts (Flyway, Liquibase, Alembic, Rails or Django migrations) that run automatically in its pipeline. Because old and new versions of the service run at the same time during rolling or canary deployments, every change must be backward compatible. The standard technique is expand and contract: first add (new columns, tables, indexes) in a way the old code ignores; then deploy code that uses the new structure while still tolerating the old; then backfill data; and only after no running version needs the old structure, remove it. Other services are never affected directly, because they access the data only through the service’s API or events.
Read full answerHide full answer
Applies torelational databases in microservices (PostgreSQL examples; Flyway/Liquibase/Alembic)
Expand and Contract: Renaming a Column
Goal: rename customers.fullname to display_name without downtime.
-- Release 1 (expand): add the new column; old code keeps working
ALTER TABLE customers ADD COLUMN display_name text;
-- Release 2: the code writes both columns and reads display_name, falling back to fullname
-- Backfill in batches so the table is not locked for long:
UPDATE customers SET display_name = fullname
WHERE display_name IS NULL AND id BETWEEN 1 AND 10000;
-- Release 3: the code reads and writes only display_name
-- Release 4 (contract): after every running version has stopped using fullname
ALTER TABLE customers DROP COLUMN fullname;
Every release can be rolled back safely, because the schema always supports both the previous and the current code.
Safe and Unsafe Changes
Change
Safe while old code runs?
Approach
Add a nullable column, or one with a default
Yes
Expand step
Add a table or index
Yes
In PostgreSQL, CREATE INDEX CONCURRENTLY avoids blocking writes
Add NOT NULL to an existing column
No
Backfill first; add a CHECK … NOT VALID constraint, validate it, then set NOT NULL
Rename a column or table
No
Expand and contract (add, copy, switch, drop)
Drop a column
Only when no running version reads it
Contract step, releases later
Change a column type
Usually no
New column, backfill, switch
Long data migrations
Risky
Batches, run as a separate job, throttled
Running Migrations
Option
Notes
A step in the CI/CD pipeline before deployment
Simple; fails the release if the migration fails
A Kubernetes Job or init container
Make sure only one runs at a time (migration tools use a lock table)
On application start-up
Risky with many replicas starting at once; avoid for large migrations
Changes That Affect Other Services
API and event contracts change separately from the schema: an internal column rename should not change the public API.
When an event’s shape must change, use schema versioning (a schema registry with compatibility rules) and keep publishing the old format until consumers move.
Read models in other services that copy your data are updated through events, not by reading your tables.
Pitfalls
Shared databases, where one team’s migration breaks another service.
Big, locking migrations in a single transaction on large tables.
Destructive changes in the same release as the code change, which make rollback impossible.
Migrations edited after they were applied; always add a new migration.
No testing against production-like data volumes.
Interview Tips
Lead with “each service owns and migrates its own schema”.
Explain expand and contract, and why both versions must work during deployment.
Mention automated, versioned migrations and online index creation.
27
Discuss the pros and cons of using a shared database vs. a database-per-service.
KnownRevise
With a shared database, several services read and write the same schema. This is easy at first: you get joins, ACID transactions across services and one system to operate, but the services become tightly coupled. A schema change can break other services, teams must coordinate deployments, and one service’s heavy queries slow everyone down. With a database per service, each service owns its data and exposes it only through APIs or events. This keeps services independently deployable and scalable, and lets each choose a suitable storage technology, at the cost of no cross-service joins or transactions, eventual consistency, data duplication and more databases to operate. For real microservices, database per service is the default; a shared database is a transitional step or a deliberate exception.
Read full answerHide full answer
Applies tomicroservices data architecture (vendor-neutral)
Comparison
Aspect
Shared database
Database per service
Coupling
High: schema is a hidden contract
Low: explicit APIs and events
Independent deployment
Hard: migrations must suit every service
Yes
Transactions
ACID across services
Local only; sagas across services
Queries across domains
SQL joins
API composition, CQRS read models, data replication
One database server, one schema or database per service, with separate credentials
Lower cost; still shares performance and failure domain
Shared database during migration
Services extracted from a monolith keep using its database temporarily
Acceptable briefly; plan the split
Read replicas of another service’s data
Another service gets an event-fed copy
Duplication, eventual consistency
The key rule is no service touches another service’s tables, even if they live on the same server.
Handling the Consequences of Database per Service
Need
Pattern
Show data from several services
API composition in a BFF or gateway, or a CQRS read model
Keep a business process consistent
Saga with compensations
Publish changes reliably
Transactional outbox or change-data-capture
Reporting and analytics
Stream data to a warehouse or lake (BigQuery, Snowflake, a lakehouse)
Reference data used everywhere
Owned by one service and replicated through events, or cached
When a Shared Database Can Be Acceptable
A small system with one team, where a modular monolith would be better anyway.
Read-only reporting access through views, with a clear owner.
A migration phase, with a deadline to split.
Pitfalls
“Microservices” that share tables, which is a distributed monolith.
Integration through the database (triggers or polling another service’s tables).
Splitting the database without a plan for queries and reports, leading to slow chains of API calls.
Interview Tips
State the default: database per service, because it preserves independent deployment.
Name the costs and the patterns that handle them: sagas, outbox, CQRS, API composition.
Mention separate schemas on a shared server as a pragmatic middle ground.
28
Explain the concept of ‘Event Sourcing’ in the context of microservices.
KnownRevise
Event sourcing stores the state of an entity as an append-only sequence of events, such as Deposited 100, Withdrawn 30 or OrderShipped, instead of storing only its current values. The current state is derived by replaying the events in order; snapshots can make that faster. In microservices, this gives a complete audit trail, lets you rebuild state at any point in time, and fits naturally with publishing domain events to other services and building CQRS read models from them. The costs are real: event schemas must evolve carefully, queries need projections, and the design is harder to learn and to change. Use event sourcing for domains where history matters, not as a default for every service.
Read full answerHide full answer
Applies toevent-driven microservices (example in Python 3.12)
Code Example
from dataclasses import dataclass, field
@dataclass(frozen=True)
class Event:
stream: str # one stream per aggregate, e.g. "account-42"
version: int # position in the stream; used for optimistic concurrency
type: str
data: dict
class ConcurrencyError(Exception):
pass
class EventStore:
"""Append-only log. Real systems: EventStoreDB, Kafka plus a database, or PostgreSQL tables."""
def __init__(self) -> None:
self.events: list[Event] = []
def load(self, stream: str) -> list[Event]:
return [e for e in self.events if e.stream == stream]
def append(self, stream: str, expected_version: int, new: list[tuple[str, dict]]) -> None:
current = len(self.load(stream))
if current != expected_version: # someone else wrote first
raise ConcurrencyError(f"expected v{expected_version}, stream is at v{current}")
for i, (type_, data) in enumerate(new, start=1):
self.events.append(Event(stream, current + i, type_, data))
@dataclass
class Account:
"""State is never stored directly; it is rebuilt by applying events in order."""
balance: int = 0
version: int = 0
history: list[str] = field(default_factory=list)
def apply(self, event: Event) -> None:
if event.type == "Deposited":
self.balance += event.data["amount"]
elif event.type == "Withdrawn":
self.balance -= event.data["amount"]
self.version = event.version
self.history.append(f"{event.type} {event.data['amount']}")
@classmethod
def rehydrate(cls, events: list[Event]) -> "Account":
account = cls()
for event in events:
account.apply(event)
return account
def withdraw(store: EventStore, stream: str, amount: int) -> None:
"""Command handler: load, check the business rule, append new events."""
account = Account.rehydrate(store.load(stream))
if amount > account.balance:
raise ValueError("insufficient funds") # rule checked against current state
store.append(stream, account.version, [("Withdrawn", {"amount": amount})])
store = EventStore()
store.append("account-42", 0, [("Deposited", {"amount": 100}), ("Deposited", {"amount": 50})])
withdraw(store, "account-42", 30)
account = Account.rehydrate(store.load("account-42"))
print(account.balance, account.version) # 120 3
print(account.history) # ['Deposited 100', 'Deposited 50', 'Withdrawn 30']
past = Account.rehydrate(store.load("account-42")[:2])
print(past.balance) # 150 (state at version 2: time travel for audits)
try:
store.append("account-42", 2, [("Withdrawn", {"amount": 10})]) # stale writer
except ConcurrencyError as error:
print(error) # expected v2, stream is at v3
The account’s balance is never stored; it is rebuilt from three events. Version numbers give optimistic concurrency: the stale writer expected version 2, but the stream had moved to version 3, so the append is rejected rather than silently overwriting.
Key Concepts
Concept
Meaning
Event
An immutable fact in the past tense (OrderPlaced), with data and metadata
Stream
All events for one aggregate (for example order-123)
Event store
Append-only storage with ordering per stream (EventStoreDB/KurrentDB, a PostgreSQL table, or Kafka with care)
Rehydration
Rebuilding state by applying the events
Snapshot
Saved state at version N, so replay starts from there
Projection
A read model built by consuming events (for example “orders per customer”)
Optimistic concurrency
Append only if the stream is at the expected version
Benefits and Costs
Benefits
Costs
Complete, trustworthy audit history
More complex than CRUD; a learning curve for the team
Rebuild or correct read models by replaying events
Queries need projections; no simple SELECT of current state
Time travel: state at any point in the past
Event schema evolution (upcasting old events)
Natural source of integration events
Eventual consistency between writes and read models
Debugging by replaying real sequences
Deleting personal data is hard in an immutable log (crypto-shredding helps)
Event Sourcing Versus Event-Driven Communication
Event sourcing
Event-driven messaging
Purpose
Persistence model inside a service
Communication between services
Events
Internal domain events, complete history
Integration events designed for consumers
Required together?
No: a service can publish events without event sourcing, and vice versa
Publish integration events designed for consumers rather than exposing your internal event stream directly, so you can change internal events freely.
When to Use It
Good fit
Poor fit
Finance, ledgers, payments, inventory, bookings
Simple CRUD data (user preferences, content)
Audit or regulatory needs
Teams new to distributed systems under deadline
Complex workflows where “how did we get here?” matters
Data where only the current value matters
Pitfalls
Using Kafka as the only event store without per-entity ordering and optimistic concurrency; it is a log, not an aggregate store.
Events that are really commands or CRUD diffs (CustomerUpdated with every field) lose business meaning.
No snapshot strategy for long-lived streams.
Breaking event schema changes: version events, and keep old events readable.
Interview Tips
Define it: state = fold over an append-only event log.
Mention projections and CQRS, snapshots and optimistic concurrency.
Show judgement: say when not to use it.
29
What is CQRS, and how can it be applied to microservices?
KnownRevise
CQRS (Command Query Responsibility Segregation) separates the write side of a system, where commands change state and enforce business rules, from the read side, where queries read from models optimised for reading. In microservices, the write side is usually a service’s own normalised database. The read side is one or more projections, kept up to date by events, that can combine data from several services into exactly the shape a screen or API needs (for example, “order history with product names and shipping status”). This solves the main query problem of database-per-service, and lets reads and writes scale independently. The trade-offs are eventual consistency between the two sides and more moving parts.
Read full answerHide full answer
Applies tomicroservices data architecture (example in Python 3.12)
Code Example
from collections import defaultdict
events: list[dict] = [] # stands in for a broker topic or an event store
read_model: dict[int, dict] = defaultdict(lambda: {"orders": 0, "spent_cents": 0, "last_order": None})
def place_order(order_id: str, customer_id: int, total_cents: int) -> None:
"""Command side: validate, change the write model, publish an event."""
if total_cents <= 0:
raise ValueError("total must be positive")
events.append({"type": "OrderPlaced", "order_id": order_id,
"customer_id": customer_id, "total_cents": total_cents})
def cancel_order(order_id: str) -> None:
placed = next(e for e in events if e["type"] == "OrderPlaced" and e["order_id"] == order_id)
events.append({"type": "OrderCancelled", "order_id": order_id,
"customer_id": placed["customer_id"], "total_cents": placed["total_cents"]})
def project(from_position: int = 0) -> int:
"""Query side: a projection folds events into a denormalised view built for reads."""
for event in events[from_position:]:
view = read_model[event["customer_id"]]
if event["type"] == "OrderPlaced":
view["orders"] += 1
view["spent_cents"] += event["total_cents"]
view["last_order"] = event["order_id"]
elif event["type"] == "OrderCancelled":
view["orders"] -= 1
view["spent_cents"] -= event["total_cents"]
return len(events) # checkpoint: where to resume next time
place_order("o-1", 42, 2500)
place_order("o-2", 42, 1200)
place_order("o-3", 7, 900)
checkpoint = project()
print(dict(read_model[42])) # {'orders': 2, 'spent_cents': 3700, 'last_order': 'o-2'}
cancel_order("o-1")
print(dict(read_model[42])["orders"]) # 2 (the read model lags until the projection runs)
checkpoint = project(checkpoint)
print(dict(read_model[42])) # {'orders': 1, 'spent_cents': 1200, 'last_order': 'o-2'}
Commands only append events. The projection builds a denormalised view per customer and tracks a checkpoint, so it can resume or be rebuilt. The middle print shows the read model lagging until the projection runs, which is the eventual consistency you must design for.
Architecture in Microservices
Commands --> order-service (write model, PostgreSQL) --outbox--> OrderPlaced, OrderShipped
|
products-service --> ProductRenamed --------------------------------+
v
order-history projection (read DB: Elasticsearch,
MongoDB or a denormalised SQL table)
Queries --> order-history API --> read DB
Write Side Versus Read Side
Aspect
Write side (commands)
Read side (queries)
Model
Normalised, aggregates enforce invariants
Denormalised, shaped per query or screen
Storage
Relational database or event store
Search engine, document store, cache, wide tables
Scaling
Usually fewer instances
Scale out with replicas or caches
Consistency
Strong, inside one service
Eventual, usually seconds behind
Changes
Validated commands
Rebuilt from events when the view changes
When CQRS Helps
Good fit
Not needed
Queries that combine data from several services
Simple CRUD screens over one service’s data
Read and write loads that differ greatly
Low traffic
Complex search, filtering or reporting
Strict read-after-write needs everywhere
Event-sourced services (projections are required)
Teams without event infrastructure
CQRS does not require event sourcing, and event sourcing does not require separate services; they are often used together but are independent choices.
Handling Eventual Consistency
Show the user their own write: return the new state from the command, or update the UI optimistically.
Expose freshness: include the projection’s last processed position or timestamp.
Idempotent, ordered projections: process events per entity in order (partition by ID) and tolerate redelivery.
Rebuilds: replay events into a new read model, then switch reads over.
Pitfalls
Applying CQRS everywhere, adding complexity to simple services.
Projections that call other services synchronously while processing; they should rely on events.
No monitoring of projection lag, so stale data goes unnoticed.
Ignoring schema evolution in events consumed by projections.
Interview Tips
Define it: separate models for writes (commands) and reads (queries).
Explain how it solves cross-service queries under database per service.
Address eventual consistency and projection rebuilds, and say when not to use it.
30
What strategies handle data consistency without using distributed transactions?
KnownRevise
Without distributed transactions, consistency across microservices is achieved by designing for eventual consistency with techniques that make every step reliable, retryable and reversible: - sagas with compensating actions for multi-step workflows; - the transactional outbox or change-data-capture, so a state change and its event are never out of sync; - idempotent consumers and idempotency keys, so retries and duplicates are safe; - reservations (the try-confirm/cancel pattern) that hold resources temporarily instead of locking across services; - versioning and optimistic concurrency to reject stale updates; - per-entity ordering of events; - reconciliation jobs that detect and repair drift.
Above all, draw boundaries so that each strong invariant lives inside one service.
Read full answerHide full answer
Applies todistributed system design (example in Python 3.12)
Code Example: Reservation With Expiry (Try-Confirm/Cancel)
import uuid
class Inventory:
"""Reservation (try/confirm/cancel): hold stock with a deadline instead of locking across services."""
def __init__(self, stock: int) -> None:
self.available = stock
self.holds: dict[str, tuple[int, float]] = {} # reservation id -> (quantity, expires_at)
self.confirmed: set[str] = set()
def try_reserve(self, qty: int, now: float, ttl: float = 900) -> str | None:
self.expire(now)
if qty > self.available:
return None
hold_id = str(uuid.uuid4())
self.available -= qty
self.holds[hold_id] = (qty, now + ttl)
return hold_id
def confirm(self, hold_id: str) -> bool:
"""Idempotent: confirming twice has no further effect."""
if hold_id in self.confirmed:
return True
if hold_id not in self.holds:
return False # expired or cancelled
del self.holds[hold_id]
self.confirmed.add(hold_id)
return True
def cancel(self, hold_id: str) -> None:
"""Idempotent: cancelling an unknown or already released hold does nothing."""
qty, _ = self.holds.pop(hold_id, (0, 0))
self.available += qty
def expire(self, now: float) -> None:
for hold_id, (qty, expires_at) in list(self.holds.items()):
if expires_at <= now:
self.cancel(hold_id) # abandoned checkouts release stock
inv = Inventory(stock=3)
a = inv.try_reserve(2, now=0)
b = inv.try_reserve(2, now=10)
print(a is not None, b, inv.available) # True None 1
print(inv.confirm(a), inv.confirm(a)) # True True (retry-safe)
c = inv.try_reserve(1, now=20)
print(inv.available) # 0
inv.expire(now=20 + 901) # payment never arrived for c
print(inv.available, inv.confirm(c)) # 1 False
The order service tries to reserve stock, charges the payment, then confirms the reservation, or cancels it if payment fails. If the order service crashes and never answers, the expiry releases the stock automatically. Confirm and cancel are idempotent, so retries are safe.
Strategies
Strategy
Problem it solves
Example
Design boundaries around invariants
Some rules need ACID
Keep the account balance and its ledger in one service
Kafka partition key = order ID; reject older versions
Semantic locks
Others acting on intermediate state
Order status PENDING_PAYMENT blocks shipping
Commutative operations
Order-independent updates
Increments instead of “set to”; CRDT counters
Reconciliation
Bugs, lost messages, third-party mismatches
Nightly comparison of payments with the provider’s report
Read-your-writes techniques
User confusion from lag
Return the new state from the command; read from the owner
Why Not Distributed Transactions?
Two-phase commit problem
Effect
Blocking: participants hold locks waiting for the coordinator
Lower throughput; stuck locks after failures
Availability multiplies
Every participant must be up to commit
Limited support
Many message brokers, NoSQL stores and SaaS APIs do not support XA
Tight coupling
Services must share a transaction manager
Choosing a Strategy
Can one service own the whole invariant? If so, use a local ACID transaction.
Is it a workflow across services? Use a saga; put irreversible steps last.
Does a resource need to be held briefly? Use a reservation with expiry.
Does a service publish events after writing? Use the outbox.
Can messages repeat or arrive out of order? They can: make consumers idempotent and version-aware.
What if something still goes wrong? Add reconciliation and alerts.
Pitfalls
Dual writes (database, then broker) without an outbox.
Compensations that are not idempotent, which double-refund on retry.
Reservations without expiry, leaking stock when callers crash.
Assuming exactly-once delivery from a broker.
Interview Tips
Say clearly: “design for eventual consistency; keep strong invariants inside one service”.
Give the toolkit: sagas, outbox, idempotency, reservations, versioning, reconciliation.
Explain why 2PC is avoided in microservices.
31
How do you implement authentication and authorization in microservices?
KnownRevise
In microservices, authentication (who is calling) is usually centralised in an identity provider using OAuth 2.0 and OpenID Connect (Keycloak, Auth0, Okta, Microsoft Entra ID, AWS Cognito). Users log in there, and clients receive short-lived access tokens, typically JWTs signed with asymmetric keys. The API gateway validates tokens at the edge, and each service validates them again using the provider’s public keys (JWKS), so no service trusts the network blindly. Authorization (what the caller may do) happens at two levels: coarse checks (scopes, roles) at the gateway or mesh, and fine-grained, business-level decisions inside the service that owns the data, sometimes through a policy engine such as OPA or Cedar. Service-to-service calls use workload identity: mTLS (often from a service mesh) and/or OAuth2 client-credentials tokens.
Read full answerHide full answer
Applies toOAuth 2.0 / OpenID Connect, JWT, Kubernetes and service meshes
Typical Flow
1. User logs in at the identity provider (Authorization Code flow with PKCE)
2. Client receives an access token (JWT, about 5-15 minutes) and a refresh token
3. Client --Bearer token--> API gateway: check signature, expiry, issuer, audience
4. Gateway --> orders-service (passes the token): the service validates it again
and authorises: "may user 42 cancel order 981?" (does it belong to them?)
5. orders-service --> payment-service: mTLS identity + its own client-credentials token
(or a token exchange that keeps the user context)
Example: Mesh-Level Authorization for Service Calls
Asymmetric signatures (RS256, ES256), keys from JWKS with rotation
Services verify without sharing a secret
Validate iss, aud, exp, nbf and the algorithm
Prevents token reuse and algorithm-confusion attacks
Short-lived access tokens, refresh tokens kept by the client securely
Limits damage from a leaked token
Minimal claims (subject, scopes, tenant)
Tokens are readable by anyone who holds them; do not put secrets in them
Token exchange (RFC 8693) for downstream calls
Keeps user context with the right audience for each service
Opaque tokens with introspection where immediate revocation matters
Trade a network call for instant revocation
Authorization Models
Model
Example
RBAC (roles)
admin, support, customer
ABAC (attributes)
Allow if user.tenant == order.tenant and the order is not shipped
ReBAC (relationships)
“Editors of this folder can edit its documents” (Zanzibar-style systems such as OpenFGA, SpiceDB)
Pitfalls
Checking tokens only at the gateway and trusting all internal traffic.
Long-lived tokens with no way to revoke them.
Putting business authorization in the gateway: it does not know who owns order 981.
Accepting alg: none or the wrong algorithm in JWT libraries.
Logging tokens.
Interview Tips
Separate authentication (identity provider, OIDC, JWT) from authorization (coarse at the edge, fine-grained in services).
Cover service-to-service identity with mTLS and client credentials.
Mention JWKS validation, short lifetimes and token exchange.
32
What are some common security concerns when handling inter-service communication?
KnownRevise
Inter-service traffic is a large attack surface, because microservices turn function calls into network calls. The main concerns are: - eavesdropping and tampering with unencrypted traffic; - spoofed callers, when services trust anything on the internal network; - excessive permissions, where any service can call any endpoint; - stolen or misused credentials and tokens; - lateral movement after one service is compromised; - injection and deserialization attacks through messages; - server-side request forgery (SSRF); - leaking sensitive data in logs and traces; - denial of service from retry storms or abusive callers; - supply-chain risks in the images and libraries every service uses.
The response is a zero-trust approach: authenticate and encrypt every connection (mTLS), authorize every call with least privilege, segment the network, validate all inputs, manage secrets centrally, and monitor and audit traffic.
Read full answerHide full answer
Applies tomicroservices security (Kubernetes and service-mesh examples)
Threats and Controls
Threat
Example
Controls
Eavesdropping / tampering
Sniffing tokens on a shared network
TLS everywhere, mTLS between services (mesh or libraries)
Spoofing
A compromised pod pretends to be orders-service
Workload identity (SPIFFE IDs in mTLS certificates), signed tokens with audience checks
Only pods labelled app: orders in the orders namespace can reach the payment pods on port 8443. Network policies need a CNI plugin that enforces them, such as Calico or Cilium. Combine them with mTLS authorization, because network rules alone do not prove which workload is calling.
Zero-Trust Principles Applied
Never trust the network: internal traffic is authenticated and encrypted like external traffic.
Least privilege: each service can reach only the services, endpoints and topics it needs.
Verify explicitly on every request: identity, token, audience and permission.
Assume breach: limit the blast radius with segmentation, and detect anomalies.
Monitoring and Response
Audit logs of who called what, without logging secrets.
Alerts on unusual call patterns (a service calling an endpoint it never used before).
Mesh telemetry to see unexpected connections.
Regular penetration tests and dependency updates.
Interview Tips
Use zero trust as the organising idea: mTLS + per-call authorization + least privilege.
Mention NetworkPolicies, secret management, and input validation for messages.
Include less obvious risks: SSRF, log leakage and supply-chain issues.
33
Describe how you would use OAuth2 or JWT in a microservices architecture.
KnownRevise
Use OAuth 2.0 (with OpenID Connect for login) as the protocol and JWTs as the token format. A central authorization server (Keycloak, Auth0, Okta, Entra ID, Cognito) authenticates users and issues short-lived access tokens as JWTs, signed with its private key. Clients send them as Authorization: Bearer <token>. The API gateway and every service verify each token locally, using the server’s public keys (JWKS): they check the signature, issuer, audience, expiry and scopes, then apply their own business authorization. Services calling other services use the client-credentials grant, or token exchange to keep the user’s identity, usually together with mTLS. This gives stateless, scalable verification without a central session store.
Read full answerHide full answer
Applies toOAuth 2.0 / OpenID Connect and JWT (RFC 7519); example in Python 3.12 using only the standard library
Which Grant for Which Caller
Caller
Grant
Notes
Web or mobile app with a user
Authorization Code with PKCE
Never the implicit grant (deprecated in OAuth 2.0 security best practice)
Service with no user (jobs, internal calls)
Client credentials
The token represents the service itself
Service calling downstream on behalf of a user
Token exchange (RFC 8693) or forwarding with a narrowed audience
Keeps the user context; each token targets one audience
Devices without a browser
Device authorization grant
Smart TVs, CLIs
What Each Service Must Verify
The example uses an HMAC (HS256) key so it runs with the standard library alone. Real services verify asymmetric signatures (RS256 or ES256) with public keys fetched from the provider’s JWKS endpoint, using a maintained library (PyJWT, jose, Spring Security, jsonwebtoken).
import base64
import hashlib
import hmac
import json
def b64url(data: bytes) -> str:
return base64.urlsafe_b64encode(data).rstrip(b"=").decode()
def b64url_decode(text: str) -> bytes:
return base64.urlsafe_b64decode(text + "=" * (-len(text) % 4))
def sign(claims: dict, key: bytes) -> str:
"""Teaching only: HS256 with a shared key. Production services verify RS256/ES256 via JWKS."""
header = b64url(json.dumps({"alg": "HS256", "typ": "JWT"}).encode())
body = b64url(json.dumps(claims).encode())
sig = hmac.new(key, f"{header}.{body}".encode(), hashlib.sha256).digest()
return f"{header}.{body}.{b64url(sig)}"
def verify(token: str, key: bytes, audience: str, issuer: str, now: int) -> dict:
"""The checks every service must make before trusting a token."""
header_b64, body_b64, sig_b64 = token.split(".")
header = json.loads(b64url_decode(header_b64))
if header.get("alg") != "HS256": # never accept "none" or a switched algorithm
raise PermissionError("unexpected algorithm")
expected = hmac.new(key, f"{header_b64}.{body_b64}".encode(), hashlib.sha256).digest()
if not hmac.compare_digest(expected, b64url_decode(sig_b64)):
raise PermissionError("bad signature")
claims = json.loads(b64url_decode(body_b64))
if claims["iss"] != issuer:
raise PermissionError("wrong issuer")
if audience not in ([claims["aud"]] if isinstance(claims["aud"], str) else claims["aud"]):
raise PermissionError("token not meant for this service")
if now >= claims["exp"]:
raise PermissionError("expired")
return claims
KEY = b"demo-key-from-a-secret-manager"
token = sign({"iss": "https://id.example.com", "sub": "user-42", "aud": "orders-api",
"scope": "orders:read orders:write", "exp": 1_800_000_900}, KEY)
claims = verify(token, KEY, "orders-api", "https://id.example.com", now=1_800_000_000)
print(claims["sub"], "orders:write" in claims["scope"].split()) # user-42 True
for audience, now in [("payments-api", 1_800_000_000), ("orders-api", 1_800_001_000)]:
try:
verify(token, KEY, audience, "https://id.example.com", now)
except PermissionError as error:
print(error)
# token not meant for this service
# expired
tampered = token[:-2] + ("AA" if token[-2:] != "AA" else "BB")
try:
verify(tampered, KEY, "orders-api", "https://id.example.com", now=1_800_000_000)
except PermissionError as error:
print(error) # bad signature
Check
Prevents
Algorithm is the expected one
alg: none and algorithm-switching attacks
Signature valid against the issuer’s key
Forged or modified tokens
iss (issuer)
Tokens from another identity provider
aud (audience)
A token for one service being replayed against another
JWT payloads are encoded, not encrypted: anyone holding the token can read them. Keep claims minimal and never include secrets or sensitive personal data.
Architecture Decisions
Decision
Recommendation
Token lifetime
Access tokens around 5–15 minutes; refresh tokens rotated and stored securely by the client
Revocation
Short lifetimes, plus introspection or a deny-list for high-risk actions
Where to validate
At the gateway and in every service (defence in depth)
Browser apps
Prefer a backend-for-frontend that keeps tokens server-side and gives the browser an HttpOnly cookie
Key rotation
Publish new keys in JWKS before using them; services cache JWKS and refresh on unknown kid
Fine-grained permissions
Decide inside the owning service, not from token claims alone
Pitfalls
Treating a JWT as a session that cannot be revoked, with long lifetimes.
Sharing one HMAC secret across many services, so any of them can mint tokens.
Skipping audience checks, which lets tokens be replayed across services.
Storing tokens in localStorage, which exposes them to XSS.
Writing your own JWT parser in production.
Interview Tips
Separate the roles: OAuth2/OIDC is the protocol, JWT is the token format.
List the verification checks and JWKS-based key rotation.
SCA and image scanning (Trivy, Grype, Dependabot), minimal or distroless base images, SBOMs, image signing (Sigstore cosign), and admission policies that admit only signed images
Build pipeline
Protected branches, reviewed changes, isolated runners, OIDC to the cloud instead of stored keys, provenance (SLSA levels)
Runtime
Non-root users, readOnlyRootFilesystem, dropped Linux capabilities, seccomp profiles, Pod Security Standards (restricted), resource limits
Network
mTLS everywhere, default-deny NetworkPolicies, egress restrictions, WAF and rate limiting at the edge
Identity
OIDC for users, workload identity for services, least-privilege IAM and RBAC, MFA for humans
The image is pinned by digest, so a modified tag cannot change what runs. The service account token is not mounted unless the service needs the Kubernetes API.
Detection
Signal
Tool or approach
Suspicious process or file activity in containers (a shell spawned in a pod, writes to /etc)
Falco, Cilium Tetragon, cloud runtime security
Unusual network connections
Mesh telemetry, flow logs, network detection
Authentication anomalies (token reuse, impossible travel, many failures)
Identity provider logs, SIEM rules
Kubernetes control-plane abuse
Kubernetes audit logs, alerts on new ClusterRoleBindings or exec into pods
Data exfiltration
Egress monitoring, unusual volumes from databases or storage
Known vulnerabilities in running images
Continuous scanning of deployed images
Send all of this to a SIEM (Splunk, Elastic Security, Microsoft Sentinel, Google SecOps), with alert rules and on-call ownership.
Response
Contain: scale down or isolate the service with network policies; revoke its tokens and credentials.
Eradicate: redeploy from known-good, signed images; patch the vulnerability.
Recover: restore data from backups if needed; verify integrity.
Learn: run a blameless postmortem; add detections for the attack path.
Pitfalls
Perimeter-only security, trusting everything inside the cluster.
Logs that contain secrets, or that are deletable by the attacker.
Alerts nobody owns.
Unpatched base images shared by dozens of services.
Interview Tips
Organise the answer as prevent → detect → respond.
Mention supply-chain security, runtime hardening and zero-trust networking for prevention.
For detection, name runtime tools (Falco), audit logs and a SIEM.
35
How do you ensure that sensitive data is protected when using microservices?
KnownRevise
Protect sensitive data (personal data, credentials, payment and health data) by limiting where it exists and protecting it everywhere it goes: - Classify data, then minimise it: collect and store only what is needed, in as few services as possible. - Isolate sensitive data in dedicated services with strict access. - Encrypt in transit (TLS and mTLS) and at rest (database and disk encryption with keys in a KMS), and use field-level encryption or tokenization for the most sensitive values. - Enforce least-privilege access with audited, short-lived credentials. - Keep sensitive data out of logs, traces, events and caches. - Meet regulatory needs (GDPR, HIPAA, PCI DSS) with retention rules, deletion workflows and audit trails.
Read full answerHide full answer
Applies todata protection in distributed systems (vendor-neutral)
Controls by Stage
Stage
Controls
Design
Data classification (public, internal, confidential, restricted); data-flow diagrams; privacy by design
Collection
Data minimisation; consent tracking where required
Storage
Encryption at rest with KMS-managed keys (AWS KMS, Google Cloud KMS, Azure Key Vault, Vault Transit); per-tenant or per-service keys
Highly sensitive fields
Tokenization (replace a card number with a token; only a vault service maps it back), or field-level encryption in the application
In transit
TLS 1.2+ at the edge, mTLS between services, encrypted broker connections
Access
Least-privilege IAM, separate database users per service, just-in-time human access, audit logs
Processing
Masking in non-production environments, synthetic test data, no production data copies on laptops
Sharing
Pass references (IDs, tokens) in events instead of raw data
Observability
Log redaction, allow-listed log fields, scrubbing in tracing pipelines
Retention and deletion
TTLs, deletion events propagated to every copy, crypto-shredding (destroy the key to make data unreadable)
Reducing the Blast Radius
Before: name, email, address and card number copied into orders, shipping, analytics, email
After: customer-profile-service (PII, encrypted) payment-vault (card data, tokenized)
other services store only customer_id and card_token
With PCI DSS, isolating card data in one tokenization service can shrink the audit scope to that service. Similarly, propagating only IDs makes GDPR erasure requests manageable.
Field-Level Encryption Versus Tokenization
Field-level encryption
Tokenization
How
Encrypt the value with a key (envelope encryption: data key wrapped by a KMS key)
Replace the value with a random token; the mapping is stored in a secured vault
Who can read it
Anyone allowed to use the key
Only the vault service
Searching
Hard (needs deterministic or searchable encryption, with trade-offs)
Lookups through the vault
Typical use
PII fields in databases
Card numbers, national IDs
Events and Logs
Event payloads are copied to brokers, consumers, data lakes and replays. Put references in events, not full personal data, or encrypt the sensitive fields.
Logs and traces must not contain tokens, passwords or full PII. Use structured logging with redaction filters, and test them.
Caches (Redis) holding sensitive data need encryption in transit, authentication and expiry.
Compliance Points (General)
Regulation
Typical requirements
GDPR (EU)
Lawful basis, minimisation, access and erasure rights, breach notification, records of processing
HIPAA (US health)
Safeguards for protected health information, audit controls, access controls
PCI DSS (payment cards)
Protect stored card data, restrict access, monitor and test networks
Work with legal and compliance teams for specifics; these notes are general.
Pitfalls
Personal data spreading through events into every service and the data lake.
Production database dumps used in testing.
Encryption keys stored next to the data they protect.
Backups forgotten in deletion and retention policies.
Interview Tips
Start with classification and minimisation, then encryption in transit and at rest.
Mention tokenization and isolating sensitive data in dedicated services.
Cover logs, events and backups as common leak paths.
36
How do you ensure that a microservice is scalable?
KnownRevise
A microservice scales well when it can add instances to handle more load without code changes, and when nothing it depends on becomes the bottleneck. In practice that means: - Stateless instances: sessions and state live in databases, caches or tokens, so any instance can serve any request. - Horizontal autoscaling on meaningful signals: CPU, request rate, queue lag. - Data stores that scale with the service: read replicas, partitioning or sharding, caching. - Asynchronous processing for slow work, and bounded resources: connection pools, timeouts, back-pressure. - Load tests that show the real limits.
Scaling is end to end: doubling the pods does not help if the database or a downstream service cannot keep up.
Read full answerHide full answer
Applies tocloud-native services on Kubernetes (vendor-neutral principles)
CPU utilisation is measured against the pods’ CPU requests, so requests must be set. The scale-down rules prevent flapping. For queue consumers, scale on queue lag with KEDA instead of CPU.
Design Checklist
Area
Practice
State
No in-memory sessions; use Redis, a database or stateless tokens; idempotent handlers
Startup and shutdown
Fast startup, readiness probes, graceful SIGTERM handling so scaling events do not drop requests
Horizontal scaling
HPA on CPU or custom metrics (requests per second, latency); KEDA for event-driven scaling; Cluster Autoscaler or Karpenter for nodes
Database
Indexes, connection pooling (PgBouncer), read replicas, caching, partitioning or sharding by key, CQRS read models
Caching
Cache-aside with TTLs; CDN for static and cacheable responses
Partition data and work by tenant or key to avoid hotspots
Efficiency
Profile hot paths; avoid N+1 queries; batch calls
Finding the Limits
Step
Detail
Load test
k6, Gatling or Locust with realistic traffic mixes; ramp up until SLOs break
Find the bottleneck
CPU, memory, database connections, locks, a downstream service, network
Capacity model
Requests per second per instance at target latency, then instances needed at peak plus headroom
Repeat
After major changes and before seasonal peaks
Scaling Dimensions (Scale Cube)
Axis
Meaning
Example
X: clone
Run more identical instances behind a load balancer
HPA replicas
Y: split by function
Separate services per capability
Extract search from the monolith
Z: partition by data
Each instance or shard handles a subset of keys
Shard orders by customer ID
Pitfalls
Sticky sessions or local caches that make instances unequal.
Each new pod opening many database connections, exhausting the database at scale; use pooling and set limits.
Autoscaling on the wrong metric, such as CPU for an I/O-bound service.
No resource requests, so the scheduler and HPA cannot work properly.
Shared hot rows or keys (a global counter) that serialise all writes.
Interview Tips
Say stateless + horizontal autoscaling + scalable data layer.
Explain that the bottleneck moves: databases and downstream services must scale too.
Mention load testing, KEDA for queue-based scaling, and the scale cube.
37
What metrics would you monitor to assess a microservice’s performance?
KnownRevise
Assess a microservice’s performance with user-facing service metrics first: request rate, error rate and latency percentiles (p50, p95, p99), which together form the RED method. Add saturation and resource metrics that explain them: CPU, memory, garbage collection pauses, thread and connection-pool usage, queue depth and consumer lag. Add dependency metrics: latency and errors of databases, caches and downstream services, and retry and circuit-breaker counts. Finally, add business metrics such as orders per minute or payment success rate. Compare these against SLOs, and label them by endpoint, version and instance, so regressions can be traced to a release or a single route.
# Requests per second by endpoint
sum by (route) (rate(http_server_requests_total{service="orders"}[5m]))
# Error ratio (5xx)
sum(rate(http_server_requests_total{service="orders",status=~"5.."}[5m]))
/ sum(rate(http_server_requests_total{service="orders"}[5m]))
# p99 latency from a histogram
histogram_quantile(0.99,
sum by (le) (rate(http_server_request_duration_seconds_bucket{service="orders"}[5m])))
# CPU throttling (a common hidden cause of latency in containers)
rate(container_cpu_cfs_throttled_periods_total{container="orders"}[5m])
/ rate(container_cpu_cfs_periods_total{container="orders"}[5m])
Metric names depend on the instrumentation library. OpenTelemetry’s HTTP semantic conventions use http.server.request.duration, which Prometheus exports as http_server_request_duration_seconds.
SLIs and SLOs
SLI
Example SLO
Availability: share of successful requests
99.9% over 30 days
Latency: share of requests faster than a threshold
99% of POST /orders under 300 ms
Freshness (asynchronous)
99% of events processed within 60 s
Alert on error-budget burn rate, for example “at this rate the monthly budget is spent in 2 days”, rather than on every spike.
Reading the Metrics Together
Observation
Likely cause
Latency up, CPU flat, database time up
Slow queries or lock contention
Latency up, CPU throttled
CPU limits too low
Errors up after a deployment, only on the new version
Regression; roll back
Rising consumer lag, normal errors
Consumers too slow or too few; scale or optimise
Pool wait time up
Connection pool too small, or slow downstream calls
Pitfalls
Averages instead of percentiles.
Summaries that cannot be aggregated across instances; prefer histograms.
High-cardinality labels (user IDs, raw URLs) that overload the metrics system; use route templates.
Only technical metrics, missing broken business flows.
Interview Tips
Start with RED (rate, errors, duration) and saturation, then dependencies and business metrics.
Insist on percentiles and SLO-based alerting.
Mention per-version labels to catch bad releases.
38
Discuss strategies to handle high-load or peak traffic in a microservices architecture.
KnownRevise
Handle peak traffic by combining capacity, efficiency and protection: - Scale out ahead of known peaks (scheduled or predictive scaling), and autoscale for unknown ones. - Absorb load before it reaches services: CDNs, caching, and queues that turn spikes into steady work. - Reduce work per request: cache-aside, precomputed views, batching, cheaper degraded responses. - Protect the system when demand still exceeds capacity:rate limiting, load shedding of low-priority traffic, circuit breakers, bulkheads, and graceful degradation, such as turning off recommendations to keep checkout working.
Prove all of it with load tests and game days before the peak.
Read full answerHide full answer
Applies todistributed system design (example in Python 3.12)
Code Example: Token-Bucket Rate Limiting
class TokenBucket:
"""Allow bursts up to `capacity`, refilled at `rate` tokens per second."""
def __init__(self, rate: float, capacity: int) -> None:
self.rate, self.capacity = rate, capacity
self.tokens, self.last = float(capacity), 0.0
def allow(self, now: float, cost: float = 1.0) -> bool:
self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
self.last = now
if self.tokens >= cost:
self.tokens -= cost
return True
return False # caller answers 429 Too Many Requests
limiter = TokenBucket(rate=5, capacity=10) # 5 requests/s sustained, bursts of 10
burst = [limiter.allow(now=0.0) for _ in range(15)]
print(burst.count(True), burst.count(False)) # 10 5
print(limiter.allow(now=0.1), limiter.allow(now=0.2), limiter.allow(now=0.2)) # False True False
later = [limiter.allow(now=2.2) for _ in range(12)]
print(later.count(True)) # 10 (refilled to capacity, not beyond)
A token bucket allows short bursts up to its capacity, then limits clients to a sustained rate; excess requests get 429 Too Many Requests with a Retry-After header. In production, limits are enforced at the gateway or edge, often with a shared counter in Redis so every gateway instance applies the same limit.
Strategies
Strategy
How it helps
Examples
Pre-scaling
Capacity is ready before a sale or launch
Scheduled scaling; warm node pools; raise database capacity in advance
Autoscaling
Reacts to unexpected load
HPA, KEDA on queue lag, cluster autoscaling
Edge caching
Most reads never reach services
CDN for static assets and cacheable API responses
Application caching
Reduces database load
Redis cache-aside; request coalescing to stop cache stampedes
Queues and asynchronous work
Flattens spikes
Accept orders quickly, process payments and emails from a queue
Rate limiting and quotas
Fair share per client; stops abuse
Token bucket per API key or user at the gateway
Load shedding
Drops low-priority work first when overloaded
Reject background or bot traffic; prioritise checkout
Graceful degradation
Keeps core flows working
Hide recommendations, serve cached prices, disable expensive features with flags
Bulkheads
One overloaded feature cannot starve others
Separate pools or deployments for checkout and browsing
Load test at 1.5–2× the forecast with realistic traffic mixes.
Fix the bottlenecks found: queries, pool sizes, cache hit ratios.
Pre-scale services, databases and caches; check cloud quotas.
Prepare feature flags for degradation, and runbooks.
Freeze risky changes around the event; staff an on-call war room.
Watch the SLO dashboards live, and review afterwards.
Protecting the Weakest Dependency
Dependency
Protection
Database
Pool limits, read replicas, caching, queue writes
Third-party API (payments, SMS)
Concurrency limits, retries with backoff, fallbacks, queuing
Legacy system
Rate limit calls to it; cache its responses
Pitfalls
Autoscaling too slowly: new pods and nodes take time; scale ahead and keep headroom.
Retry storms that multiply load during the spike.
Cache stampedes when hot keys expire at once; use jittered TTLs and request coalescing.
Scaling stateless services while the database stays the same.
No priority between traffic types.
Interview Tips
Structure the answer as add capacity → reduce load → protect the system.
Name rate limiting, load shedding, queues, caching and graceful degradation.
Mention load testing and pre-scaling for known events.
39
How do Microservices handle load balancing?
KnownRevise
Microservices balance load at two levels. At the edge, external traffic reaches a cloud load balancer or ingress / API gateway that spreads requests across instances of the front services. Between services, calls are balanced either server-side, through a virtual address such as a Kubernetes Service that routes to healthy pods, or client-side, where the caller (a library or a service-mesh sidecar such as Envoy) learns the instance list through service discovery and picks an instance per request. Good balancing combines a sensible algorithm (round robin, least requests, power of two choices, consistent hashing), health checks that remove bad instances, and connection awareness, which matters for long-lived HTTP/2 and gRPC connections.
import itertools
import random
def simulate(choose, servers: int = 5, requests: int = 20_000, seed: int = 42) -> int:
"""Return the worst queue length seen; one server is 3x slower than the others."""
rng = random.Random(seed)
speed = [1.0] * (servers - 1) + [1 / 3]
queues, worst = [0] * servers, 0
for _ in range(requests):
target = choose(queues, rng)
queues[target] += 1
worst = max(worst, max(queues))
for s in range(servers): # each server finishes work at its own speed
if queues[s] and rng.random() < speed[s] * 0.23:
queues[s] -= 1
return worst
cycle = itertools.cycle(range(5))
round_robin = lambda queues, rng: next(cycle)
random_pick = lambda queues, rng: rng.randrange(len(queues))
least_loaded = lambda queues, rng: min(range(len(queues)), key=queues.__getitem__)
def two_choices(queues, rng):
a, b = rng.sample(range(len(queues)), 2) # "power of two choices"
return a if queues[a] <= queues[b] else b
for name, strategy in [("round robin", round_robin), ("random", random_pick),
("least connections", least_loaded), ("power of two choices", two_choices)]:
print(f"{name:22} worst queue = {simulate(strategy)}")
# round robin worst queue = 2486 (the slow server's queue keeps growing)
# random worst queue = 2496
# least connections worst queue = 29
# power of two choices worst queue = 62 (exact numbers depend on the seed)
With one slow server, round robin and random keep sending it the same share of work, so its queue grows without limit. Least connections and power of two choices look at current load and steer traffic away. Power of two choices needs only two samples per request, which is why proxies such as Envoy use it for their least-request algorithm.
Where Balancing Happens
Layer
Component
Notes
Global / DNS
GeoDNS, anycast, global load balancers
Choose a region close to the user
Edge (L4/L7)
Cloud load balancers (AWS ALB/NLB, Google Cloud Load Balancing), ingress controllers, Gateway API implementations
TLS termination, path-based routing
Service-to-service, server-side
Kubernetes Service (kube-proxy iptables/IPVS, or eBPF with Cilium)
Balances requests, with outlier detection and retries
Algorithms
Algorithm
How
Good for
Round robin
Next instance in turn
Similar instances and requests
Weighted round robin
Proportional to weights
Mixed instance sizes, canary splits
Least connections / least requests
Fewest active requests
Uneven request costs
Power of two choices
Sample two, pick the less loaded
Large clusters; near least-loaded quality at low cost
Consistent hashing / ring hash / Maglev
Same key → same instance
Caches, sticky sessions, sharded state
Locality-aware
Prefer the same zone, fail over to others
Lower latency and cross-zone cost
Health and Resilience
Readiness probes remove pods that cannot serve from Kubernetes endpoints.
Active health checks (the balancer probes) and passive outlier detection (eject hosts with many 5xx errors) in Envoy and Istio.
Retries to a different instance, with budgets, for idempotent requests.
Connection draining during deployments, so in-flight requests finish.
The gRPC and HTTP/2 Pitfall
HTTP/2 and gRPC keep one long-lived connection with many requests multiplexed over it. A Kubernetes Service balances connections, so all requests from one client may go to one pod, while new pods get nothing. Fixes: - Client-side balancing in gRPC, with a headless Service and the round_robin policy. - A service mesh, which balances per request. - Connection max-age limits, so clients reconnect periodically.
Pitfalls
Session affinity that pins heavy users to one instance; prefer stateless services.
Ignoring zones, which adds latency and cross-zone traffic cost.
Health checks that are too shallow (a process that is up but cannot reach its database) or too deep (that fail when a shared dependency is down).
Uneven pod sizes, without weights.
Interview Tips
Distinguish edge versus internal, and server-side versus client-side balancing.
Name the algorithms, especially least requests and power of two choices.
Mention the gRPC long-lived connection problem and outlier detection.
40
How does performance influence choosing a message broker over direct service calls?
KnownRevise
Direct service calls (HTTP or gRPC) give the lowest latency for a single request-response: one network hop, typically about a millisecond or less inside a data centre. But the caller waits, and its throughput is limited by the slowest dependency. A message broker (Kafka, RabbitMQ, NATS, SQS) adds a hop and some latency, from a few milliseconds upward, depending on persistence and batching. In return it gives higher, smoother throughput: producers do not wait for consumers, spikes are buffered, consumers scale independently and batch their work, and a slow consumer does not slow the producer. So choose direct calls when the caller needs a fast answer now, and a broker when throughput, spike absorption and decoupling matter more than immediate results.
Read full answerHide full answer
Applies todistributed system design (Kafka, RabbitMQ, NATS, cloud queues; HTTP/gRPC)
Performance Comparison
Factor
Direct call (REST/gRPC)
Message broker
Latency for one interaction
Lowest: one hop
Higher: producer → broker → consumer, plus persistence
Caller blocking
Waits for the response
Returns after the broker acknowledges
Throughput under spikes
Limited by the callee’s current capacity; overload causes errors
Broker buffers; consumers drain at their own pace
Batching
Usually per request
Natural: producers and consumers batch messages
Fan-out to many consumers
One call per consumer
One publish; the broker delivers to each subscriber group
Tail latency
Grows with chain length and slow dependencies
Producer latency stays stable; end-to-end latency includes queue time
Back-pressure
Errors or timeouts
Queue depth and consumer lag
Resource use
Threads or connections held while waiting
Producers free immediately
Typical Numbers (Orders of Magnitude)
Path
Typical latency
gRPC call within a zone
Sub-millisecond to a few milliseconds
Kafka publish with acks=all and replication
A few milliseconds or more, depending on batching settings (linger.ms) and disks
RabbitMQ with persistent messages and publisher confirms
A few milliseconds
Cloud queues (SQS, Pub/Sub)
Tens of milliseconds
These vary widely with configuration and hardware, so measure in your own environment.
Choosing by Workload
Workload
Better choice
Reason
User waiting for data (product details, auth check)
Acknowledgement modes (acks=1 vs acks=all) trading latency for durability
Pitfalls
Using a broker for synchronous request/reply with blocking waits: you pay the broker latency and keep the coupling.
Long synchronous chains in the name of speed, where latency and failures add up.
Ignoring consumer lag: low producer latency can hide minutes of delay before work is done.
Over-tuning for throughput (large batches) on latency-sensitive paths.
Interview Tips
Frame the choice as latency of one interaction versus throughput and resilience under load.
Mention buffering, batching, fan-out and independent scaling as the broker’s performance strengths.
Advise measuring end-to-end latency, including queue time.
41
What are the advantages and drawbacks of using REST over gRPC in microservice communication?
KnownRevise
REST (resources over HTTP, usually JSON) is universal and simple. Every language, browser, proxy, cache and debugging tool understands it, and payloads are human-readable, which makes it the default for public APIs and simple internal calls. gRPC (Protocol Buffers over HTTP/2) is faster and stricter: it has compact binary messages, multiplexed connections, generated typed clients from .proto contracts, deadlines, and streaming in both directions. That suits high-volume internal service-to-service traffic. Choosing REST over gRPC gains reach, simplicity and HTTP caching, and gives up performance, strict contracts and streaming. Many systems use REST or GraphQL at the edge and gRPC inside.
Read full answerHide full answer
Applies toHTTP/REST with JSON (OpenAPI 3.1) and gRPC with Protocol Buffers (proto3)
Comparison
Aspect
REST (JSON over HTTP/1.1 or HTTP/2)
gRPC (Protobuf over HTTP/2)
Payload
Text JSON: readable, larger
Binary Protobuf: compact, faster to parse
Contract
Optional (OpenAPI); easy to drift
Required .proto; generated code in many languages
Typing
Validation at runtime
Compile-time types in generated stubs
Streaming
Server-sent events, WebSockets as add-ons
Built in: server, client and bidirectional streaming
Browser support
Native
Needs gRPC-Web or Connect, plus a proxy or adapter
Caching
HTTP caching (Cache-Control, ETags, CDNs)
No standard HTTP caching
Tooling and debugging
curl, browsers, Postman; easy
grpcurl, Postman, Buf; binary payloads need tools
Error model
HTTP status codes plus a body (RFC 9457 problem details)
gRPC status codes with details
Timeouts
Client-specific
Deadlines propagated in metadata
Load balancing
Easy (per request at L7)
Long-lived HTTP/2 connections need L7 or client-side balancing
Evolution
Add JSON fields; versioned URLs or media types
Add fields with new numbers; never reuse field numbers
Example Contract in gRPC
syntax = "proto3";
package orders.v1;
service OrderService {
rpc GetOrder(GetOrderRequest) returns (Order);
rpc WatchOrders(WatchOrdersRequest) returns (stream OrderEvent); // server streaming
}
message GetOrderRequest {
string order_id = 1;
}
message Order {
string id = 1;
string customer_id = 2;
int64 total_cents = 3;
string status = 4;
reserved 5; // a removed field number is never reused
}
message WatchOrdersRequest {
string customer_id = 1;
}
message OrderEvent {
string order_id = 1;
string status = 2;
}
The REST equivalent would be GET /v1/orders/{id} returning JSON, described in an OpenAPI document.
Advantages of REST
Works everywhere: browsers, mobile, partners, low-code tools.
HTTP caching and CDNs for read-heavy resources.
Easy to debug and explore; no code generation needed.
A familiar design style for teams and API consumers.
Drawbacks of REST (Compared With gRPC)
Larger payloads and slower serialisation, which matters at high request volumes.
Weak contracts unless OpenAPI is enforced, so clients and servers drift.
No native streaming or deadline propagation.
Over- and under-fetching, often leading to many calls or custom endpoints.
When to Choose Which
Situation
Choice
Public or partner API, browsers
REST (or GraphQL)
Internal, high-throughput, low-latency calls between services
gRPC
Streaming updates between services
gRPC streaming (or a broker)
Polyglot teams wanting generated, typed clients
gRPC
Cacheable resources (catalog, content)
REST with HTTP caching
Simple CRUD service with few consumers
REST
Pitfalls
gRPC behind connection-level load balancers: all requests stick to one pod. Use L7 proxies, a mesh or client-side balancing.
Breaking Protobuf compatibility by renumbering or reusing fields.
REST APIs without versioning or a schema, so consumers break silently.
Choosing gRPC for a public browser API without gRPC-Web or Connect.
Interview Tips
Summarise: REST for reach and simplicity, gRPC for performance, contracts and streaming.
Mention HTTP/2 multiplexing, Protobuf, deadlines and code generation for gRPC, and HTTP caching and universality for REST.
Note the common hybrid: REST at the edge, gRPC inside.
42
How would you implement versioning in microservices API?
KnownRevise
API versioning lets a service change its contract without breaking existing consumers, which matters in microservices because consumers upgrade on their own schedules. The best strategy is to avoid breaking changes: make additive, backward-compatible changes, such as new optional fields or endpoints, and have consumers ignore unknown fields. Introduce a new major version only for breaking changes, run the versions side by side for a deprecation period, announce it with Deprecation and Sunset headers, and track which consumers still call the old version before removing it. For REST, URL versioning (/v1/, /v2/) is the most common and visible scheme; for gRPC, the version goes in the package name (orders.v1); for events, use a schema registry with compatibility rules.
Read full answerHide full answer
Applies toREST (OpenAPI), gRPC/Protobuf and event schemas
Versioning Schemes for REST
Scheme
Example
Pros
Cons
URL path
GET /v2/orders/42
Visible, easy to route and cache
URLs change between versions
Header
Api-Version: 2026-10-01
Clean URLs; date-based versions are popular in public APIs
Less visible; harder to test in a browser
Media type
Accept: application/vnd.example.order.v2+json
Fine-grained, RESTful
Complex for clients and tooling
Query parameter
/orders/42?version=2
Simple
Easy to misuse; messy with caching
Choose one scheme and use it consistently across services. Only the major version normally appears in the contract.
Compatible Versus Breaking Changes
Usually compatible (no new version needed)
Breaking (new major version)
Adding an optional request field
Removing or renaming a field
Adding a response field
Changing a field’s type or meaning
Adding a new endpoint or operation
Making an optional field required
Adding a new enum value (if clients tolerate unknown values)
Changing URL structure or status codes
Relaxing validation
Tightening validation
Clients must follow the tolerant reader rule: ignore fields they do not know.
Prefer one service with two API layers over two copies of the business logic.
The gateway can route versions to different deployments if the implementations differ.
Announce deprecation in documentation and with response headers, for example Deprecation: true and Sunset: Wed, 01 Apr 2027 00:00:00 GMT.
Measure usage per version and per client before removing anything.
gRPC and Events
Contract
Practice
gRPC / Protobuf
Package per major version (orders.v1, orders.v2); add fields with new numbers; reserved removed numbers and names; lint with Buf, including breaking-change checks
Events (Avro, Protobuf, JSON Schema)
A schema registry with a compatibility mode (backward, forward or full); include a schema version in messages; publish both old and new formats during a migration
Testing Compatibility
Consumer-driven contract tests (Pact) show which consumers would break.
OpenAPI diff tools flag breaking changes in pull requests.
Buf breaking-change detection for Protobuf.
Pitfalls
Versioning every small change, which leaves dozens of versions to maintain.
Breaking changes without a version bump, relying on “nobody uses that field”.
No deprecation timeline, so old versions live forever.
Duplicated business logic in each version.
Interview Tips
Lead with “avoid breaking changes; additive evolution plus tolerant readers”.
Name the REST schemes and recommend one, usually the URL path.
Explain side-by-side versions, deprecation headers, usage tracking and contract tests.
43
What causes network latency in microservices, and how can it be mitigated?
KnownRevise
In microservices, a single user request often becomes many network calls, so latency comes from: - the number of hops and whether they run in series; - per-call overhead: connection setup, TLS handshakes, serialisation, proxies such as sidecars and gateways; - distance between zones and regions; - queueing in overloaded services; - tail latency, where the slowest of many parallel calls decides the total; - retries and timeouts that add waiting.
Mitigate it by reducing and parallelising calls (coarser APIs, aggregation, async events, local read models), making each call cheaper (connection reuse, HTTP/2 or gRPC, compact payloads, caching), keeping chatty services close (same zone, locality-aware routing), controlling tail latency (timeouts, hedged requests, load-aware balancing), and measuring with distributed tracing.
Read full answerHide full answer
Applies todistributed system performance (vendor-neutral)
Where the Time Goes
Source
Example
Mitigation
Sequential hops
Gateway → orders → customers → pricing, each waiting
If each backend call is slow only 1% of the time, a request that waits for 100 parallel calls is slow with probability 1 − 0.99¹⁰⁰ ≈ 63%. So p99 latency per service matters a lot in systems with large fan-out.
Technique
Effect
Hedged requests
Send a second copy after, say, the p95 delay; use the first response (idempotent reads only)
Load-aware balancing
Least requests or power of two choices avoids slow instances
Partial results
Return what arrived in time, for non-critical parts
Timeouts per call
Bound the damage from one slow dependency
Architectural Mitigations
Approach
Example
Local read models (CQRS)
The order service keeps customer names from events, so it does not call the customer service per request
Caching
Cache-aside in Redis; HTTP caching; CDN at the edge
Asynchronous work
Respond after the critical write; send emails and analytics via events
Co-location
Put services that talk constantly in the same zone or node pool
Merge services
If two services always call each other, they may belong together
Measuring
Distributed tracing (OpenTelemetry with Jaeger or Tempo) shows each hop’s share of total time and which calls are sequential.
Track p95 and p99 per hop, not averages.
Compare server time with client-observed time: the difference is network, proxies and queueing.
Pitfalls
Optimising one service while the request is slow because of a sequential chain.
Retries with long timeouts that double latency.
Cross-region calls in the hot path.
Ignoring connection reuse, paying TLS handshakes on every request.
Interview Tips
Name the main causes: hops, chattiness, connection overhead, distance, queueing, tail latency.
Give mitigations at both levels: per call (pooling, gRPC, caching) and architecture (fewer hops, async, read models).
Mention tracing and the tail-at-scale effect of fan-out.
44
What is the difference between message queues and event buses?
KnownRevise
A message queue delivers each message to one consumer out of a group of competing workers (point-to-point). It is used to distribute work: “resize this image” or “send this email”. Once processed and acknowledged, the message is removed. An event bus (publish/subscribe) delivers each event to every interested subscriber. It is used to announce facts, such as “OrderPlaced”, that several services react to independently, without the publisher knowing who they are. Event streaming platforms such as Kafka add a durable, replayable log: consumers track their own position and can re-read history. Choose a queue for commands and work distribution, and an event bus or stream for notifications, integration between bounded contexts, and replayable history.
Read full answerHide full answer
Applies tomessaging in distributed systems (RabbitMQ, Amazon SQS/SNS/EventBridge, Kafka, NATS, Google Pub/Sub, Azure Service Bus)
Comparison
Aspect
Message queue
Event bus / pub-sub
Event stream (log)
Delivery
One consumer per message
Every subscriber gets a copy
Every consumer group reads the log
Typical message
Command: “do this”
Event: “this happened”
Event: “this happened”
Coupling
Sender expects the work to be done
Publisher does not know its subscribers
Same as pub/sub
Retention
Until acknowledged
Usually until delivered to current subscribers
Time- or size-based retention; replayable
Replay
No
Limited (some offer archives)
Yes, by resetting offsets
Ordering
FIFO options (SQS FIFO, single queue)
Often not guaranteed
Per partition
Scaling consumers
Add competing workers
Each subscriber scales separately
Consumers in a group share partitions
Examples
RabbitMQ queues, SQS, Azure Service Bus queues
SNS, EventBridge, Google Pub/Sub, RabbitMQ fanout/topic exchanges, NATS subjects
Queue (work distribution):
order-service --"GenerateInvoice #981"--> invoice-queue --> one of N invoice workers
Event bus (fan-out):
order-service --"OrderPlaced #981"--> orders topic --> payment-service
--> inventory-service
--> email-service
--> analytics
A common combination: subscribe a queue per consuming service to a topic. Each service then gets every event, and its own workers compete for them (for example, SNS topic → one SQS queue per service, or Kafka consumer groups).
Choosing
Use case
Choice
Background jobs, task processing, rate-limited work
At-least-once delivery is typical, so make consumers idempotent.
Use dead-letter queues for messages that keep failing.
Version message schemas, and use a schema registry for streams.
Monitor queue depth and consumer lag.
Use the transactional outbox to publish reliably after database writes.
Pitfalls
Using a queue to broadcast, so only one service receives an event that several need.
Using events as hidden commands (SendEmailEvent), coupling publisher and consumer again.
Treating Kafka like a traditional queue: per-message acknowledgement and per-message delay are not native (newer Kafka versions are adding queue-like consumer groups).
Large payloads in messages; pass references to stored data instead.
Interview Tips
Summarise: queue = one consumer, work; event bus = all subscribers, facts.
Add event streams as replayable logs.
Mention the topic-to-queue-per-service pattern and idempotent consumers.
45
How can transactional outbox patterns be used in microservices?
KnownRevise
The transactional outbox pattern solves the dual-write problem: a service must update its database and publish an event, but there is no transaction that covers both the database and the broker. Instead, the service writes the event into an outbox table in the same local transaction as the business change. Both commit or neither does. A separate process then relays outbox rows to the broker, either by polling the table or, more efficiently, through change-data-capture (CDC) from the database’s log (Debezium). Delivery is at least once, so consumers must be idempotent. The outbox is the standard building block for reliable event-driven integration, sagas and CQRS read models.
Read full answerHide full answer
Applies torelational databases with Kafka or other brokers (PostgreSQL + Debezium example)
The Problem It Solves
Approach
What can go wrong
Write the database, then publish
Crash after the commit: the event is lost
Publish, then write the database
Database write fails: the event describes something that never happened
Distributed transaction (2PC)
Blocking, poor availability, limited broker support
Outbox
Database and event commit atomically; publishing is retried until it succeeds
Schema and Write
CREATE TABLE outbox (
id uuid PRIMARY KEY,
aggregatetype text NOT NULL, -- e.g. 'order' (used for topic routing)
aggregateid text NOT NULL, -- e.g. the order ID (used as the message key)
type text NOT NULL, -- e.g. 'OrderPlaced'
payload jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
BEGIN;
INSERT INTO orders (id, customer_id, total_cents, status)
VALUES ('981', 42, 2599, 'PLACED');
INSERT INTO outbox (id, aggregatetype, aggregateid, type, payload)
VALUES (gen_random_uuid(), 'order', '981', 'OrderPlaced',
'{"orderId": "981", "customerId": 42, "totalCents": 2599}');
COMMIT;
gen_random_uuid() is built into PostgreSQL 13 and later.
Debezium reads inserts from PostgreSQL’s write-ahead log, with no polling. Its outbox event router turns each row into a Kafka message keyed by aggregateid and routed by default to a topic named after aggregatetype (outbox.event.order). Old outbox rows can be deleted regularly, or deleted right after insert, because CDC reads the log rather than the table.
Polling Relay Versus CDC
Polling relay
CDC (Debezium)
How
Select unsent rows, publish, mark them sent
Read the database transaction log
Latency
Polling interval
Near real time
Database load
Repeated queries
Low
Ordering
ORDER BY an increasing column, one relay per partition
Log order
Operations
Simple; part of the service
Kafka Connect cluster to run
Multiple relay instances
Need locking (FOR UPDATE SKIP LOCKED)
Handled by the connector
Making It Correct End to End
Idempotent consumers: store processed event IDs, or use upserts. A relay crash between publishing and marking causes duplicates.
Ordering: use the aggregate ID as the message key, so events for one entity stay in order within a partition.
Schema versioning: include a type and version; consumers tolerate unknown fields.
Cleanup: delete or archive published rows, so the table does not grow forever.
The inbox pattern on the consumer side (a processed-messages table updated in the same transaction as the consumer’s change) gives effectively-once processing.
Pitfalls
Publishing directly in the request handler “and also” writing the outbox, creating duplicates.
A polling relay without ordering or locking, which reorders or double-sends.
Large payloads in the outbox; store references if events are big.
Forgetting to monitor relay lag.
Interview Tips
Explain the dual-write problem first, then the same-transaction outbox insert.
Compare the polling relay with CDC (Debezium).
Close with at-least-once delivery → idempotent consumers and the inbox pattern.
46
How would you design a microservice to be fault-tolerant?
KnownRevise
A fault-tolerant microservice keeps serving, possibly in a degraded way, when parts of it or its dependencies fail. Design it in layers: - Redundancy: several stateless instances across availability zones, with no single points of failure. - Isolation: bulkheads, separate pools, limits per dependency. - Defensive calls: timeouts, bounded retries with jitter, idempotency, circuit breakers. - Graceful degradation: fallbacks and cached data when a dependency is down. - Durable asynchronous work: queues, outbox, dead-letter queues, so work survives crashes. - Self-healing: health probes, automatic restarts, rollbacks. - Protection against overload: rate limiting and load shedding.
Then verify it with chaos experiments and game days, and measure it against SLOs.
Chaos experiments: kill pods, add latency, drop network packets, fail a zone (Chaos Mesh, LitmusChaos, AWS Fault Injection Service).
Game days: practise incident response with the team.
Load tests combined with injected failures.
SLOs and error budgets show whether the design works in practice.
Pitfalls
Retries without timeouts or budgets, which amplify failures.
Liveness probes that depend on the database, causing restart storms.
Fallbacks that are never tested.
All replicas in one zone.
Assuming the cloud provider makes the service fault-tolerant without application-level design.
Interview Tips
Structure the answer as redundancy, isolation, defensive calls, degradation, durability, self-healing.
Walk through a failure-mode table for one service.
Mention chaos engineering to verify the design.
47
Discuss the importance of timeouts and retry logic in a microservices architecture.
KnownRevise
Timeouts stop a service from waiting forever on a slow or dead dependency, which would hold threads, connections and memory until the caller fails too. Without them, one slow service can cause a cascading outage. Retries recover from transient failures such as dropped connections, brief overload or a pod restart, and make the system more available. Both are dangerous when misused. Timeouts that are longer than the caller’s own deadline waste work. Retries at every layer multiply load: three layers with three attempts each can mean up to 27 calls per request. Retrying non-idempotent operations can duplicate side effects. Good practice is deadline propagation, bounded retries with exponential backoff and jitter, retry budgets, and idempotency keys.
Read full answerHide full answer
Applies todistributed system resilience (example in Python 3.12)
Code Example: Deadline Propagation and a Retry Budget
class DeadlineExceeded(Exception):
pass
class Clock:
"""Fake clock so the example runs instantly and deterministically."""
def __init__(self) -> None:
self.now = 0.0
class RetryBudget:
"""Allow retries only up to a fraction of recent requests, so retries cannot multiply load."""
def __init__(self, ratio: float = 0.2, min_tokens: int = 3) -> None:
self.ratio, self.tokens = ratio, float(min_tokens)
def on_request(self) -> None:
self.tokens = min(self.tokens + self.ratio, 100)
def try_spend(self) -> bool:
if self.tokens >= 1:
self.tokens -= 1
return True
return False
def call(clock: Clock, deadline: float, cost: float, fails: bool) -> str:
"""A downstream call that takes `cost` seconds; it never runs past the caller's deadline."""
remaining = deadline - clock.now
if remaining <= 0:
raise DeadlineExceeded("no time left; not sending the request")
clock.now += min(cost, remaining)
if cost > remaining:
raise DeadlineExceeded(f"timed out after {remaining:.2f}s")
if fails:
raise ConnectionError("connection reset")
return "ok"
def call_with_retries(clock, deadline, budget, attempts):
"""attempts: list of (cost, fails) for each try. Stops on success, deadline or empty budget."""
budget.on_request()
log = []
for n, (cost, fails) in enumerate(attempts):
if n > 0 and not budget.try_spend():
log.append("retry budget exhausted")
break
try:
log.append(call(clock, deadline, cost, fails))
break
except ConnectionError as error:
log.append(str(error))
except DeadlineExceeded as error:
log.append(str(error))
break
return log
clock, budget = Clock(), RetryBudget()
# The gateway gives the whole request 1.0 s; each hop passes the same absolute deadline downstream.
print(call_with_retries(clock, 1.0, budget, [(0.2, True), (0.3, False)]))
# ['connection reset', 'ok'] 0.5 s used
clock.now = 0.0
print(call_with_retries(clock, 1.0, budget, [(0.6, True), (0.6, False)]))
# ['connection reset', 'timed out after 0.40s'] the retry got only the time that was left
clock.now = 0.0
print(call_with_retries(clock, 1.0, budget, [(0.1, True)] * 6))
# ['connection reset', 'connection reset', 'retry budget exhausted']
The gateway sets one absolute deadline for the whole request, and every hop passes it on (gRPC does this automatically with deadlines; with HTTP, use a header). The second call shows that a retry only gets the time that is left. The retry budget caps retries at about 20% of traffic (plus a small reserve), so during an outage retries cannot multiply the load.
Timeout Guidelines
Guideline
Why
Every network call has a timeout
Default library timeouts are often infinite or very long
Set connection and request timeouts separately
Connect quickly or fail; allow realistic processing time
Base timeouts on measured latency (for example a bit above p99)
Too short causes false failures; too long causes pile-ups
Inner timeouts shorter than outer ones
A caller should not give up before its dependency does
Propagate deadlines
Downstream services stop work the caller no longer wants
Cancel work when the deadline passes
Saves resources (context cancellation in Go, CancellationToken in .NET, asyncio timeouts in Python)
Retry Guidelines
Guideline
Why
Retry only transient errors (timeouts, connection resets, 502, 503, 429 with Retry-After)
4xx validation errors will fail again
Retry only idempotent operations, or use idempotency keys
Retry at one layer (often the client closest to the failure, or the mesh)
Avoid multiplicative amplification
Retry budgets and circuit breakers
Stop retrying when a dependency is clearly down
Respect Retry-After headers
Cooperate with the server’s back-pressure
Amplification Example
gateway (3 attempts) -> orders (3 attempts) -> inventory (3 attempts)
worst case: 3 x 3 x 3 = 27 calls to inventory for one user request
During an incident, that multiplication can turn a partial slowdown into a full outage. Limit retries to one layer, or use budgets everywhere.
Where Retries Live
Layer
Pros
Cons
Application library (Resilience4j, Polly, tenacity in Python)
Knows which operations are idempotent
Must be configured in every service
Service mesh (Istio retries, Linkerd retry budgets)
Uniform, no code changes
Does not know business semantics; risks double retries
Message consumers
Natural with redelivery and dead-letter queues
Needs idempotent handlers
Pitfalls
No timeout, or the infinite default in an HTTP client.
Retries without jitter, all clients retrying in sync.
Retrying POST requests without idempotency keys.
Timeouts longer than the user’s patience, or than the gateway’s timeout.
Hiding failures: retries succeed, but nobody notices that the error rate is rising. Track retry counts as metrics.
Interview Tips
Explain why timeouts prevent cascading failures and why retries need limits.
Mention deadline propagation, backoff with jitter, idempotency keys and retry budgets.
Show the amplification arithmetic.
48
What strategies can be used to achieve high availability in microservices?
KnownRevise
High availability (HA) means the system stays usable through failures, deployments and maintenance, and it is usually measured against an availability SLO such as 99.9% (about 43 minutes of downtime per 30 days) or 99.99% (about 4 minutes). The main strategies are: - Redundancy everywhere: several instances of each service spread across availability zones, replicated databases with automatic failover, redundant load balancers. - No single points of failure, including shared dependencies and configuration systems. - Fast detection and recovery: health checks, self-healing orchestration, automated failover. - Safe change management: rolling, canary and blue-green deployments, with disruption budgets and rollback. - Resilience patterns that contain failures: timeouts, circuit breakers, bulkheads, graceful degradation. - Multi-region designs for the highest targets.
Two independent instances, either can serve (1 − 0.005²)
about 99.997%
Three services in series at 99.9% each (0.999³)
about 99.7%
Redundancy multiplies availability up; serial dependencies multiply it down. Fewer synchronous dependencies in the request path, plus redundancy, is the core of HA.
Strategies by Layer
Layer
Strategy
Compute
3+ replicas per service; spread across zones; autoscaling; PodDisruptionBudgets
Cluster
Multi-zone control plane (managed Kubernetes), several node pools; for critical systems, several clusters
Data
Synchronous replication within a region (for example a multi-AZ database), automatic failover, tested backups, cross-region replicas for disaster recovery
Traffic
Redundant load balancers, health-checked routing, global load balancing or DNS failover across regions
Dependencies
Timeouts, circuit breakers, fallbacks, caching, asynchronous queues to decouple
Deployments
Rolling updates with readiness gates, canary releases, automatic rollback, backward-compatible schema changes
Node drains and cluster upgrades will not evict a pod if that would leave fewer than two orders pods running. Combine this with topologySpreadConstraints across zones.
Multi-Region Options
Pattern
Availability
Complexity
Notes
Single region, multi-zone
High (survives zone failures)
Moderate
The usual starting point
Active-passive (warm standby)
Survives region failure; failover takes minutes
Higher
Replicate data asynchronously; possible data loss within the replication lag
Active-active
Highest
Highest
Conflict handling, global databases (Spanner, CockroachDB, DynamoDB global tables), data residency
Pitfalls
Hidden single points of failure: one Redis instance, a single NAT gateway, or a configuration service all services need at startup.
Correlated failures: all replicas in one zone, or the same bad configuration pushed everywhere at once.
Failover never tested.
Long synchronous chains, which reduce availability multiplicatively.
Maintenance causing downtime because disruption budgets are missing.
Interview Tips
Use the availability arithmetic to explain redundancy versus serial dependencies.
Cover multi-zone replicas, replicated data with failover, safe deployments and resilience patterns.
Mention SLOs, PodDisruptionBudgets and testing failover.
49
How do you approach disaster recovery in a microservices-based system?
KnownRevise
Disaster recovery (DR) is the plan for restoring service after a large failure: a region outage, a deleted database, ransomware, or a bad change that corrupts data. Start with business targets per service: the RTO (recovery time objective: how long it may be down) and the RPO (recovery point objective: how much data may be lost). Group services into tiers by these targets. Then choose a DR strategy for each tier, from backup and restore through pilot light and warm standby to active-active multi-region. Make the system recreatable from code (infrastructure as code, GitOps, container images in replicated registries), protect data with backups and cross-region replication (including immutable, isolated backups against ransomware and mistakes), and rehearse failover and restores regularly. In microservices the hard part is data consistency across many services after a restore, so plan the recovery order and reconciliation too.
Backups copied to another region; rebuild everything there
Hours; RPO equals the backup interval
Low
Pilot light
Data replicated continuously; compute kept minimal or off
Tens of minutes
Low to medium
Warm standby
A scaled-down full copy running in the second region
Minutes
Medium to high
Active-active
Several regions serve traffic all the time
Near zero
Highest; needs multi-region data design
What Must Be Recoverable
Asset
Approach
Infrastructure
Terraform or similar, tested by creating environments from scratch
Kubernetes workloads
GitOps (Argo CD, Flux): a new cluster syncs from Git
Cluster state and volumes
Velero backups of resources and persistent volumes
Container images
Registry replicated to the DR region
Databases
Point-in-time recovery, cross-region replicas, backups in a separate account
Event streams
Kafka cross-cluster replication (MirrorMaker 2, Cluster Linking), with offset translation for consumers
Secrets and keys
Replicated secret stores and KMS keys; recovery procedures for root credentials
DNS and traffic
Health-checked global routing or DNS failover with low TTLs
Data Consistency After Recovery
Each service restores its own database, so the restored copies may come from slightly different points in time: an order may exist without its payment record. - Restore in dependency order, and define which service is the source of truth for each piece of data. - Replay events from the outbox tables or the event log to fill gaps. - Run reconciliation jobs that compare services (orders versus payments) and fix or flag mismatches. - Idempotent consumers make replays safe.
Runbook Outline
1. Declare the disaster; assign an incident commander; freeze deployments.
2. Fence the failed region (stop writes there to avoid split brain).
3. Promote the DR databases; confirm the replication lag at cut-over (actual RPO).
4. Scale up DR compute; sync workloads from Git.
5. Switch traffic (global load balancer or DNS); watch SLO dashboards.
6. Run reconciliation; replay missing events.
7. Communicate status; later fail back in a planned, rehearsed way.
Testing
Regular DR drills (game days), including real restores into a clean environment; measure the actual RTO and RPO.
Chaos experiments that simulate a zone or region loss.
Automate the runbook as far as possible; manual steps fail under stress.
Pitfalls
Backups never restored, so they turn out to be incomplete or unusable.
DR region missing a dependency: an image registry, a secret, a quota limit, or a third-party allow list.
Backups in the same account that an attacker or a mistake can delete.
Ignoring cross-service consistency after restores.
Failback never planned.
Interview Tips
Start with RTO and RPO per service tier.
Name the four DR strategies and their trade-offs.
Highlight rebuild from code, data replication, immutable backups, consistency after restore and drills.
50
Explain how you would handle a cascading failure in a microservice ecosystem.
KnownRevise
A cascading failure happens when one failing or slow service causes its callers to fail, and the failure spreads through the dependency graph. For example: the inventory service slows down, so order-service threads wait on it, its pools fill, the gateway times out, and clients retry, which adds even more load. Handle it in two parts: - During the incident: stop the spread first. Shed load and cut traffic to the failing component (rate limits, circuit breakers, turning off non-critical features), stop retry storms, roll back the change that triggered it, scale or restart the bottleneck, then ramp traffic back gradually. - Prevent it by design: every call has a timeout, retries are bounded with backoff, jitter and budgets, dependencies are isolated with bulkheads, circuit breakers fail fast, services shed load before they collapse, and graceful degradation keeps the core working.
Read full answerHide full answer
Applies tomicroservice resilience and incident response (vendor-neutral)
How Failures Cascade
Mechanism
Example
Resource exhaustion
Threads or connections held while waiting on a slow dependency
Retry amplification
Each layer retries, multiplying load on the struggling service
Overload spread
Traffic fails over to the remaining instances, which then fail too
Shared dependencies
One database or cache serves many services
Thundering herd on recovery
Everyone reconnects at once and knocks it over again
Health-check cascades
Liveness probes that depend on the database restart every pod at once
Responding During an Incident
Step
Action
1. Identify the bottleneck
Tracing and dependency dashboards: which service’s latency or errors rose first?
2. Stop the trigger
Roll back the recent deployment or configuration change; disable a feature flag
3. Reduce load
Rate-limit at the gateway, shed low-priority traffic, pause batch jobs and consumers
4. Stop retry storms
Lower retry counts, open circuit breakers, ask clients to back off (429 with Retry-After)
5. Restore capacity
Scale out the bottleneck, increase pool sizes carefully, fail over the database
6. Recover gradually
Ramp traffic up slowly; warm caches; watch for a second wave
7. Learn
Blameless postmortem; add the missing safeguards
Preventing Cascades by Design
Pattern
What it does
Timeouts and deadlines
Free resources when a dependency is slow
Circuit breakers
Fail fast instead of waiting when a dependency is failing
Bulkheads
Separate thread or connection pools per dependency, so one cannot drain the others
Retry budgets and jitter
Limit retry load; avoid synchronised waves
Load shedding and admission control
Reject excess work early (fast 503s) instead of queueing until everything times out
Graceful degradation
Serve cached or partial results; switch off non-critical features
Autoscaling with headroom
Capacity for spikes, plus limits so scaling does not overload the database
Asynchronous decoupling
Queues absorb bursts; consumers process at their own pace
Correct health checks
Liveness checks only the process itself; readiness reflects whether it can serve
Caching with jittered expiry
Avoids stampedes on the backing store
Example: Checkout Under a Slow Inventory Service
Without protection:
inventory p99 rises from 50 ms to 5 s
-> order-service threads all wait -> order-service stops responding
-> gateway times out -> clients retry 3x -> 3x load on order-service
-> the whole site is down
With protection:
300 ms timeout + circuit breaker on inventory
-> breaker opens after a burst of errors; calls fail fast
-> checkout shows "stock confirmed by email" and reserves later from a queue
-> other pages stay healthy; inventory recovers without the extra load
Testing
Chaos experiments that inject latency rather than only crashes; slow dependencies cause most cascades.
Load tests above capacity, to check that load shedding works.
Review the dependency graph for critical services that depend synchronously on non-critical ones.
Pitfalls
Scaling the callers when the bottleneck is the dependency, which makes it worse.
Restarting everything at once.
Unbounded queues that hide overload until latency explodes.
Fallbacks that call another fragile service.
Interview Tips
Explain how cascades start: resource exhaustion and retry amplification.
Give the incident response order: find, stop the trigger, shed load, recover gradually.
List the preventive patterns and mention latency injection in chaos testing.
51
What tools or practices would you recommend for logging in a distributed microservices system?
KnownRevise
In a distributed system, logs from dozens of services and hundreds of pods must be collected centrally, searchable and correlated. Recommended practices: - Structured logs (JSON) with consistent field names. - A correlation ID: ideally the OpenTelemetry trace ID in every log line, so one request can be followed across services. - Log to stdout, and let a node-level agent (Fluent Bit, Vector or the OpenTelemetry Collector) ship logs to a central store such as Elasticsearch or OpenSearch, Grafana Loki, Splunk, Datadog, or a cloud service. - Consistent levels and sampling of noisy debug logs. - No secrets or personal data. - Retention policies matched to cost and compliance.
Logs complement metrics (for alerting) and traces (for request flow). They are not a replacement for either.
import contextvars
import io
import json
import logging
request_id = contextvars.ContextVar("request_id", default="-")
class JsonFormatter(logging.Formatter):
def format(self, record):
entry = {
"level": record.levelname,
"service": "orders",
"request_id": request_id.get(),
"msg": record.getMessage(),
}
entry.update(getattr(record, "fields", {})) # structured key-value fields
return json.dumps(entry)
stream = io.StringIO()
handler = logging.StreamHandler(stream)
handler.setFormatter(JsonFormatter())
log = logging.getLogger("orders")
log.addHandler(handler)
log.setLevel(logging.INFO)
def handle(headers, order_id):
# Reuse the caller's ID so one request has one ID across every service.
request_id.set(headers.get("X-Request-ID", "generated-7f3a"))
log.info("order placed", extra={"fields": {"order_id": order_id, "total_cents": 2599}})
log.warning("inventory slow", extra={"fields": {"latency_ms": 840}})
handle({"X-Request-ID": "req-91c2"}, "981")
for line in stream.getvalue().splitlines():
print(line)
The request ID arrives in a header and is stored in a ContextVar, so every log line in that request carries it without passing it through every function. In production, use the OpenTelemetry trace ID instead of a custom header, so logs link directly to traces.
Logging Pipeline
service (JSON to stdout) -> container runtime log files on the node
-> agent (Fluent Bit / Vector / OpenTelemetry Collector), as a DaemonSet
-> parse, enrich (pod, namespace, service), filter, redact
-> store (Loki / OpenSearch / Elasticsearch / Splunk / cloud logging)
-> query and dashboards (Grafana / Kibana / OpenSearch Dashboards)
Tool Choices
Need
Options
Collection agent
Fluent Bit, Vector, OpenTelemetry Collector
Cheap, label-indexed storage
Grafana Loki
Full-text search and analytics
Elasticsearch or OpenSearch with Kibana or OpenSearch Dashboards
Managed services
Datadog, Splunk, Google Cloud Logging, AWS CloudWatch Logs, Azure Monitor
Long-term archive
Object storage (S3, GCS) with lifecycle rules
Practices
Practice
Detail
Consistent schema
Same field names in every service: timestamp, level, service, trace_id, span_id, msg; the OpenTelemetry semantic conventions are a good base
Correlation
Propagate traceparent (W3C Trace Context) and log the trace ID
Meaningful levels
ERROR needs action; WARN is unusual; INFO marks business events; DEBUG is off in production or sampled
Log events, not noise
One line per significant event, not per loop iteration
Redaction
Never log passwords, tokens, card numbers; mask personal data; redact in the agent as a safety net
Bounded cardinality of labels
In Loki, labels such as service and namespace; never user ID or request ID as labels
Retention tiers
Short hot retention (7–30 days), cheaper archive for compliance
Audit logs
Separate, append-only and access-controlled
Access control
Logs often contain sensitive data; restrict who can query them
Pitfalls
Plain-text logs that need fragile regular expressions to parse.
No correlation ID, so finding one request across services is guesswork.
Logging everything at DEBUG, which drives cost up and hides important lines.
Writing log files inside containers, which are lost when the pod dies.
Alerting on log patterns for things a metric would measure more reliably.
Interview Tips
Lead with structured logs, trace IDs and central aggregation.
Describe the agent → store → query pipeline and name tools.
Mention redaction, sampling, retention and logs alongside metrics and traces.
52
How do you trace requests across boundaries of different microservices?
KnownRevise
Requests are traced with distributed tracing. The first service, often the gateway, starts a trace with a unique trace ID. Each unit of work, such as handling a request or calling a database, is a span with its own ID, a parent span ID, a start time and a duration. When a service calls another, it propagates the trace context in request headers (the W3C traceparent header) or message metadata (Kafka headers). The next service continues the same trace, creating child spans. Each service exports its spans to a tracing backend, which assembles them into a timeline of the whole request. The standard is OpenTelemetry (OTel): SDKs and automatic instrumentation for most languages, the OTel Collector for processing, and backends such as Jaeger, Grafana Tempo or Zipkin, or commercial tools.
Read full answerHide full answer
Applies toOpenTelemetry and W3C Trace Context (vendor-neutral)
Propagation in Headers
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
| | | |
| trace ID (16 bytes) parent span ID flags (01 = sampled)
version
tracestate: vendor-specific values (optional)
baggage: tenant=acme,plan=pro (optional key-value context)
Every hop keeps the trace ID and puts its own span ID as the new parent.
What a Trace Looks Like
trace 4bf92f35... total 412 ms
gateway POST /checkout |==============================| 412 ms
orders POST /orders |========================| 350 ms
orders SELECT customers |==| 28 ms
inventory gRPC Reserve |============| 190 ms
inventory UPDATE stock |=========| 150 ms
kafka publish OrderPlaced |=| 12 ms
This shows immediately that the inventory database update dominates the latency.
Implementation Steps
Step
Detail
1. Instrument services
OTel SDK plus automatic instrumentation for HTTP servers and clients, gRPC, databases and messaging libraries; add manual spans for important business steps
2. Propagate context
Use W3C Trace Context everywhere; make sure gateways, meshes and proxies pass the headers through
3. Cross async boundaries
Inject context into message headers when publishing; extract it when consuming (as a child span or a span link)
4. Export
OTLP to an OTel Collector (agent or gateway)
5. Sample
Head sampling (for example 10%) at the start, or tail sampling in the Collector to keep slow and error traces
6. Store and view
Jaeger, Grafana Tempo, Zipkin, or commercial APMs
7. Correlate
Put the trace ID in logs; link metrics to traces with exemplars
This keeps every error trace, every trace slower than 500 ms and 5% of the rest. Tail sampling needs all spans of a trace to reach the same Collector instance, so scaled-out setups route by trace ID.
Service Meshes
Istio and Linkerd can create spans for each hop automatically, but applications must still forward the trace headers from incoming to outgoing requests. Otherwise each hop starts a new, disconnected trace.
Pitfalls
Broken context propagation through a proxy, thread pool or message queue, giving fragmented traces.
Sampling everything at 100% in busy systems, with high cost.
High-cardinality or sensitive span attributes (tokens, personal data).
No link from logs to traces.
Interview Tips
Define trace, span and context propagation with the traceparent header.
Name OpenTelemetry, the Collector and a backend (Jaeger or Tempo).
Mention async propagation, sampling strategies and log correlation.
53
Discuss the importance of metrics and alerts in maintaining a microservices architecture.
KnownRevise
With many services, nobody can watch logs to see whether the system is healthy, so metrics provide the numbers that summarise behaviour over time: request rate, error rate, latency, saturation for every service, plus business signals such as orders per minute. Metrics are cheap to store and fast to query, so they drive dashboards, autoscaling, capacity planning and alerts. Alerts turn metrics into action: they notify a human when users are, or soon will be, affected. Good alerting is based on SLOs (service level objectives) and error-budget burn rates, which page for real user impact and avoid noise. Alerts that page too often cause alert fatigue, so real incidents get missed. Every page should be urgent, actionable and linked to a runbook.
Read full answerHide full answer
Applies toPrometheus-style metrics and SLO-based alerting (Python 3.12 example)
What to Measure
Method
Signals
Applies to
RED
Rate, Errors, Duration (latency histograms)
Request-driven services
USE
Utilisation, Saturation, Errors
Resources: CPU, memory, pools, queues, disks
Four golden signals
Latency, traffic, errors, saturation
Any user-facing service
Business metrics
Orders, sign-ups, payments succeeded
Catch failures that technical metrics miss
Use histograms for latency, so p95 and p99 can be computed across instances. Keep label cardinality bounded: never use user IDs as labels.
Code Example: Multi-Window Burn-Rate Alerts
SLO = 0.999 # 99.9% of requests succeed over 30 days
BUDGET = 1 - SLO # 0.1% of requests may fail
# (long window, short window, burn-rate threshold, action), from the SRE workbook
RULES = [
("1h", "5m", 14.4, "page"), # 2% of the monthly budget gone in 1 hour
("6h", "30m", 6.0, "page"), # 5% gone in 6 hours
("3d", "6h", 1.0, "ticket"), # 10% gone in 3 days
]
def burn_rate(errors, total):
return (errors / total) / BUDGET if total else 0.0
def evaluate(windows):
"""windows maps a window name to (errors, total requests)."""
alerts = []
for long_w, short_w, threshold, action in RULES:
long_burn = burn_rate(*windows[long_w])
short_burn = burn_rate(*windows[short_w])
# Both windows must burn fast: the long one proves it is significant,
# the short one proves it is still happening.
if long_burn > threshold and short_burn > threshold:
alerts.append(f"{action}: {long_w}/{short_w} burn {long_burn:.1f}x")
return alerts or ["ok"]
outage = {"1h": (1_800, 100_000), "5m": (160, 8_000),
"6h": (2_100, 600_000), "30m": (900, 50_000),
"3d": (2_400, 7_200_000)}
recovered = dict(outage, **{"5m": (2, 8_000)})
slow_leak = {"1h": (150, 100_000), "5m": (12, 8_000),
"6h": (900, 600_000), "30m": (75, 50_000),
"3d": (10_800, 7_200_000)}
print(evaluate(outage)) # ['page: 1h/5m burn 18.0x']
print(evaluate(recovered)) # ['ok'] (short window shows it has stopped)
print(evaluate(slow_leak)) # ['ticket: 3d/6h burn 1.5x']
A burn rate of 1 uses the error budget exactly over 30 days; 14.4 would use it in about two days. Requiring both a long and a short window to burn fast means real incidents page quickly, a brief blip does not, and the alert clears soon after recovery. A slow leak opens a ticket rather than waking someone up.
Burn-rate alerts instead of arbitrary static limits
Runbooks
Each alert links to diagnosis and mitigation steps
Routing and grouping
Alertmanager or PagerDuty route to the owning team; group related alerts; silence during maintenance
Review regularly
Delete or fix alerts that fire without action
Pitfalls
Too many alerts, so on-call engineers ignore them.
Only cause-based alerts (CPU, memory) that miss real user impact.
Averages instead of percentiles for latency.
High-cardinality labels that overload the metrics system.
No alert for missing data: a service that stops reporting looks healthy.
Interview Tips
Explain RED and USE and why metrics drive dashboards, scaling and alerts.
Describe SLOs, error budgets and multi-window burn-rate alerts.
Stress actionable alerts with runbooks and avoiding alert fatigue.
54
How do you handle performance bottlenecks in microservices?
KnownRevise
Handle performance bottlenecks with a measure-first loop: 1. Define the target, for example a p99 latency or a throughput SLO. 2. Find where time is spent across services with distributed tracing, then inside the service with profiling (continuous profilers, flame graphs). 3. Find which resource saturates (CPU, memory, connection pools, database, locks, network) with metrics. 4. Fix the biggest contributor. 5. Verify with load tests and production metrics, then repeat.
Give concrete bottlenecks with fixes (N+1 queries, pool exhaustion, chatty calls, CPU throttling).
Mention scaling the bottleneck and tail latency.
55
What is distributed tracing and which tools help in microservices?
KnownRevise
Distributed tracing records the path and timing of a single request as it moves through many services, queues and databases. Each request gets a trace ID, each unit of work is a span (with a name, start time, duration, attributes and a parent span), and the trace context travels between services in headers such as W3C traceparent. A tracing backend joins the spans into one timeline, which shows where latency comes from, which call failed, and how services depend on each other. The standard toolkit is OpenTelemetry for instrumentation and collection, with backends such as Jaeger, Grafana Tempo or Zipkin (open source), or commercial APMs such as Datadog, Honeycomb, New Relic, Dynatrace, AWS X-Ray, Google Cloud Trace and Azure Application Insights.
One operation: an HTTP handler, a database query, a message publish
Parent and child spans
Show which operation caused which
Attributes
Key-value details: http.route, db.system, order.id (bounded, no secrets)
Events and status
Timestamped notes inside a span; OK or ERROR status
Context propagation
Passing trace and span IDs across process boundaries
Span links
Connect related traces, for example a batch consumer to the producers’ traces
Sampling
Keep only a share of traces (head-based or tail-based) to control cost
What Tracing Answers
Which service made this request slow? The longest span in the critical path.
Where did the error start? The first span with an error status.
Are calls sequential that could run in parallel? The shape of the timeline.
What depends on what? Service maps built from traces.
Is the slowness one user or everyone? Compare traces by attributes.
Tooling Landscape
Layer
Tools
Instrumentation
OpenTelemetry SDKs and automatic instrumentation (Java agent, Python, Node.js, .NET, Go libraries); eBPF-based auto-instrumentation (Grafana Beyla, OpenTelemetry eBPF)
Collection and processing
OpenTelemetry Collector: batching, tail sampling, attribute redaction, export to several backends
Open-source backends
Jaeger, Grafana Tempo, Zipkin, SigNoz
Commercial and cloud
Datadog APM, Honeycomb, New Relic, Dynatrace, Elastic APM, Lightstep (ServiceNow), AWS X-Ray, Google Cloud Trace, Azure Application Insights
Service mesh
Istio and Linkerd emit spans for each hop (apps must still forward headers)
OpenTelemetry keeps instrumentation vendor-neutral: the backend can change by reconfiguring the Collector, without touching service code.
Rolling Out Tracing
Start with automatic instrumentation in the most important services and the gateway.
Make sure every proxy and queue propagates context.
Add manual spans for key business steps.
Use tail sampling to keep errors and slow traces.
Link logs and metrics to traces (trace IDs in logs, exemplars in metrics).
Pitfalls
Partial instrumentation: one uninstrumented hop breaks the trace.
Too many attributes, or sensitive ones, raising cost and privacy risk.
100% sampling in high-traffic systems.
Treating tracing as a replacement for metrics: sampled traces are bad for alerting on rates.
Interview Tips
Define trace, span and context propagation.
Name OpenTelemetry + Collector + Jaeger or Tempo, and a commercial option.
Mention sampling and correlation with logs and metrics.
56
Explain the role of Docker in developing and deploying microservices.
KnownRevise
Docker packages each microservice with its runtime, libraries and configuration defaults into an image: an immutable, versioned artifact that runs the same on a laptop, in CI and in production. For microservices this provides: - Consistency: no more “works on my machine”. - Isolation: each service has its own dependencies and language version. - Fast, repeatable builds with layer caching. - Easy local development of many services with Docker Compose. - A standard deployment unit that orchestrators such as Kubernetes schedule, scale and restart.
Images follow the OCI standard, so other runtimes (containerd, CRI-O, Podman) run the same images. Kubernetes runs them through containerd or CRI-O; it does not need the Docker daemon.
Read full answerHide full answer
Applies toDocker Engine and Docker Compose v2, OCI images
Example: A Multi-Stage Dockerfile for a Go Service
# syntax=docker/dockerfile:1
FROM golang:1.23 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download # cached unless dependencies change
COPY . .
RUN CGO_ENABLED=0 go build -o /out/orders ./cmd/orders
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/orders /orders
USER nonroot
EXPOSE 8080
ENTRYPOINT ["/orders"]
The final image has only the binary: it is small, starts fast and has a minimal attack surface.
Cover Compose for local work, one image through CI to production, and Kubernetes for orchestration.
Mention multi-stage builds, non-root users and scanning.
57
How do container orchestration tools like Kubernetes help with microservice deployment?
KnownRevise
Running dozens of services with many replicas by hand is not practical. An orchestrator such as Kubernetes lets teams declare the desired state, for example “run 4 replicas of orders v3.2, with these resources and this configuration”, and control loops keep reality matching it. For microservices, Kubernetes provides: - scheduling containers onto nodes; - self-healing: restarting crashed containers and replacing lost pods; - rolling updates and rollbacks; - service discovery and load balancing (Services, DNS); - horizontal autoscaling; - configuration and secrets management; - health checks; - resource isolation; - an extensible API for ingress, service meshes, GitOps and operators.
Other orchestrators exist, such as HashiCorp Nomad, Amazon ECS and Docker Swarm, but Kubernetes is the de facto standard.
Changing the image tag triggers a rolling update: a new pod starts, becomes ready, and only then does an old pod stop.
The Deployment Workflow
git push -> CI builds and scans the image -> pushes ghcr.io/example/orders:3.2.0
-> updates the manifest in the GitOps repo
-> Argo CD / Flux applies it -> Kubernetes rolls out
-> optional canary analysis (Argo Rollouts, Flagger) -> full rollout or automatic rollback
It does not fix application design: services still need timeouts, idempotency and graceful shutdown.
Stateful systems are harder; managed databases are often simpler.
For a handful of services, simpler platforms (Cloud Run, ECS, App Service) may be enough.
Pitfalls
No resource requests, so the scheduler packs nodes badly and pods are evicted.
Missing readiness probes, so rolling updates send traffic to pods that are not ready.
Ignoring graceful shutdown (SIGTERM handling), causing errors during deployments.
Interview Tips
Lead with declarative desired state and reconciliation.
List scheduling, self-healing, rolling updates, discovery, autoscaling and configuration.
Mention GitOps and progressive delivery on top.
58
Describe the lifecycle of a container within a microservices architecture.
KnownRevise
A container’s lifecycle starts before it runs: source code is built into an image, scanned and signed, then pushed to a registry. At deploy time, the orchestrator schedules a pod, the node pulls the image, init containers run, and the main container starts. It then passes a startup probe, becomes ready (and receives traffic), and runs, monitored by liveness probes and restarted when it fails. It may be scaled, updated or rescheduled. On termination it gets SIGTERM, is removed from load balancing, drains in-flight work within a grace period, and is killed with SIGKILL if it has not exited. Containers are ephemeral: their filesystem disappears, so state lives in databases, volumes or object storage.
Read full answerHide full answer
Applies toDocker/OCI containers on Kubernetes 1.30+
End-to-End Stages
Stage
What happens
Key practices
Build
Dockerfile → image layers
Multi-stage builds, small base images, reproducible builds
Verify
Tests, vulnerability scan, SBOM, signing
Trivy or Grype, Syft, Cosign; fail CI on critical vulnerabilities
Store
Push to a registry with a version tag and digest
Immutable tags, retention policies
Schedule
Scheduler picks a node by resources and constraints
The short preStop sleep gives load balancers time to stop sending traffic before the app receives SIGTERM. The app then stops accepting connections, finishes in-flight requests and closes database pools within the 45-second grace period. Recent Kubernetes versions also support a built-in sleep action for preStop.
Pitfalls
Ignoring SIGTERM: the process is killed mid-request, so users see errors on every deployment.
A shell as PID 1 that does not forward signals; use exec-form ENTRYPOINT or a small init such as tini.
Liveness probes that are too strict, restarting healthy but busy containers.
Local state in the container filesystem, lost on restart.
Interview Tips
Walk through build → registry → schedule → pull → start → ready → run → terminate.
Explain probes and graceful shutdown with SIGTERM and grace periods.
Stress that containers are ephemeral and immutable.
59
How do you ensure that containers are secure and up-to-date?
KnownRevise
Container security covers the whole supply chain and runtime: - Build minimal images from trusted, pinned base images. - Scan them for known vulnerabilities (CVEs) and secrets in CI, and keep rescanning images already in the registry, since new CVEs appear daily. - Rebuild automatically when base images or dependencies get patches (Renovate or Dependabot plus scheduled rebuilds). - Sign images and generate SBOMs, and enforce at admission that only signed images from approved registries, with no critical vulnerabilities, can run. - Harden the runtime: run as non-root with a read-only filesystem, drop Linux capabilities, apply seccomp, keep resource limits, and use network policies. - Detect suspicious behaviour at runtime. - Patch the hosts and the orchestrator too.
Read full answerHide full answer
Applies toOCI containers on Kubernetes 1.30+ (tool examples are open source)
Supply-Chain Controls
Control
Tools
Minimal base images
Distroless, Chainguard or Wolfi images, Alpine, -slim variants
Dependency updates
Renovate, Dependabot (including base image updates in Dockerfiles)
Deploying by digest guarantees the exact scanned and signed image runs. Enforce these settings cluster-wide with the Pod Security Admission “restricted” level.
Keeping Images Up to Date
1. Renovate opens a PR when the base image or a dependency has a new version.
2. CI rebuilds, tests, scans and signs the image.
3. Merging updates the GitOps manifest; Argo CD / Flux rolls it out.
4. A scheduled job (for example weekly) rebuilds every service even without code changes,
so OS package fixes in the base image are picked up.
5. Continuous registry scanning alerts when a running image gains a new critical CVE.
Track a patch SLA, for example critical CVEs fixed in production within 7 days, and measure it.
Runtime and Platform Security
Area
Practice
Runtime detection
Falco or Tetragon alert on shells in containers, unexpected processes, writes to sensitive paths
Network
Default-deny NetworkPolicies; mTLS through a service mesh
Identity
Separate service accounts per workload; least-privilege RBAC; workload identity for cloud access
Secrets
External secret stores (Vault, cloud secret managers); mounted at runtime
Nodes
Patched, minimal OS images (Bottlerocket, Container-Optimized OS); regular node upgrades
Cluster
Supported Kubernetes versions; audit logging; restricted API access
Pitfalls
Scanning only at build time, then running the image for months.
Using latest tags, so it is unclear what is running and whether it is patched.
Mention scanning, SBOMs, signing and admission policies.
Explain automated rebuilds for staying current, plus runtime hardening and detection.
60
What are the best practices for container networking in microservices?
KnownRevise
Good container networking for microservices means services find each other reliably, traffic is encrypted and restricted to what is needed, and the network is observable and resilient. In Kubernetes: - Every pod gets its own IP on a flat network provided by a CNI plugin (Cilium, Calico, or a cloud CNI). - Services expose stable virtual IPs and DNS names. - Ingress or the Gateway API handle external traffic.
Best practices: - Use DNS-based service discovery, never hard-coded pod IPs. - Default-deny NetworkPolicies with explicit allow rules. - mTLS between services, usually through a service mesh. - Separate edge and internal traffic. - Plan IP address space. - Tune timeouts, connection reuse and DNS. - Monitor network flows.
With default deny on egress, every service also needs explicit egress rules to its own dependencies, such as the database. Remember to allow DNS, or nothing resolves.
Best Practices
Area
Practice
Discovery
Use Service DNS names; keep clients’ DNS caching and connection reuse sensible
Segmentation
Namespaces per team or domain; default-deny policies; allow only the needed ports
Encryption
mTLS between services (mesh or library); TLS at the edge; certificate rotation with cert-manager
Edge
One entry point (gateway) with TLS, authentication, rate limiting and a WAF; internal services not exposed publicly
Egress control
Restrict outbound traffic; route external calls through egress gateways or FQDN policies (Cilium)
Load balancing
gRPC and HTTP/2 need L7 or client-side balancing, because long-lived connections stick to one pod
Timeouts
Set timeouts at the gateway, mesh and clients consistently, with inner ones shorter
IP planning
Pod and Service CIDRs that do not overlap with VPCs and on-premises networks; enough addresses for scaling
DNS performance
NodeLocal DNSCache; watch the ndots:5 default, which causes extra lookups for external names (use fully qualified names ending in a dot, or tune ndots)
Observability
Flow logs and service maps (Cilium Hubble, mesh telemetry); latency and error metrics per hop
Pitfalls
Flat networks with no policies, so one compromised pod can reach every database.
Exposing internal services with LoadBalancer Services by accident.
Hard-coding IPs or depending on pod IPs, which change on every restart.
Default-deny without DNS rules, which breaks everything.
Connection-level balancing of gRPC, overloading one pod.
Interview Tips
Explain pod IPs, Services and DNS, and Ingress or the Gateway API at the edge.
Stress default-deny NetworkPolicies and mTLS.
Mention gRPC load balancing, DNS tuning and egress control.