What does caching mean in programming?
Caching is the practice of storing a temporary copy of data in a faster, closer storage layer so that repeated requests for that same data can be served without re-running the full, more expensive retrieval or computation.
Read full answerHide full answer
Applies toall versions
The Problem It Solves
Before caching, every request would have to go all the way to its original source to get data. For example, a news feed could hit a database for every user refresh, or a product page could recompute the same pricing for thousands of views. These paths are often orders of magnitude slower: a memory read is roughly ~100 ns, an SSD read roughly ~100 µs, a same-datacenter round trip roughly ~0.5 ms, and a cross-region round trip roughly ~50-150 ms (all rough orders of magnitude). Caching cuts down on this cost by serving from a closer tier.
Where It Fits In A Real System
Caching can exist at nearly every layer depending on the goal:
-
Browser/HTTP cache: Uses headers like
Cache-Control,ETag,max-age, ands-maxageto avoid re-downloading static assets. -
CDN: Caches edgeable responses closer to users across regions.
-
Reverse-proxy cache: Sits in front of your services (e.g. Nginx) to cache safe, repeatable HTTP responses.
-
Application cache: In-process or shared caches held by your API to avoid calling other services repeatedly.
-
Distributed cache: Shared across many API instances (commonly used to hold serialized objects) to prevent every instance from re-fetching the same data.
-
Database query/buffer cache: Internal to the database to speed up repeated reads, but this is more of a database internals topic than an application caching design choice.
Trade-Offs
Nothing is free when introducing caching:
-
Latency vs consistency: You gain speed, but the copy can become stale the moment the source changes.
-
Complexity vs simplicity: You now have two sources of truth to reason about, with invalidation rules, keys, and expiry.
-
Cost vs savings: Caching can save database I/O and egress, but memory for a distributed cache costs money and adds operational effort.
-
Availability vs correctness: Serving stale data can keep a read path up during partial failures, but it can also serve incorrect data for sensitive flows.
For this reason, you would not use caching for highly volatile, per-user sensitive actions where the smallest bit of staleness is unacceptable (e.g. showing the exact remaining balance in a payment service). The choice must be based off of the workload.
Rules To Follow When Applying It
-
Invalidation is mandatory: A cache without a plan to remove or update its entries is almost always the mistake candidates make. Expiry alone is not an “ignore invalidation” strategy.
-
Use eviction intentionally:
LRU,LFU,FIFO, orTTLall behave differently under pressure. Pick based off of access patterns, not hype. -
Plan for cache-specific failures: Watch for cache stampedes/thundering herd, hot keys, and cache penetration. Protect them with techniques such as locking, request coalescing, early expiry with jitter, caching negative results, or bloom filters when appropriate.
-
Be strict with what goes in: Never put raw per-user secrets, one-time tokens, or anything that must be strictly fresh across all users onto a shared cache. Hit-ratio can be a useful metric, but chasing 100% is a red herring if it comes at the cost of correctness.
Interview Tips
-
What interviewers look for: The ability to explain the problem before the solution, state concrete trade-offs, and talk about invalidation without jumping straight to a tool.
-
Common follow-up: “When would you not cache this?”
-
Biggest mistake: Adding a cache to “make it faster” without defining consistency, keying, and invalidation for that exact data flow.