← all posts

Cost per request is the new scalability question

Cost per request is the new scalability question

For a decade, system design interviews rewarded one instinct: make it scale. Add replicas, add caches, add regions. In 2026, loops at senior level grade a second axis explicitly: what does this cost, and is it worth it? Candidates report "what's the monthly bill for this design?" and "cost per request?" as standard follow-ups, with interviewers calibrating senior versus staff on the quality of the answer.

The shift has an obvious cause. Zero-interest-rate infrastructure habits met cloud bills and LLM per-token pricing, and every platform team spent the last two years doing cost reviews. Interviewers now want evidence you were in the room for those.

What cost reasoning sounds like in an interview

You don't need exact AWS prices. You need proportions and drivers: which component dominates the bill and what it scales with.

"The vector search cluster dominates here — it's memory-bound, so cost scales with corpus size, not traffic. The LLM calls are second and scale linearly with usage, so that's where caching pays. Everything else is noise by comparison."

Two sentences, no pricing table, and it demonstrates the thing being graded: you know where the money goes and which knob moves it.

The levers interviewers expect you to know

Caching as a cost tool, not just a latency tool. A 40% cache hit rate on an expensive downstream call is a 40% discount. Say it that way.

Tiering. Hot data on fast expensive storage, cold data on object storage. Recent events queryable, old events archived. Most "design a feed/analytics/logging system" questions hide a tiering decision.

Right-sizing the guarantee. Strong consistency, multi-region active-active, and 99.999% are expensive. The senior move is asking whether this data needs that. A shopping cart, yes. A like counter, no.

Batch versus real-time. Real-time pipelines cost multiples of batch ones. If the product need is a daily report, saying "this doesn't need streaming" earns more points than designing beautiful streaming.

Model routing, for LLM systems. Small model for easy queries, large model for hard ones, cache in front of both. In any AI-flavored question, this is the expected cost answer.

When to bring it up

Don't wait to be asked. The natural moments are right after capacity estimation ("at this volume, the driver of cost will be X") and whenever you choose between two viable architectures ("option A is simpler, option B is roughly a third the cost at this scale — I'd take B if the traffic projection is real"). One unprompted cost observation per interview is the right dose. A candidate who narrates dollars constantly reads as dodging the actual design.

The trap on the other side

Cost-cutting instinct without judgment fails too. Under-provisioning a payments system's durability to save money is a worse signal than overspending. The graded skill isn't frugality. It's knowing which guarantees the business is actually buying. "This is the one place I'd happily overpay" is a senior sentence.

Scalability asked: can you make it big? Cost asks: should you? Loops now want both answers.