A year ago, "design a system around a large language model" was reserved for ML infrastructure roles. In 2026 it reaches general software engineering loops. Candidates report prompts like "design a customer-support chatbot on top of a third-party LLM" and "add a generative summary feature to this feed" in ordinary backend interviews. Interviewers now expect you to reason about LLMs, vector stores, and retrieval as part of standard system design, not as a niche specialty.
You don't need to know how transformers work. You need the systems vocabulary around them.
The shape of the question
Most LLM prompts in general loops are integration problems with an AI label on them. "Design a support chatbot" decomposes into things you already know: an API gateway, a session store, a queue, rate limiting, and one unusual dependency, a model endpoint that is slow, expensive, occasionally wrong, and priced per token. Treat the model as a flaky, costly downstream service and most of your existing instincts apply.
The four topics interviewers probe
Latency and streaming. Model calls take seconds, not milliseconds. Interviewers expect you to reach for token streaming to mask latency, timeouts and fallbacks around the call, and async patterns for anything that doesn't need to be interactive. Saying "we stream tokens to the client so first-token latency is what the user feels" signals you've actually operated one of these.
Cost. Every call has a per-token price. Expect "what does this cost at scale?" as a follow-up, and have levers ready: caching identical or near-identical queries, routing easy requests to a smaller model, capping context length, batching offline work. Cost reasoning gets graded now; it's worth its own post.
Retrieval. If the bot must answer from company data, you're designing RAG. Chunk documents, embed them, store vectors, retrieve top-k at query time, stuff them into the prompt. Know that pipeline at the box-diagram level, including where it goes stale (re-embedding on document updates) and where it fails (retrieval misses turning into confident wrong answers).
Guardrails and evaluation. The model will sometimes produce wrong or unsafe output. Strong answers mention input/output filtering, a fallback to human handoff, and, this is the differentiator, how you'd know quality regressed: logging model outputs, sampling them for review, tracking thumbs-down rates. "How do you evaluate it?" is the follow-up that separates candidates who've shipped LLM features from those who've read about them.
A 30-second frame you can reuse
"I'll treat the LLM as an expensive, slow, unreliable external dependency. That means: async or streaming interfaces around it, aggressive caching in front of it, a budget and rate limits on it, retrieval to ground it in our data, and logging plus sampled evaluation so we know when it degrades."
Deliver that early, then design the ordinary system around it. Interviewers aren't looking for ML research depth in a general loop. They're checking whether you can be trusted to ship a feature that has a model in the middle of it. Candidates fail this question by treating the LLM as either magic or as someone else's problem. It's neither. It's a component, with the worst SLA on your diagram.