The most common AI-flavored system design prompt of 2026 is some variant of: "we have internal documents / a knowledge base / support articles — design a system that answers questions from them." This is a retrieval-augmented generation (RAG) question, and it has become to this decade what "design a URL shortener" was to the last one. A standard prompt with a known shape, where all the differentiation is in how you handle the edges.
The pipeline everyone draws
Ingestion: documents get chunked, each chunk gets embedded, vectors plus metadata go to a vector store. Query: embed the user's question, retrieve the top-k similar chunks, assemble them into the model's context, generate an answer, ideally with citations.
Drawing this earns you nothing. It's the assumed baseline. The interview happens in the follow-ups.
Where the follow-ups go
Freshness. "A document was updated an hour ago — what does the system answer?" You need an update path: re-chunk and re-embed changed documents, delete stale vectors. Event-driven (on document save) versus scheduled re-indexing is a real trade-off worth saying out loud. Stale retrieval is the number-one real-world RAG failure, and interviewers who've operated one will always probe it.
Retrieval quality. Pure vector similarity misses exact identifiers: error codes, product names, SKUs. The expected answer is hybrid retrieval, combining semantic search with keyword search (BM25 or your search engine's equivalent) and merging results. Mentioning a re-ranking step over the merged candidates is a senior-level flourish. Knowing why hybrid matters is the actual requirement.
Permissions. This is the sleeper question that kills otherwise strong answers: "can an intern's question retrieve the executive compensation doc?" Access control must be enforced at retrieval time. Filter chunks by the caller's permissions in the vector store query, never after generation. If you only say one security sentence in the round, make it this one, because post-generation filtering means the secret already reached the model's context.
Wrong answers. The model will sometimes answer confidently from bad retrieval, or from nothing. Mitigations worth naming: require citations to retrieved chunks, threshold on retrieval similarity ("if nothing scores well, say 'I don't know'"), and log question/retrieval/answer triples with user feedback so quality is measurable. "How do you know it's getting worse?" is coming. A sampled evaluation set is the answer.
Capacity and cost, briefly
Vector stores are memory-hungry. Cost scales with corpus size, not query volume, and that asymmetry is worth stating. Embedding calls at ingestion are cheap and batchable; generation calls at query time are the expensive part, so semantic caching of common questions is the lever. Two sentences of this satisfies the cost follow-up that's now standard in senior loops.
A closing frame
RAG questions reward candidates who treat them as boring distributed-systems problems: an indexing pipeline with a freshness SLA, a query path with a latency budget, access control at the data layer, and observability on quality. The AI part is two boxes on the diagram. Candidates who fixate on the model do worst. Candidates who fixate on the data flow around it do best.