System Design Interview Questions for AI/ML

Every queue, webhook, and retry loop you'll ever deploy at a customer hides this distinction. Interviewers use it to separate people who've shipped from people who've read about shipping.

A staple at OpenAI and Anthropic in both coding and design rounds. Most candidates can name the algorithms; few can say which one lets 2x your limit through, and that's the part that gets scored.

Averages lie, and interviewers know most candidates quote them anyway. The fastest way to sound production-grade in any latency discussion is to reason about percentiles correctly, including the one trap almost everyone falls into.

One malformed message can silently wedge an entire pipeline, and it happens at every customer eventually. The DLQ answer interviewers want covers what most candidates forget: what happens *after* the message lands there.

This isn't a deployment question, it's a judgment question wearing one. Interviewers are scoring whether you have a decision framework, a safety checklist, and the spine to say no to a customer politely.

A Fermi estimate with a paycheck attached. The interviewers' favorite filter: candidates who jump to a dollar figure miss the two numbers that actually break deployments, peak QPS and tokens-per-minute limits.

A signature FDE triage question with a hidden rubric: interviewers score the *order* of your moves, not just the list. There's a reason 'what changed?' beats 'check the dashboards', and a reason mitigation beats diagnosis.

The signature FDE debugging genre: a moving fault, an opaque client, and a customer who's sure it's your fault. The winning method splits the problem at the boundary, and knows the four classic culprits hiding on their side.

The bug only appears under load, the code 'hasn't changed,' and the customer just got charged twice. There's one root-cause pattern behind almost every version of this incident, and a fix with a subtle race most candidates miss.

Vercel's signature triage question. 3s TTFB is almost never 'the server is slow', it's a rendering-strategy problem with four classic causes, and the rubric rewards proving each fix with a measurement, not vibes.

Standard observability tells you the request returned 200 in 800ms. It cannot tell you the answer was wrong. The strong answer names the new failure plane LLMs introduce, and takes a real position on the prompts-and-PII question.

Prompts are production code with a worse failure mode: the regression returns 200 OK. The answer interviewers reward treats a one-word prompt edit with the same machinery as a schema migration, with one LLM-specific twist at every stage.

A 'simple' queue question with four traps wired in: minutes-long tasks break every default timeout, cancellation has to actually stop spend, progress needs a contract, and one bad job must never wedge the lane.

API keys for systems you don't own, in environments you can't always reach, audited by security teams who don't trust you. The strong answer has one organizing principle, and a concrete story for rotation and the air-gapped case.

The Palantir decomposition round has no right answer, it has a right METHOD, and it's the most-failed interview in the FDE loop. Here is the four-move rubric interviewers actually score: clarify, model, spine, iterate.

The classic Palantir decomposition prompt. The winning answer hinges on one asymmetry candidates rarely surface unprompted, and on starting with something embarrassingly simpler than an ML model.

The most open-ended prompt in the Palantir canon, no scale given, no user given, no disaster given. What separates passing answers is one design assumption about the data that almost nobody states out loud.

The purest data-modeling prompt Palantir asks, the whole interview lives in the entity-resolution edge cases. Twins, maiden names, shared insurance IDs... and one architectural decision about merging that candidates get fatally wrong.

Looks like an algorithms question; it's actually an operations question wearing one. The candidates who fail reach for 'solve VRP'; the ones who pass design for the 2pm reality that invalidates every morning plan.

The decomposition prompt we use to teach the skill that actually decides these interviews: graceful iteration under perturbation. Your score isn't your design; it's the delta when the design gets punched.

You control the sender; thousands of customer endpoints you've never seen control your fate. The rubric hides in three places: what 'ordering' really costs, why retries need the receiver's cooperation, and the debugging surface most designs forget.

Rate limits protect your servers; quotas protect your business model, and most candidates design one and think they've designed both. The strong answer also covers the part customers actually feel: what hitting the limit looks like.

Enterprise deals die in this design review. The answer has three layers most candidates blur together: who you are, what you can see, and how an embedded product proves both, plus the permission-staleness trap that fails Glean loops.

The defining Palantir-style deployment question. Everything your laptop did silently (package pulls, API calls, telemetry, auth callbacks, clock sync you never thought about) becomes a named workstream. Strong answers produce the checklist by category, not by anecdote.

The pattern is expand-contract; the interview is everything around it: the backfill that locks the table, the rollback nobody rehearsed, and the fact that it's the customer's database, not yours. One named technique carries the whole answer.

The CFO sees API invoices; nobody sees the cost of the inference team they're about to hire. The strong answer models utilization, the variable that decides this question, and names the break-even honestly enough to be trusted.

Two questions wearing one trenchcoat: classic multi-region (where the hard part is data, not compute) and dependency failover (where the hard part is that your fallback model gives different answers). Degraded modes are designed, not improvised.

The model is an untrusted code author with an unlimited imagination, and prompt injection means it can be *aimed*. Strong answers pick an isolation primitive deliberately, then close the channel most designs leave wide open: the data the code is allowed to touch.

The question that defines forward-deployed engineering as a discipline: production you're responsible for but can't touch. The answer is a system (version skew, telemetry tiers, diagnostic bundles, release trains), not a collection of favors from customer IT.

The capstone FDE question, where every topic in this category shows up as one workstream. The rubric isn't the target architecture; it's whether you can sequence the gap-closing while the customer keeps using the pilot.

A live-reported Palantir FDSE decomp with a built-in twist: after you design for the driver, the interviewer flips you to Uber's side. The expected-value-per-hour model, the tip-prediction ethics trap, and how to survive the flip.

A reported Palantir FDSE decomposition where the fatal move is putting triage inside an algorithm. The bed-capability data model, the visibility-before-optimization v1, and the conflicting-priorities answer that keeps clinicians in charge.

Reported loop detail: roughly 5 hours to build a RAG system, agent, or eval harness on OpenAI's APIs, a recorded video walkthrough, then a 60-minute defense. The hour-by-hour allocation, why the eval section is the differentiator, and what the video is actually screening for.

The round is conversational discovery plus live Excalidraw diagramming, and it's graded on sequence: candidates who draw before asking fail. The discovery battery, the voice-pipeline boxes worth drawing, and the latency budget that anchors the whole design.

The reported Anthropic open-ended design, and the round where candidates get tested on GPU memory and KV-cache arithmetic mid-whiteboard. The serving spine, the memory math that explains every product limit, and the safety layer the rubric explicitly wants.

Active-active is easy until two regions write the same agent's memory at once. The strong answer picks a write model on purpose, names what conflicts and what cannot, and refuses to pretend a long-running agent run survives a regional failover for free.

No egress is the easy half. The hard half is that an auditor will read your logs, your access trail, and your model-update process for the next three years. The strong answer designs the evidence the audit consumes as a first-class output.

Latency, cost, and quality are one budget with three claims on it. The strong answer treats the p99 SLA and the spend ceiling as a joint constraint, finds where they fight (batching), and names the lever it pulls when traffic exceeds what the ceiling can buy at SLA.

A crashed agent is easy. The hard one returns a confident answer after a tool quietly returned nothing and the model made up the rest. The strong answer instruments the steps that fail without throwing, ties traces to an outcome signal, and refuses to call a 200 a success.

A port is three coupled scheduling problems wearing a trench coat: where ships dock, which cranes work them, and where containers land in the yard. The strong answer decomposes them, names where they couple, and refuses to solve the whole thing as one giant optimizer.

Per-account scoring catches the lone fraudster and misses the ring whose accounts each look clean. The strong answer reframes the problem as a graph, makes entity resolution the load-bearing step, and treats a flagged ring as a case for an investigator, not an automatic block.

CAP is a trap if you recite it. The interviewer wants to see you map a customer's real datastore to a real failure mode, then decide where stale reads are free and where they corrupt an agent's decision.

Caching LLM output looks like free latency and cost savings until a cache key collides across tenants or a popular key expires under load and stampedes your model. The senior answer is mostly about what you refuse to cache.

It reproduces once in a thousand runs, never on your laptop, and the customer's data is wrong. This question separates engineers who name the race and reach for the right lock from those who sprinkle mutexes and pray.

The customer's team has to live with this API for years, on a schedule you don't control. The interview is less about REST vs GraphQL and more about how you ship changes without a 2 a.m. page on their side.

The best FDE deployment is the one that doesn't need the FDE anymore. This question screens for whether you build for handoff from day one or quietly make yourself indispensable, which is the failure mode that looks like success.

A Copilot/Cursor-style assistant lives or dies on the latency budget. The strong answer spends every millisecond of a 200ms end-to-end budget on purpose, separates the inline-ghost-text path from the chat path, and shows why prefix caching and speculative decoding are the only ways to hit it at 10k concurrent users.

A gateway that screens every prompt and streamed token against a live policy without blowing the latency budget. The strong answer separates the policy decision from its data, runs deterministic checks inline and model checks in parallel, and redacts a leaking secret mid-stream before it renders.

Five LLM agents over a hundred steps, and the bar is zero contradictions with a 500-page contract. The real test is the state and grounding layer, not the agent prompts.

No data leaves, you cannot pre-inspect the corpus, and your toolchain arrives on a single USB. Design adaptation and evaluation when you will never observe a single input or output.

Ten X-ray studies a second, a 2-second budget, and a missed cancer is unacceptable. The hard parts are recall-first thresholding, a monitor that watches the radiologist, and surviving a new scanner.

Fifty milliseconds at p99 to catch toxicity, PII, and injection on every message. An LLM in the hot path blows the budget. The strong answer tiers the checks and keeps the LLM off the critical path.

Under 500ms from end of speech to first audio out, across three models and a network. The trick is that you never wait for any stage to finish: you stream and pipeline everything.

Cache hits on similar prompts cut cost and latency, but one false hit serves a stranger's answer to your question. Picking the threshold and handling user-specific data is the whole game.

A 2-second inference and a user who closed the tab. Naive stacks keep the GPU burning to completion. Real cancellation means propagating disconnect all the way to the batch scheduler.

One GPU, callers blocking on the response, and a hardware unit that is only efficient when fed wide batches. The whole job is trading a few milliseconds of queue wait for a large jump in throughput without ever stalling a request.

A 4-hour outage is ~5.6x your entire monthly error budget on a 99.9% SLA. You cannot ride it out. The design is a provider abstraction, a circuit breaker, and a tiered set of degraded modes, plus a clear line on what you cache versus what you replay.

The source can only upload the file a handful of times before it saturates, but every receiver also has upload bandwidth. The win is making peers serve each other, so total capacity grows with the swarm instead of bottlenecking on one machine.

Every request emits a usage event worth real money, clients retry on timeouts, and tenants must be stopped at their quota and never overcharged. The hard parts are idempotent usage events and the succeeded-but-confirmation-lost case where the work happened but the client never heard back.

Modulo hashing reshuffles everyone the day you add a cell. Consistent hashing fixes that and then loses to the two constraints that actually decide placement in an enterprise deployment: jurisdiction and who must not share a blast radius with whom.

A back-of-the-envelope round where the envelope has a purchase order attached. The arithmetic is the easy half. The scoring is in which assumptions you write down, which way you round, and what you say about the number being wrong.

Every integration lesson assumes a queue and nobody designs one. Ordering per entity rather than globally, a dead letter path somebody actually reads, and what your pipeline does when the consumer is slower than the producer for four hours.

Aggregate metrics hide exactly the customer who is about to churn. Adding a tenant label to everything is the obvious fix and it is also how you get a cardinality explosion that takes down the monitoring before it takes down the platform.

The abstain path is designed last and decides whether the deployment is adopted. Routing to the right person, an SLA the queue can actually meet, and what happens to the case the agent already half-processed before it gave up.