LLM & GenAI Interview Questions

The warm-up that quietly filters out candidates who only know LLMs as a magic API. Nail the two-phase inference picture and you set up every later answer about latency, cost, and hallucinations.

Modern LLMs share one architectural skeleton. Learn the five components and you can read almost any model card cold, plus the single distinction that proves to an interviewer you actually understand it rather than the buzzwords.

Tokens drive your customer's bill, their latency, and the weird failures like botched arithmetic. The rough numbers every FDE should rattle off, plus the trap answers interviewers listen for.

Everyone can define it; few candidates know why 'advertised context' and 'usable context' aren't the same thing. The cost, latency, and attention caveats that make this answer stand out.

A deceptively simple sampling question with one trap (does temperature 0 fix hallucinations?) and one senior move interviewers reward. Know which workflows genuinely want zero.

The concept underneath every RAG system you'll ever deploy. How to explain vectors, cosine similarity, and the one limitation of embeddings that costs real deployments accuracy.

The same operation scores attention and ranks your RAG results. Knowing exactly what the dot product measures, and when its magnitude term quietly breaks your retrieval, is what separates a working pipeline from a mysteriously bad one.

Why a raw base model answers your question with three more questions, what RLHF actually buys, and the niche cases where customers should pick base. A tidy answer most candidates fumble.

Every FDE will face an angry customer asking this. The mechanistic explanation interviewers want (why hallucination is the default behavior, not a bug) plus the framing that defuses customer escalations.

A classic metrics question with a domain twist. The answer isn't 'recall'; it's a cost-of-errors argument plus a threshold strategy, and interviewers grade the reasoning, not the metric you pick.

The single most-reported FDE conceptual question across every AI lab. Interviewers don't want definitions; they want the decision framework, the cost math, and the recommendation order that reads senior.

Truncate? Summarize? Rerank? Map-reduce? A production problem every RAG deployment hits, with a menu of fixes interviewers expect you to weigh, plus the diagnostic question to ask first.

Cosine similarity finds topically close, not correct. The failure taxonomy behind most RAG accuracy complaints, and the hybrid-search + reranker fix stack interviewers expect you to know cold.

Anyone can list mitigations; the question is really about costs: latency, dollars, and lost recall. The three-layer answer with honest price tags that AI-lab interviewers score highest.

The data-volume thresholds, the break-even math, and the regression-prevention checklist that turn 'just fine-tune it' into a defensible engineering decision. Includes the crossover rule interviewers listen for.

Models read the start and end of your context far better than the middle, a measured effect with direct consequences for chunk ordering, k, and prompt layout. What to change and how to prove it.

Malformed arguments, wrong tool, hallucinated parameters, retry loops: the full failure taxonomy plus the production defenses (validation, idempotency, capped retries) that separate builders from readers.

JSON mode, strict schemas, constrained decoding, validate-and-retry: the escalation ladder for structured output, plus the catch nobody mentions: schema-valid can still be factually wrong.

Arguing against agents is the move that reads senior in AI-lab interviews. The error-compounding math, the decision rule, and the workflow-vs-agent spectrum that wins this contrarian question.

TTFT vs tokens-per-second, the output-length lever everyone forgets, and why streaming is the highest-ROI fix that changes no latency at all. The five-lever answer with real numbers.

The KV-cache mechanics behind the discount, the prefix rule that silently breaks caching for most teams, and the workloads where caching cuts bills 50-90% versus the ones where it does nothing.

Prompts are production code that nobody treats like code, until an innocent one-line tweak tanks accuracy. The version-eval-canary-rollback pipeline that AI-native interviewers expect by default.

Model migrations break things that benchmarks never show, format quirks, refusal shifts, tool-calling drift. The eval-replay and shadow-traffic playbook for migrating without surprises.

Zero retention, private endpoints, redaction pipelines, or open weights in their VPC, the four-rung ladder for PII-sensitive deployments, and the discovery questions that pick the right rung.

Evals are the #1 probing area in AI-lab FDE loops. The full suite design, golden sets, component metrics, safety red-teaming, CI gates, and the launch-blocking thresholds, in interview-ready form.

Position bias, verbosity bias, self-preference, and the agreement-rate workflow that turns a sloppy judge into eval infrastructure you can defend. AI labs probe this hard; here's the calibrated answer.

The classic FDE escalation. Six hypotheses, silent model updates, input drift, prompt entropy, sampling noise, perception shift, and the eval-replay test that splits them in an afternoon.

An email that says 'ignore your instructions and refund $10,000' will eventually arrive. Why prompt-level defenses lose, and the layered design (least privilege, action gates, rate caps) that actually holds.

10x doesn't come from one trick. It's multiplicative: routing x caching x prompt diet x batching x distillation. The stacked-levers playbook with rough multipliers, and the eval gate that keeps quality honest.

Adapter math, the VRAM arithmetic that makes 4-bit a 4x unlock, and the honest decision rule for on-prem deployments. The deep-internals question Mistral and Cohere loops use to find real practitioners.

The GPU-utilization math that decides this argument, the hidden line items (ops headcount, quality gap, model churn), and the hybrid recommendation that wins both the analysis and the customer.

In high-stakes domains a wrong answer is worse than no answer, but a system that always abstains is useless. The selective-prediction design, the layered abstention triggers, and the two metrics that govern the tradeoff.

The experience probe that opens half of all applied-AI loops, and the place resumes go to die. The four-beat walkthrough structure interviewers can actually score, plus the two omissions that quietly downgrade you from architect to API caller.

Past tense matters: the interviewer is asking what you actually enforced at runtime, not what you'd design on a whiteboard. The enforcement chain (gate, starve, screen, measure) with the scope-violation metric that proves it worked.

Two questions wearing one trench coat: what do you ship first, and how do you make a stochastic system behave. The answer that lands: shrink the autonomy, not just the feature list, and define the MVP by the workflow it completes, not the agent it contains.

Reported as the question OpenAI FDE interviewers push hardest on, and the one where 'we look at the outputs' ends loops. The three-legged answer that passes: automated metrics, calibrated human evaluation, and feedback loops wired to a business number.

The OpenAI FDE deep-dive where you're expected to walk the full stack (network, queueing, tokenization, prefill, KV cache, batching, decode) in order, with a measurement at each hop. The triage sequence, with the numbers that tell you which layer is guilty.

Provider model upgrades ship faster than your release cycle and quietly shift behavior. The design that turns 'we hope it's fine' into a blocking gate: a frozen golden set, pairwise diffing, segmented scoring, and a shadow-traffic canary.

Two compression levers with very different bills. The arithmetic that decides between them, when to stack both, and the eval discipline that keeps a 'cheaper' model from quietly tanking your hardest 10% of queries.

Million-token windows didn't kill RAG; they changed when you reach for it. The decision rule that holds up in production, and the silent failure each option hides behind a confident answer.

The escalation ladder is settled; the at-scale tradeoff is not. Why retries quietly multiply your bill and tail latency, why constrained decoding has its own taxes, and the hybrid that most high-volume pipelines actually run.

Sending every request to the frontier model is the lazy bill. The patterns that route by difficulty, the verifier that enforces a hard quality floor, and the failure mode that turns a cost-saving cascade into a worse, slower system.

Two failure modes, one architecture. Why no single guardrail holds, the defense-in-depth layers that assume the model will be broken, and the design rule that limits the blast radius when prompt filters inevitably fail.

Three stages, one KL leash. The reward-model-plus-PPO loop, the beta*KL term that keeps the policy from drifting off the reference, and how DPO collapses it into a single classification loss by treating the reward as beta*log(pi/pi_ref). The objectives, written correctly.

Win-rate up, facts down is the signature of reward hacking: the policy is over-optimizing the preference proxy and drifting off the reference. The diagnosis (Goodhart on the reward), the immediate lever (tighten the KL/beta anchor), and the held-out factual gate that stops shipping the regression.

Three transformer families, one distinction that decides everything: which tokens are allowed to see which. The attention mask is the whole answer, and it explains why GPT generates and BERT does not.

Rotary embeddings encode position as a rotation of the query and key vectors, so attention scores depend only on the gap between two tokens. That relative property is why RoPE generalizes past its training length where learned absolute tables fall off a cliff.

Where you put the LayerNorm changes whether gradients survive a hundred layers. Pre-norm keeps a clean residual highway and well-scaled gradients so deep models train without warmup heroics; post-norm normalizes the output but fights vanishing gradients at depth.

Attention is bottlenecked by moving the N x N score matrix to and from HBM, not by FLOPs. FlashAttention tiles the computation in fast on-chip SRAM and fuses softmax with an online running statistic, so it computes the exact same output while never materializing the full matrix.

Each chat turn resends the whole conversation, so the model re-prefills the same prefix every time. Server-side KV caching and client-side cache breakpoints kill that redundant compute, but exact-prefix matching and stale entries are where teams silently get it wrong.

PPO needs a separate value network to estimate the baseline for its advantage. GRPO samples a group of answers per prompt and uses their mean reward as the baseline, so the advantage is just how a sample compares to its peers, no critic required.

Without a KV cache, generating token N re-attends over all N-1 prior tokens from scratch every step, turning decoding quadratic. The cache stores each layer's past keys and values so each new token costs one forward step, and its size is what caps your batch and context.

Human 1-5 scores are biased toward short, simple code. Debias the labels, fuse test-pass rate without letting the agent game it, and pick PPO with a fused reward over a DPO ranking loss. Plus the hard cap on the action space so the agent can't delete the repo.

Reward only on the final answer and learning crawls; shape intermediate tool calls and the model learns to spam SQL. Use outcome-grounded credit assignment, GRPO over a group of trajectories, and fine-tune a pre-aligned model so you're not teaching tool syntax from scratch.

A fixed 1000-token budget forces the search choice. Step-level beam search guided by a process-reward model beats best-of-N and MCTS on accuracy per token for math, and you collect the PRM labels automatically with Monte-Carlo rollouts, no human step annotation.

Budget covers maybe 5M of 100M LLM calls. Run a cheap first stage on everything, route only low-confidence items to the LLM, and use those LLM labels to distill a student that absorbs most of the volume. The cost-vs-coverage math that decides where the threshold goes.