Measure throughput and user latency before choosing a batching policy
Use a scoped roofline estimate and a three-request timing fixture to distinguish queueing, prefill and decode. Calculate aggregate throughput over a common window, compare goodput against latency limits, and test batching rather than assuming a universal sweet spot.
30 MIN · PLUS
a free account unlocks the core curriculum tier · no card
