vLLM in Production: Evaluating Inference Beyond a Single Benchmark

Inference performance is a property of a workload running on a system. A fast result from one prompt, one concurrency level, or one precision setting does not establish how a service will behave when real users arrive.

Tokyo AI’s vLLM Deep Dive, held on 24 April 2026, brought together perspectives on distributed inference, KV caches, compression, and production serving. The useful connection between them is how to evaluate an optimization without losing sight of the service it must support.

Begin with what the user is waiting for

Leonard Lin, CTO and Co-founder of Shisa.AI, separates time to first token, time per output token, total throughput, and quality in his production discussion. Each describes a different part of the experience.

An interactive application may feel unresponsive if the first token arrives late, even when the eventual generation is fast. A batch process may tolerate that delay if it completes a large queue efficiently. A long answer may begin promptly but still take too long to finish. A single “tokens per second” figure cannot settle all three cases.

Lin’s approach uses workload distributions, including input and output lengths, alongside concurrency sweeps. The practical lesson is to preserve the shape of the work. Short prompts that produce long answers put different pressure on the system from long documents that require a few extracted fields.

A cache moves work, but it also creates a data path

Tony Valderrama, Head of Product at Momento, describes KV cache architecture through the separation of storage and compute. The key-value cache holds intermediate attention state that can avoid repeating part of the computation. His talk explores local storage, sharing between workers, and more persistent remote storage.

The attractive possibility is reuse: computation already performed can become useful to another stage or request. The constraint is moving the resulting data quickly enough. Storage capacity alone does not tell you whether a cache arrangement helps latency. Transfer volume, available bandwidth, locality, and the cost of a miss all matter.

Prefill, which processes the input, and decode, which generates subsequent tokens, can also place different demands on hardware. Separating them introduces opportunities to allocate resources differently, alongside additional coordination and transfer requirements. It is an architecture to test against a workload, not a universal next step for every deployment.

Compression changes the evaluation question

Lin’s examples put quality measurements next to performance measurements when comparing precision choices. That pairing is essential. Reducing memory use can make a configuration feasible, but the application still needs acceptable outputs and useful response times.

There is also a distinction between compressing model weights and changing how cached state is stored or transferred. They act on different resources and can introduce different overheads. A memory-saving configuration can be valuable without being the fastest configuration for a particular request pattern.

For a Japanese-language application, the evaluation set should reflect its actual language and tasks. A general score cannot tell the team whether a smaller representation preserves an important name, correctly follows a document format, or produces an acceptable translation. The relevant tradeoff is application quality within the service’s operating limits.

Foundry’s perspective: benchmark the service you intend to run

Imagine an illustrative service that summarizes Japanese support histories and generates a short draft response. The initial PoC might test a handful of clean conversations. Before selecting production infrastructure, we would build a representative evaluation set that includes short and long histories, repeated context, unusual terminology, and the busy periods the service must handle.

First establish a quality baseline with a configuration that performs the task adequately. Then change one important serving choice at a time: concurrency, precision, caching, or hardware. Record whether the output still satisfies the task and what happens to initial delay, completion time, and resource use. Retain the model, software version, hardware, and test conditions with the result.

The rollout also needs behavior under pressure. When the queue grows, should the service wait, reject new work, or use a separately evaluated fallback? That decision affects both the customer experience and capacity planning. An optimization that looks good during a stable run may be less useful if recovery from a worker restart is slow or difficult to observe.

This is the connection to Foundry Labs’ production systems and compute work: choose the configuration that meets the business service’s requirements, and keep the evidence needed to revisit that choice. For implementation details, the current vLLM production metrics documentation provides the maintained reference; the April talks supply the architectural questions.

Sources and further reading

This article draws on the A vLLM Deep Dive - Distributed Inference, KV Cache Evolution, and Model Compression programme and the talks identified above. Tokyo AI (TAI) connects researchers, engineers, and industry; explore Foundry Labs’ community work for more context.

Talk materials from Tun Jian Tan, Tony Valderrama, and Leonard Lin are available upon request. Please use our contact form and include the event title and speaker name.

Featured speaker profiles

Tun Jian Tan. AI Engineer at Embedded LLM and vLLM committer. His talk covers the project and its development. LinkedIn.

Tony Valderrama. Head of Product at Momento. His talk traces the evolution of the KV cache in vLLM. LinkedIn.

Leonard Lin. Co-founder and CTO of Shisa.AI, working on multilingual models and production inference. LinkedIn.

All articles