Why DeepSeek Is Fast and Inexpensive at Scale but Costly for Local Deployment
If you have ever wondered why DeepSeek models are renowned for their speed and cost efficiency in large cloud deployments, yet appear slow and expensive when used locally, you are about to discover the fascinating story of transformer inference batching.
The Basics: Throughput Versus Latency
The entire topic begins with a fundamental trade-off in machine learning—the choice between throughput and latency. Throughput represents a system's ability to process a large number of requests simultaneously, while latency measures how quickly it responds to an individual request. Service providers often face a dilemma: choose high throughput for many users with slower individual responses, or low latency for fast individual responses with inefficient utilization across many users. At small scale, models such as DeepSeek-V3 use GPUs so inefficiently that they require many user requests to be batched together to achieve reasonable throughput. This characteristic is not accidental—it is a consequence of the transformer architecture and the way these models process information.
The Power of Batching Across Users
The key to DeepSeek's efficiency is batching inference across multiple user requests instead of processing each request individually. Transformers are designed so that generating completions for a batch of inputs is nearly as fast as generating one for a single input, provided the batch is constructed efficiently. What does this mean in practice? Imagine a situation where thousands of users are simultaneously sending queries to your model. Instead of processing each query in isolation, you can group these queries into batches and process them in parallel. The transformer architecture allows processing operations and other computations to be performed simultaneously for the entire batch, dramatically increasing efficiency. However, efficient batching requires sequences within a batch to have similar lengths so that token generation can be synchronized. At scale, when many users are submitting requests, DeepSeek can process many of them in parallel, maximizing GPU utilization and minimizing the cost per request.
Batching Challenges and Limitations
However, batching is not a cure-all. It is efficient only when the sequences being processed are at the same stage—for example, when they are all generating the same token position. Differences between sequences require separate attention operations and reduce batching efficiency. Modern inference stacks use "continuous batching," where new requests are added to a batch as soon as capacity becomes available. However, the fundamental trade-off between throughput and latency remains. Locally, you rarely have enough simultaneous requests to create large, efficient batches, resulting in poor GPU utilization, slower responses, and higher costs per request. For truly large-scale use cases, distributed batch inference spreads batches across multiple GPU nodes. This is precisely how cloud providers achieve scale and cost savings with models such as DeepSeek. When hundreds or thousands of GPUs organized into clusters are available, enormous batches can be created, achieving incredible efficiency.
For an individual user or a small team running DeepSeek locally, however, this scale is unattainable, so these efficiency benefits are lost. A single query or a few simultaneous queries simply do not create a large enough batch to use the model's computing power efficiently.
Key Observation
DeepSeek models are architecturally designed to deliver efficiency through large-batch, high-throughput inference. This is achievable only at scale, where many user requests can be grouped together. Running DeepSeek locally or for individual requests results in small batch sizes, poor GPU utilization, and much higher costs per request, making it impractical for scenarios that lack scale.
This phenomenon explains why we see such a contrast between the experiences of cloud service users employing DeepSeek and those attempting local deployment. It is not a flaw in the model or its implementation—it is a natural consequence of how these advanced language models are fundamentally designed and optimized.
