DeepSeek V4.1 Flash Hits 848 Tok/s on GB300 NVL72 Rack Infrastructure

(SAN FRANCISCO —) Inference economics for hyperscale artificial intelligence systems shifted dramatically this week as telemetry data confirmed that DeepSeek V4.1 Flash achieved an extraordinary 848 tokens per second per agent when deployed across Nvidia’s dense GB300 NVL72 rack architecture. This performance milestone represents a massive 252 token-per-second acceleration compared to baseline readings captured on the exact same physical infrastructure barely a month prior, underlining the compounding velocity of full-stack software and algorithmic co-design.

While foundational model parameters and bare-metal hardware specifications remained constant across the testing intervals, continuous iteration in model weight quantization, KV-cache memory management, and serving compiler efficiency unlocked unprecedented decoding speeds. For enterprise architects and capacity planners steering multi-tenant agentic pipelines, these metrics signal a definitive transition away from memory-bandwidth-bound inference bottlenecks toward highly optimized, high-throughput execution regimes.

Technical Architecture & Engineering Breakdown

At the core of this performance leap is DeepSeek V4.1 Flash, a highly sophisticated multimodal Mixture-of-Experts (MoE) architecture totaling 552 billion parameters. Despite its massive parameter footprint, the model leverages extremely sparse routing mechanics, activating only 16 billion parameters during the autoregressive decoding phase when generating text or processing visual inputs. This structural sparsity is the primary driver behind its ability to sustain rapid token generation without overwhelming memory subsystems.

The hardware foundation housing this workload is Nvidia’s flagship GB300 NVL72 architecture, which crams 72 high-performance Blackwell GPUs into a single, tightly coupled compute rack. Within this dense enclosure, NVLink interconnect bandwidth allows for near-instantaneous tensor streaming across nodes, mitigating the inter-GPU communication latency that traditionally hobbles multi-billion-parameter MoE architectures during distributed tensor and pipeline parallelism.

Engineers analyzing the delta between the earlier 596 tokens/sec benchmark and the current 848 tokens/sec figure note that the improvement cannot be attributed to a single variable. Instead, the gains stem from an integrated optimization loop spanning model pruning techniques, customized Triton and TensorRT-LLM kernel rewrites, and optimized flash-decoding algorithms that minimize redundant VRAM round-trips.

Benchmark Evaluation & Rigorous Comparison

Evaluating raw decoding speed requires a strict distinction between per-agent output velocity and aggregate system throughput. Per-agent decoding speed measures the responsiveness and output generation pace of a single concurrent session, directly determining the user-perceived latency of autonomous coding agents or real-time conversational assistants. Aggregate system throughput, conversely, tracks the total concurrent volume of tokens processed across all active sessions across the entire 72-GPU rack.

Independent evaluations on alternative serving frameworks have further validated the extreme scalability of the V4.1 Flash architecture. In separate low-latency InferenceX benchmark runs, Together AI’s specialized compilation stack successfully pushed DeepSeek V4.1 Flash to an astonishing 1,205 tokens per second under targeted low-concurrency workloads. These variations highlight how modern inference compilers can unlock hidden hardware headroom by tuning memory alignment and speculative decoding parameters.

Deployment, VRAM Footprint & Inference Optimization

Deploying a 552-billion parameter model—even with sparse 16-billion parameter activation—demands stringent VRAM management and high-bandwidth interconnect fabric. In the GB300 NVL72 deployment configuration, the rack's unified memory pooling prevents the memory fragmentation issues that plague PCIe-connected clusters. However, enterprise platform engineers must still carefully provision KV-cache allocations to support the model's expansive 1-million-token context window without triggering out-of-memory (OOM) faults during prolonged multi-turn agent sessions.

# Initialize DeepSeek V4.1 Flash on Nvidia GB300 NVL72 Infrastructure
python3 -m vllm.entrypoints.openai.api_server \
    --model deepseek-ai/DeepSeek-V4.1-Flash \
    --tensor-parallel-size 8 \
    --pipeline-parallel-size 9 \
    --max-model-len 1000000 \
    --enable-prefix-caching \
    --kv-cache-dtype fp8_e4m3 \
    --gpu-memory-utilization 0.95 \
    --enforce-eager