Nvidia DGX Spark: When Benchmark Numbers Meet Production Reality

Summary

# NVIDIA DGX Spark: When Benchmark Numbers Meet Production Reality **A 6-Day Deep Dive into Real-World ML Performance** --- NVIDIA recently published [benchmarks showcasing the DGX Spark](https://developer.nvidia.com/blog/how-nvidia-dgx-sparks-performance-enables-intensive-ai-tasks/): 82,739 tokens/second for fine-tuning, sub-1% accuracy degradation with FP4, and impressive inference throughput. After spending 6+ days running intensive ML workloads on a DGX Spark—training multiple models from scratch, fine-tuning with LoRA, and conducting extensive inference benchmarks—I can tell you the real story. **The short version:** NVIDIA's numbers are technically accurate. But they don't tell you about the GPU inference crashes, memory fragmentation that requires hard reboots, or the 15 hours I spent debugging "training failures" that turned out to be inference bugs. This is the post I wish I'd read before diving in. ## What NVIDIA Showed Us NVIDIA's blog highlights some impressive numbers: **Fine-Tuning Performance:** - Llama 3.2 3B: **82,739 tokens/sec** (full fine-tuning, BF16) - Llama 3.1 8B: **53,657 tokens/sec** (LoRA, BF16) - Llama 3.3 70B: **5,079 tokens/sec** (QLoRA, FP4) **Inference Performance:** - Qwen3 14B: **5,928 tokens/sec** prompt processing, 22.71 tokens/sec generation - GPT-OSS-20B: **82.74 tokens/sec** generation **Key Claims:** - 1 petaflop of FP4 compute - Less than 1% accuracy degradation with FP4 - 273 GB/sec memory bandwidth - Support for 128GB+ models locally Impressive on paper. Let's see how it holds up. ## My Testing Environment Before we dive into results, here's what I was working with: **Hardware:** - DGX Spark (ARM64 architecture) - GB10 GPU (Blackwell generation) - Driver 580.95.05 - CUDA 13.0 - Ubuntu 24.04.3 LTS **Workloads:** 1. **Inference Benchmark:** Phi-3.5-mini-instruct (3.8B params) via Ollama and llama.cpp 2. **Fine-Tuning:** 7 LoRA experiments on Gemma-3-4b-it for medical Q&A (10,000 examples from PubMedQA) 3. **Training:** NanoCh...