Deployment field notes
Practical notes on GPU memory, quantization, multi-GPU serving, and reproducible model benchmarks.
Prefix caching: why the second LLM request can look faster
Separate cold and warm prefix-cache results, design a realistic cache test, and avoid misleading time-to-first-token comparisons.
3 min readMulti-GPU inference: tensor parallelism or more replicas?
Choose between splitting one model across GPUs and running independent replicas. Understand memory fit, interconnects, and what to benchmark.
3 min readMoE GPU requirements: active parameters vs. total weights
Why a mixture-of-experts model's active parameter count is not its GPU memory requirement, and how to compare sparse and dense deployments.
3 min readKV cache and context length: why a model runs out of VRAM
Understand how context length and concurrent requests affect KV cache memory, with a worked example and a practical out-of-memory diagnosis.
3 min read4-bit vs. 8-bit LLM inference: what actually changes?
Compare 4-bit, 8-bit, and 16-bit model weights, quantization compatibility, and a practical quality test before choosing a GPU.
3 min readA good benchmark shows its working.
A deployment benchmark checklist: model revision, hardware, runtime, request distribution, caching, and reproducibility.
4 min readFast for one user. Fast for everyone.
Understand time to first token, inter-token latency, and system throughput before comparing inference results.
4 min readHow much GPU memory do you actually need?
A practical guide to model weights, KV cache, and the memory you need beyond the parameter count.
5 min read