This study investigates how candidate-generation strategies influence the efficiency and performance of test-time scaling in large language models (LLMs). While increasing the number of candidates (N) improves reasoning accuracy—evidenced by gains of 8.4 percentage points for Phi-3-mini and 18.4 points for Qwen2.5-1.5B on GSM8K—the authors argue that N alone is insufficient to characterize computational costs. By fixing N at 8 and comparing four scheduling configurations (1x8, 2x4, 4x2, and 8x1) on A100 GPUs, the researchers analyzed latency, throughput, GPU-hours, and gross GPU-device energy. The results demonstrate that serial execution (8x1) is significantly less efficient than batched execution (1x8), consuming 4.64x to 4.86x more gross GPU energy and incurring 5.77x to 6.12x higher P95 latency. These findings, validated across multiple A100 nodes and short-output SciQ/V100 experiments, indicate that when candidates are independent and memory allows, larger batch sizes are superior for minimizing system costs. Consequently, the authors suggest that evaluations of test-time scaling must report not only accuracy and candidate counts but also specific generation schedules and GPU-level system metrics to provide a complete picture of efficiency.
Read original
huggingface/daily-papers