GPU-Accelerated Parallel Stochastic Gradient Descent

This paper studies a simplified instance of the design question raised by GPU A-SGD: how should stochastic optimization be organized so that device-level parallelism improves throughput without destabilizing convergence? Motivated by the asynchronous multi-GPU training framework of Paine et al., we examine a controlled linear-regression setting in which workers process disjoint data shards, compute stochastic gradient updates in parallel, and combine local models through either plain averaging or a lightweight asynchronous consensus rule. The resulting study isolates three effects that matter in larger systems as well: the runtime crossover between CPU and GPU execution, the point at which GPU utilization saturates despite transfer overhead, and the robustness benefit of limited cross-worker coordination when data variability increases. The measurements show a clear scale threshold beyond which GPU execution becomes favorable, near-saturated utilization on modestly sized workloads, and consistently lower root-mean-square error for the asynchronous variant in higher-variance regimes. These findings support the broader claim of the GPU A-SGD literature: parallel stochastic optimization is most effective when compute throughput and communication discipline are designed together.