Tighter Lower Bounds for Shuffling SGD: Random Permutations and Beyond

Abstract

We study convergence lower bounds of without-replacement stochastic gradient descent (SGD) for solving smooth (strongly-)convex finite-sum minimization problems. Unlike most existing results focusing on final iterate lower bounds in terms of the number of components 𝑛 and the number of epochs 𝐾, we seek bounds for arbitrary weighted average iterates that are tight in all factors including the condition number 𝜅. For SGD with Random Reshuffling, we present lower bounds that have tighter 𝜅 dependencies than existing bounds. Our results are the first to perfectly close the gap between lower and upper bounds for weighted average iterates in both strongly-convex and convex cases. We also prove weighted average iterate lower bounds for arbitrary permutation-based SGD, which apply to all variants that carefully choose the best permutation. Our bounds improve the existing bounds in factors of 𝑛 and 𝜅 and thereby match the upper bounds shown for a recently proposed algorithm called GraB.