Convergence analysis of the last iterate of gradient methods for smooth convex-concave saddle point problems

Abstract

Generative adversarial networks(GANs), which have received great attention, is formulated as a min-max problem, and developing an optimizer that can solve such problem efficiently is an important challenge. Therefore, this paper studies gradient-type algorithms for smooth convex-concave saddle point problems. The paper aims to find an algorithm that converges faster and analyzes convergence rates of an algorithm such as EG that solves a non-converging problem in the smooth convex-concave problem of GDA used for GANs. We study an inexact Halpern iteration using the restarting accelerated forward method(IHI-RAF), a variant of Kim's accelerated proximal point method with a restarting technique. We analyze the worst-case rates of the last iterate of existing methods such as EG and OGDA and compare the number of operators used to obtain an approximated solution with IHI-RAF. Our results using the performance estimation problem approach and potential function approach complements the existing analysis of the last iterate of EG that requires an additional Lipschitz derivative condition. Our numerical experiments illustrate that IHIRAF is faster than EG and OGDA for some smooth convex-concave problems.