Adaptive Proximal Gradient Methods for Structured Neural Networks

Cited 0 time in webofscience Cited 0 time in scopus
  • Hit : 96
  • Download : 0
DC FieldValueLanguage
dc.contributor.authorYun, Jihunko
dc.contributor.authorAurelie, Lozanoko
dc.contributor.authorYang, Eunhoko
dc.identifier.citation35th Conference on Neural Information Processing Systems, NeurIPS 2021-
dc.description.abstractWe consider the training of structured neural networks where the regularizer can be non-smooth and possibly non-convex. While popular machine learning libraries have resorted to stochastic (adaptive) subgradient approaches, the use of proximal gradient methods in the stochastic setting has been little explored and warrants further study, in particular regarding the incorporation of adaptivity. Towards this goal, we present a general framework of stochastic proximal gradient descent methods that allows for arbitrary positive preconditioners and lower semi-continuous regularizers. We derive two important instances of our framework: (i) the first proximal version of \textsc{Adam}, one of the most popular adaptive SGD algorithm, and (ii) a revised version of ProxQuant for quantization-specific regularizers, which improves upon the original approach by incorporating the effect of preconditioners in the proximal mapping computations. We provide convergence guarantees for our framework and show that adaptive gradient methods can have faster convergence in terms of constant than vanilla SGD for sparse data. Lastly, we demonstrate the superiority of stochastic proximal methods compared to subgradient-based approaches via extensive experiments. Interestingly, our results indicate that the benefit of proximal approaches over sub-gradient counterparts is more pronounced for non-convex regularizers than for convex ones.-
dc.publisherNeural Information Processing Systems-
dc.titleAdaptive Proximal Gradient Methods for Structured Neural Networks-
dc.citation.publicationname35th Conference on Neural Information Processing Systems, NeurIPS 2021-
dc.contributor.localauthorYang, Eunho-
dc.contributor.nonIdAuthorAurelie, Lozano-
Appears in Collection
RIMS Conference Papers
Files in This Item
There are no files associated with this item.


  • mendeley


rss_1.0 rss_2.0 atom_1.0