Scaling Neural Tangent Kernels via Sketching and Random Features

Cited 0 time in webofscience Cited 0 time in scopus
  • Hit : 423
  • Download : 0
DC FieldValueLanguage
dc.contributor.authorZandieh, Amirko
dc.contributor.authorHan, Insuko
dc.contributor.authorAvron, Haimko
dc.contributor.authorShoham, Netako
dc.contributor.authorKim, Chaewonko
dc.contributor.authorShin, Jinwooko
dc.date.accessioned2021-12-09T06:47:59Z-
dc.date.available2021-12-09T06:47:59Z-
dc.date.created2021-12-02-
dc.date.issued2021-12-07-
dc.identifier.citation35th Conference on Neural Information Processing Systems, NeurIPS 2021-
dc.identifier.urihttp://hdl.handle.net/10203/290291-
dc.description.abstractThe Neural Tangent Kernel (NTK) characterizes the behavior of infinitely-wide neural networks trained under least squares loss by gradient descent. Recent works also report that NTK regression can outperform finitely-wide neural networks trained on small-scale datasets. However, the computational complexity of kernel methods has limited its use in large-scale learning tasks. To accelerate learning with NTK, we design a near input-sparsity time approximation algorithm for NTK, by sketching the polynomial expansions of arc-cosine kernels: our sketch for the convolutional counterpart of NTK (CNTK) can transform any image using a linear runtime in the number of pixels. Furthermore, we prove a spectral approximation guarantee for the NTK matrix, by combining random features (based on leverage score sampling) of the arc-cosine kernels with a sketching algorithm. We benchmark our methods on various large-scale regression and classification tasks and show that a linear regressor trained on our CNTK features matches the accuracy of exact CNTK on CIFAR-10 dataset while achieving 150x speedup.-
dc.languageEnglish-
dc.publisherNeural Information Processing Systems-
dc.titleScaling Neural Tangent Kernels via Sketching and Random Features-
dc.typeConference-
dc.type.rimsCONF-
dc.citation.publicationname35th Conference on Neural Information Processing Systems, NeurIPS 2021-
dc.identifier.conferencecountryUS-
dc.identifier.conferencelocationVirtual-
dc.contributor.localauthorShin, Jinwoo-
dc.contributor.nonIdAuthorZandieh, Amir-
dc.contributor.nonIdAuthorHan, Insu-
dc.contributor.nonIdAuthorAvron, Haim-
dc.contributor.nonIdAuthorShoham, Neta-
dc.contributor.nonIdAuthorKim, Chaewon-
Appears in Collection
RIMS Conference Papers
Files in This Item
There are no files associated with this item.

qr_code

  • mendeley

    citeulike


rss_1.0 rss_2.0 atom_1.0