DC Field | Value | Language |
---|---|---|
dc.contributor.author | Kim, Taehyeon | ko |
dc.contributor.author | Yun, Se-Young | ko |
dc.date.accessioned | 2022-08-25T08:01:23Z | - |
dc.date.available | 2022-08-25T08:01:23Z | - |
dc.date.created | 2022-08-25 | - |
dc.date.created | 2022-08-25 | - |
dc.date.issued | 2022 | - |
dc.identifier.citation | IEEE ACCESS, v.10, pp.69741 - 69749 | - |
dc.identifier.issn | 2169-3536 | - |
dc.identifier.uri | http://hdl.handle.net/10203/298114 | - |
dc.description.abstract | Recent research in deep Convolutional Neural Networks(CNN) faces the challenges of vanishing/exploding gradient issues, training instability, and feature redundancy. Orthogonality Regularization(OR), which introduces a penalty function considering the orthogonality of neural networks, could be a remedy to these challenges but is surprisingly not popular in the literature. This work revisits the OR approaches and empirically answer the question: Even when comparing various regularizations like weight decay and spectral norm regularization, which is the most powerful OR technique? We begin by introducing the improvements of various regularization techniques, specifically focusing on OR approaches over a variety of architectures. After that, we disentangle the benefits of OR in the comparison of other regularization approaches with a connection on how they affect norm preservation effects and feature redundancy in the forward and backward propagation. Our investigations show that Kernel Orthogonality Regularization(KOR) approaches, which directly penalize the orthogonality of convolutional kernel matrices, consistently outperform other techniques. We propose a simple KOR method considering both row- and column- orthogonality, of which empirical performance is the most effective in mitigating the aforementioned challenges. We further discuss several circumstances in the recent CNN models on various benchmark datasets, wherein KOR gains more effectiveness. | - |
dc.language | English | - |
dc.publisher | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC | - |
dc.title | Revisiting Orthogonality Regularization: A Study for Convolutional Neural Networks in Image Classification | - |
dc.type | Article | - |
dc.identifier.wosid | 000838367900001 | - |
dc.identifier.scopusid | 2-s2.0-85133615093 | - |
dc.type.rims | ART | - |
dc.citation.volume | 10 | - |
dc.citation.beginningpage | 69741 | - |
dc.citation.endingpage | 69749 | - |
dc.citation.publicationname | IEEE ACCESS | - |
dc.identifier.doi | 10.1109/ACCESS.2022.3185621 | - |
dc.contributor.localauthor | Yun, Se-Young | - |
dc.description.isOpenAccess | N | - |
dc.type.journalArticle | Article | - |
dc.subject.keywordAuthor | Training | - |
dc.subject.keywordAuthor | Kernel | - |
dc.subject.keywordAuthor | Convolutional neural networks | - |
dc.subject.keywordAuthor | Redundancy | - |
dc.subject.keywordAuthor | Artificial intelligence | - |
dc.subject.keywordAuthor | Licenses | - |
dc.subject.keywordAuthor | Signal to noise ratio | - |
dc.subject.keywordAuthor | Deep neural network (DNN) | - |
dc.subject.keywordAuthor | kernel | - |
dc.subject.keywordAuthor | orthogonality regularization | - |
dc.subject.keywordAuthor | convolutional neural network (CNN) | - |
dc.subject.keywordAuthor | regularization | - |
dc.subject.keywordAuthor | image classification | - |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.