Dying ReLU and Initialization: Theory and Numerical Examples

Cited 36 time in webofscience Cited 0 time in scopus
  • Hit : 45
  • Download : 0
The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die. However, little is known about its theoretical analysis. In this paper, we rigorously prove that a deep ReLU network will eventually die in probability as the depth goes to infinite. Several methods have been proposed to alleviate the dying ReLU. Perhaps, one of the simplest treatments is to modify the initialization procedure. One common way of initializing weights and biases uses symmetric probability distributions, which suffers from the dying ReLU. We thus propose a new initialization procedure, namely, a randomized asymmetric initialization. We show that the new initialization can effectively prevent the dying ReLU. All parameters required for the new initialization are theoretically designed. Numerical examples are provided to demonstrate the effectiveness of the new initialization procedure.
Publisher
GLOBAL SCIENCE PRESS
Issue Date
2020-11
Language
English
Article Type
Article
Citation

COMMUNICATIONS IN COMPUTATIONAL PHYSICS, v.28, no.5, pp.1671 - 1706

ISSN
1815-2406
DOI
10.4208/cicp.OA-2020-0165
URI
http://hdl.handle.net/10203/297250
Appears in Collection
MA-Journal Papers(저널논문)
Files in This Item
There are no files associated with this item.
This item is cited by other documents in WoS
⊙ Detail Information in WoSⓡ Click to see webofscience_button
⊙ Cited 36 items in WoS Click to see citing articles in records_button

qr_code

  • mendeley

    citeulike


rss_1.0 rss_2.0 atom_1.0