Training neural network through learning the derivative of loss function

Abstract

Recent studies and applications in machine learning shows that neural network using gradient descent-based methods suits the need in many cases. Especially, there have been many researches which talks about the importance of choosing a loss function, when using such methods. While there also have been studies which accepted the concept of meta-learning and substituted the loss function with a computational graph or a neural network, they didn’t question about the necessity of the loss function if they were to be replaced. This paper first looks the structural properties of neural networks. Also, the paper rethinks the role of loss function in neural networks, then gives an idea and method of learning the derivative of the loss function, and finally shows some experimental results on regression.