vanishing gradient problem

Efficient gradient propagation: No vanishing gradient problem or exploding effect.

Many factors contribute to the slow speed, one being the vanishing gradient problem analyzed in 1991 by Sepp Hochreiter.

In machine learning, the vanishing gradient problem is a difficulty found in training artificial neural networks with gradient-based learning methods and backpropagation.

Sepp Hochreiter's diploma thesis of 1991 formally identified the reason for this failure as the vanishing gradient problem, which affects many-layered feedforward networks and recurrent neural networks.

The long short term memory (LSTM), developed by Hochreiter and Schmidhuber in 1997, is an artificial neural net structure that unlike traditional RNNs doesn't have the vanishing gradient problem.

Hardware advances have meant that from 1991 to 2015, compute power (especially as delivered by GPUs) has increased around a million-fold, making standard backpropagation feasible for networks several layers deeper than when the vanishing gradient problem was recognized.

In 2010, Dan Ciresan and colleagues in Jürgen Schmidhuber's group at the Swiss AI Lab IDSIA showed that despite the above-mentioned "vanishing gradient problem," the superior processing power of GPUs makes plain back-propagation feasible for deep feedforward neural networks with many layers.