In the last video, we learned how gradient descent works for the case of a single neural network. Then, we wondered how gradient descent should work for feedforward neural networks that have many layers. If you use such networks, we need to train adjustable parameters in these networks. But large networks may have many layers of neurons, with up to hundreds of thousands, or even millions of parameters. To make gradient descent work in practice in such settings, we need some sort of numerical optimization of calculation of that many gradients. Such algorithm called back-propagation that allows gradient descent to work efficiently with large neural networks was suggested in 1986 in a groundbreaking paper by Rumelhart, Hinton and Williams. As its name suggests, backpropagation works backwards from outputs to the inputs using the channel of derivative recursively. This may already sound familiar to you from our previous video about the TensorFlow, and how it implements the reverse-mode autodiff for automatic calculation of derivatives of arbitrary functions. And if it does, it does it for the right reason, because backpropagation is exactly gradient descent where all derivatives are computed, used a reverse model autodiff method. To see how it works in details, let's recall the working of the reverse-mode autodiff and TensorFlow. In our video on TensorFlow, we saw how it works on a simple example of a function of two variables X and Y. The main idea there was a combination of a forward and backward pass and a reliance on a chain rule for calculation of derivatives. Now, let's see how essentially the same method works to calculate gradients or for a neural network, with respect to all of its parameters. Assume that we minimize a mean squared expected loss for a train set as we did for linear regression. But this time, the function Y hat of W is given by the final node, fN of some neural network. For example, we might have a neural network with two hidden layers of this type. We can schematically write the output fN of such neural network as a compound function shown here. The best way to understand this formula is from outputs to the inputs. First function fN depends on parameters WN, and on inputs to this function. But the inputs to fN, are given by outputs of the previous layer, which is denoted f sub N, minus one here. This function depends on its own vector of parameters WN, W sub N minus one, and so on. Now, let's see how the chain rule works backwards from the top of the network to compute all derivatives recursively. Let's start with the last node fN. There are two weights, W sub N one, and W sub N two, that enter the final node fN of this network. Computing the gradient with respect to either weight W sub I, where I equals one or two is done as a straightforward application of the chain rule. So we have the derivative of the lost function f, With respect to W sub Ni, is given by a product of the derivative of N with respect to the output fN of the last node, times the derivative of the last node with respect to weight, W sub Ni. Let's write it as a product of delta N, times the derivative of fN with respect to the input weight, W sub Ni, where Delta N is the derivative of the lost function f with respect to the output of node fN. In this expression, the term delta N depends on the laws observed for this set of parameters, while the second factor does not depend on the laws but only depends on the form or function fN and weights used there. Now let's continue going backwards and consider the last hidden layer, L sub N minus one. For derivatives with respect to weights W sub N minus one, that specify functions fN minus one here, we again apply the chain rule to express it as a product of these three derivatives. First, we take the derivative of N with respect to fN, then the derivative of fN with respect to f sub N minus one, comma I, and then finally, the derivative of f sub N minus one, with respect to weights W sub N minus one for i equal one or two. But let's note a nice trick here. Let's denote these product of directive f with respect to fN and fN with respect to f sub N minus one, as delta sub N minus one comma i. As we already computed delta sub N, then we can easily evaluate the expression for delta sub N minus one. Now, know that by introducing delta sub N minus one as we just did that derivatives of the lost f with respect to weights W sub N minus one, have exactly the same form as in the previous expression with the only difference that now we have index and minus one, instead of N. So, instead of the error delta N at the last layer, we have a new error delta sub N minus one, made of delta N and the derivative of fN, with respect to f sub N minus one. In other words, that error, delta N has backpropagated to level N minus one from the last layer LN, and became delta sub N minus one. Otherwise, calculation of gradients of the final lost function with respect to weights of the lost human layer is the same as for the very last layer. Now, we are already starting to see a pattern here. Let's see how we can continue such recursive calculation for the layer L sub N minus two. If we want to calculate the derivative of the lost function with respect to weight WN sub N minus two comma two, we have to include both nodes at the higher level L sub N minus one. These are nodes shown here. So the derivative will include some of two terms, each having a product of four derivatives in it. But we already know that this product is equal to delta sub N minus one. That's this. So, the whole expression can be written again in a very similar form to the previous expression as a product of delta sub N minus two times these derivative. Again, the error for this layer is given by linear combination of delay of the error from the previous layer times those derivatives. These calculation can be continued for all levels and all weights over a neural network. This produces a fast recursive calculation of all derivatives with respect to all weights. Once they are all calculated, we can use the gradients to perform one step of gradient descent. For the next step, the whole procedure is repeated again until conversions. So, we saw in this video that backpropagation of the training course provides the most practical approach to using gradient descent with neural network that might have a very large number of weight parameters. Now, if we think of a software implementation of backpropagation in terms of a computational graph, made of nodes representing weights and activation functions of a neural network, then we immediately realized that in TensorFlow, it's actually available to us via TensorFlow's autodiff functionality. Now, we are almost ready to start playing with neural nets, gradient descent, and backpropagation, all in TensorFlow. Only one step remains, which is to see how it works for real world data sets that tend to be large. It turns out that the version of Gradient Descent method quotes the casting gradient descent is best suited for such tasks. Let's see how it works in the next video.
