Derivatives, Gradients, Jacobians and Hessians

Summary

This article explains how these four things fit together and shows some examples of what they are used for. Derivatives Derivatives are the most fundamental concept in calculus. If you have a function, a derivative tells you how much that function changes at each point. If we start with the function , we can calculate the derivative as . Here are those two functions graphed. One use of derivatives is for optimization – also known as finding the lowest part on a graph. If you were at and wanted to know whether you should go left or right to get lower, the derivative can tell you. Plugging 1 into gives the value -4. A negative derivative means taking a step to the right will make the y value go down, so going right is down hill. We could take a step to the right and check the derivative again to see if we’ve walked far enough. As we are taking steps, if the derivative becomes positive, that means we went too far and need to turn around, and start going left. If we shrink our step size whenever we go too far in either direction, we can get arbitrarily close to the actual minimum point on the graph. What I just described is an iterative optimization method that is similar to gradient descent. Gradient descent simulates a ball rolling down hill to find the lowest point that we can, adjusting step size, and even adding momentum to try and not get stuck in places that are not the true minimum. We can make an observation though: The minimum of a function is flat, and has a derivative of 0. If not, that would mean it was on a hill, which means that going either left or right is lower, so it wouldn’t be the minimum. Armed with this knowledge, another way to use derivatives to find the minimum is to find where the derivative is 0. We can do that by solving the equation and getting the value . Without iteration, we found that the minimum of the function is at and we can plug 3 into the original equation to find out that the minimum y value is 4. Things get more complicated when...