What is Optimization? + Learning Gradient Descent | Two Minute Papers #82

dear fellow Scholars this is 2 minute papers with caroon today we're not going to have the usual visual fireworks that we had with most topics in computer Graphics but I really hope you'll still find this episode enjoyable and stimulating this episode is also going to be a bit heavy on what optimization is and we'll talk a little bit at the end about the intuition of the paper itself we are going to talk about mathematical optimization this term is not to be confused with the word optimization that we use in our everyday lives for instance improving the efficiency of a computer code or a workflow this kind of optimization means finding one hopefully optimal solution from a set of possible candidate Solutions an optimization problem is given the following way one there's a set of variables we can play with and two there's an objective function that we wish to minimize or maximize well this Probably sounds great for mathematicians but for everyone else maybe this is a bit confusing let's build a better understanding of this concept through an example for instance let's imagine that we have to cook a meal for our friends from a given set of ingredients the question is how much salt vegetables and meat goes into the pan these are our variables that we can play with and the goal is to choose the optimal amount of these ingredients to maximize the taste tastiness of the meal tastiness will be our objective function and for a moment we shall pretend that tastiness is an objective measure of a meal this was just one toy example but the list of applications is endless in fact optimization is so incredibly ubiquitous there's hardly any field of science where some form of it is not used to solve difficult problems for instance if we have the plan of a bridge we can ask it to tell us the minimal amount of building materials we need to build it in a way that it remains stable we can also optimize the layout of the bridge itself to make sure the inner tension and compression forces line up well a big part of deep learning is actually also an optimization problem there are a given set of neurons and the variables are when they should be activated and we are fiddling with these variables to minimize the output error which can be for instance our accuracy in guessing whether a picture depicts a muffin or a Chihuahua the question for almost any problem is usually not whether it can be formulated as an optimization problem but whether it is worth it and by worth it I mean the question whether we can solve it quickly and reliably an Optimizer is a technique that is able to solve these optimization problems and offer us a hopefully satisfactory solution to them there are many algorithms that excel at solving problems of different complexities but what ties them together is that they are usually handcrafted techniques written by really smart mathematicians gradient descent is one of the simplest optimization algorithms where we change each of the variables around the bit and as a result see if the objective function changes favorably after finding a direction that leads to the most favorable changes we shall continue our journey in that direction what does this mean in practice intuitively in our cooking example after making several meals we would ask our guests about the tastiness of these meals from their responses we would recognize that adding a bit more salt led to very favorable results and since these people are notorious meat eaters decreasing the amount of vegetables and increasing the meat content also led to favorable reviews and we of course on the back of this new found knowledge will cook more with these variable changes in pursuit of the best possible meal in the history of mankind this is something that is reasonably close to what gradient descent is in mathematics a slightly more sophisticated version of gradient descent is also a very popular way of training neural networks if you have any questions regarding the gradient part we had an extended 2minute papers episode on what gradients are and how to use them to build an awesome algorithm for light transport it is available where well of course in the video description box caroy why are you even asking so what about the paper part this incredible new work of Google Deep Mind shows that an optimization algorithm itself can emerge as a result of learning an algorithm itself is not considered the same one thing as deciding what an image depicts or how we should grade a student essay it is an algorithm a sequence of steps we have to take if we are talking about outputting sequences we'll definitely need to use a

recurrent neural network for that their proposed learning algorithm can create new optimization techniques that outperform previously existing methods not everywhere but on a set of specialized problems I hope youve enjoyed the journey we'll talk quite a bit about optimization in the future you'll love it thanks for watching and for your generous support and I'll see you next time