In this episode, we discuss the bane of many machine learning algorithms - overfitting. It is also explained why it is an undesirable way to learn and how to combat it via L1 and L2 regularization.
_____________________________
The paper "Regression Shrinkage and Selection via the Lasso" is available here:
http://statweb.stanford.edu/~tibs/lasso/lasso.pdf
Andrej Karpathy's excellent lecture notes on neural networks and regularization:
http://cs231n.github.io/neural-networks-1/
The neural network demo is available here:
http://cs.stanford.edu/people/karpathy/convnetjs/demo/classify2d.html
A playlist with out neural network and deep learning-related videos:
https://www.youtube.com/playlist?list=PLujxSBD-JXglGL3ERdDOhthD3jTlfudC2
WE WOULD LIKE TO THANK OUR GENEROUS SUPPORTERS WHO MAKE TWO MINUTE PAPERS POSSIBLE:
Sunil Kim, Vinay S.
Subscribe if you would like to see more of these! - http://www.youtube.com/subscription_center?add_user=keeroyz
The thumbnail image background was created by Tony Hisgett (CC BY 2.0). It has undergone recoloring. - https://flic.kr/p/5dkbNV
Splash screen/thumbnail design: Felícia Fehér - http://felicia.hu
Károly Zsolnai-Fehér's links:
Patreon → https://www.patreon.com/TwoMinutePapers
Facebook → https://www.facebook.com/TwoMinutePapers/
Twitter → https://twitter.com/karoly_zsolnai
Web → https://cg.tuwien.ac.at/~zsolnai/
Оглавление (1 сегментов)
Segment 1 (00:00 - 04:00)
Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. In machine learning, we often encounter classification problems where we have to decide whether an image depicts a dog or a cat. We'll have an intuitive, but simplified example where we imagine that the red dots represent dogs, and the green ones are the cats. We first start learning on a training set, which means that we get a bunch of images that are points on this plane, and from these points we try to paint the parts of the plane red and green. This way, we can specify which regions correspond to the concept of dogs and cats. After that, we'll get new points that we don't know anything about, and we'll ask the algorithm, for instance, a neural network to classify these unknown images, so it tells us whether it thinks that it is a dog or a cat. This is what we call a test set. We have had a lots of fun with neural networks and deep learning in previous Two Minute Papers episodes, I've put some links in the description box, check them out! In this example, it is reasonably easy to tell that the reds roughly correspond to the left, and the greens to the right. However, if we just jumped on the deep learning hype train, and don't know much about a neural networks, we may get extremely poor results like this. What we see here is the problem of overfitting. Overfitting means that our beloved neural network does not learn the concept of dogs or cats, it just tries to adapt as much as possible to the training set. As an intuition, think of poorly made real-life exams. We have a textbook where we can practice with exercises, so this textbook is our training set. Our test set is the exam. The goal is to learn from the textbook and obtain knowledge that proves to be useful at the exam. Overfitting means that we simply memorize parts of the textbook instead of obtaining real knowledge. If you're on page 5, and you see a bus, then the right answer is B. Memorizing patterns like this, is not real learning. The worst case is if the exam questions are also from the textbook, because you can get a great grade just by overfitting. So, this kind of overfitting has been a big looming problem in many education systems. Now the question is, which kind of neural network do we want? Something that works like a lazy student, or one that can learn many complicated concepts. If we're aiming for the latter, we have to combat overfitting, which is the bane of so many machine learning techniques. Now, there's several ways of doing that, but today we're going to talk about one possible solution by the name L1 and L2 regularization. The intuition of our problem is that the deeper and bigger neural networks we train, the more potent they are, but at the same time, they get more prone to overfitting. The smarter the student is, the more patterns he can memorize. One solution is to hurl a smaller neural network at the problem. If this smaller version is powerful enough to take on the problem, we're good. A student who cannot afford to memorize all the examples is forced to learn the actual underlying concepts. However, it is very possible that this smaller neural network is not powerful enough to solve the problem. So we need to use a bigger one. But, bigger network, more overfitting. Damn. So what do we do? Here is where L1 and L2 regularization comes to save the day. It is a tool to favor simpler models instead of complicated ones. The idea is that the simpler the model is, the better it transfers the textbook knowledge to the exam, and that's exactly what we're looking for. Here you see images of the same network with different regularization strengths. The first one barely helps anything and as you can see, overfitting is still rampant. With a stronger L2 regularization, you see that the model is simplified substantially, and is likely to perform better on the exam. However, if we add more regularization, it might be that we simplified the model too much, and it is almost the same as a smaller neural network that is not powerful enough to grasp the underlying concepts of the exam. Keep your neural network as simple as possible, but not simpler. One has to find the right balance which is an art by itself, and it shows that training deep neural networks takes a bit of expertise. It is more than just a plug and play tool that solves every problem by magic. If you want to play with the neural networks you've seen in this video, just click on the link in the description box. I hope you'll have at least as much fun with it as I had! Thanks for watching, and for your generous support, and I'll see you next time!
