Faculty Talk: Sports Analytics Since Moneyball

I think I'm going to talk about something a little bit different than the title that you were given which had to do with sports analytics and since Moneyball and I thought well talking about analytics at this late stage would not be that exciting. So my suggestion is if you want to learn about analytics in sports watch the movie Moneyball. It's hard to imagine how a movie about analytics was like nominated for Academy Award as best picture. And I think it has something to do with having Brad Pitt as the star. So, um I'm instead going to talk about the interface between analytics and psychology and how psychology is important even if you're doing something highly quantitative like analytics. So, I wanted to do an experiment with the audience here. If you go to this uh QR code, it's going to ask you to answer two questions. And I want you to come up with lower and upper bounds for the answer to this question. Suppose the question was, how many miles is it between here and the moon? Well, nobody knows what that is. I don't want you to use Google. the internet. I want you to just think and I want you to come up with an interval that's going to be right nine out of 10 times. So if you say ah between here and the moon it's at least one mile and it's less than 10 million. Well that's going to cover it 100% of the time but it's so wide as to be completely worthless. So try and make a reasonable interval that you think will answer those two questions. How old was Martin Luther King Jr. when he was killed? And I doubt anybody knows about the gestation period of an Asian elephant but that's the point. Nobody knows this. you try and come up with your best guess and you put a lower bound and an upper bound that you think nine out of 10 times it'll be right. Uh so I'll give you a few seconds to do that and I will go to the app here. Boom. There we go. And let's see is can somebody raise their hand if they've done this successfully? Got a few. Oh, we got a plenty of hands going up. Okay. So, let's see what the answers are. So, I'm going to go to accuracy. And what answer would we expect here? We would expect 90% of the intervals that you came up with to contain the true answer. For the Martin Luther King question, 43% of you got it right. The true answer was 39 years. for the gestation period of an Asian elephant. You were asked to come up with an interval that nine out of 10 times would cover and 20% of you got it right. I've done this before in my classes and those numbers are typical. You are not unique. That's exactly what I would have expected. And um on question one, you can see everybody's responses and what covered and what didn't. And what one of the things? Well, let me go to question two here. Um, the right answer was 645 days. And the people that got it wrong had a width that was on average 120 days. The people that got it right, they said, "I really have no idea. " Their width was 380 days. Um and again that's fairly typical. Okay. So what do we learn about this? This experiment has been run over and over again. And what we find is 99% of people are overconfident in their abilities. And that's really important even when you're doing something quantitative like analytics. Data scientists are often overconfident in their models, overconfident in the conclusions, and overconfident in the recommendations. And you have to take everything with a big grain of salt. Just like some of the people we're talking about today, when AI tells you something, it doesn't mean it's right. It can be highly confident and can sound really amazing and it can be exactly wrong. So, let's um go on to another question for the audience here. There are two sequences there. One was generated by a random number generator, somebody basically flipping a coin 50/50. And another sequence was generated by a human trying to mimic randomness. So which one do you think is the random sequence? Sequence one or sequence two? How many people think it's sequence one that was the random one? A lot of hands. How many people think it was sequence two? And not quite as many hands. Sequence one is correct. And a lot of

people have figured this out. Why is sequence one the really random one? Because when you do these random coin flips, you will occasionally get streaks. And there you can see there is five X's in a row and four zeros in a row. And many people when they see that would say there's no way this was a valid coin flip. Somebody must have made that up. In fact, that's wrong. Sequence one was the random one. And people in their mind think randomness should be closer to alternating and you shouldn't get too many heads in a row or too many tails in a row. So people mistakenly believe that true randomness should avoid runs. So this led to some research in the 1980s. Are streaks real or they just luck? And you can imagine this is important in sports, but it's also important in investing. When you're seeing a stock price going up, is it going to continue because it's got this momentum or is it going to reverse? Well, some very famous researchers at uh Cornell and uh this is like Nobel Prize winning stuff, Amos Tverki um published a very famous paper, the hot-handed basketball on the misperception of random sequences. And they had two claims in their paper. The first claim was when they analyzed a lot of sports data, they said there is no hot hand. It does not exist. I've asked many people that have played sports, whatever, does the hot hand exist? You ask athletes, you ask coaches, whatever, they all raise their hand. Yes, the hot hand exists. This was a controversial claim because they said it's not true. We looked at the data. There is no such thing as the hot hand. Claim two is people mispersceive random sequences. Okay, you put those together and their conclusion is the hot hand, one of the core beliefs in sports is a myth. Well, it turns out of these two claims, one was tested over and over again. That's claim one. And for 20 years, people reproduced the results that said the hot hand does not exist. But in fact they made a statistical error and everybody for 20 years made the same statistical error and in fact they were wrong in the data. You could go back to the old data and you can find the hot hand because they made a mistake. So what about the second claim? Well that appears to be true and it's been replicated many times. So there's related things called the gamblers's fallacy. Have you ever gone to play roulette wheel and seen red come up four times in a row? Red, red. What are you going to do? You're going to bet on black. It is so ingrained in our brains that if you see red, red, you, oh, it's got to balance out. We know it's 50/50. People will bet on black even though it's as close to independent as possible. And that's sort of related to the hot hand. It's like people misperceive randomness. And that's been shown over and over again. Spotify has a shuffle feature and when they rolled it out it takes your playlist and randomizes the songs you hear. People complained that this can't be random because I heard Billy Joel twice in a row. It wasn't random. So Spotify scrapped their random number generation algorithm and made it non-random so people would think it was random. Okay. So you guys all raised your hand. You could quickly and accurately identify what sequence was random and what sequence wasn't. We talked about how people can misperceive randomness. So you know what the flaws and the pitfalls are. So now I want to ask you to do a simple demonstration here which is at a website called the road to Lissa. All you need to do is pretend in your mind that you're flipping a coin. And you're going to do this by hitting the left and right arrow keys on the computer. The computer's going to try and guess your next choice. How accurate should the computer be? 50%. Jing, you want to come up and be the uh the guinea pig here? All you have to do is flip the coin. And where did my phone go? Um, let me show you where it is. And in order to move this along, you have 30 seconds. I'm going to time this. Okay? — And you can't like think slow. You got to do that a lot. So, you just go left and right and go whenever you want. — Okay. Go fast. Left, right, left, right. Okay. So, um I notice you're not looking at the Oh, you can see it on the screen here. That's 10 seconds. Um, the idea is try and get that down to 50%.

20 seconds and stop. So, the computer guessed you what you're going to do 65% of the time. All you have to do is randomly, you know, come up with something. Uh, Matan, you want to give this a try? Thank you, Ching. So, I want you to know that 65% is not bad. Um, are you ready to go? — Ready. — Go. All right. 30 seconds. All you have to do is flip a coin in your head. Left, right, whatever you want to do. Maybe I shouldn't have asked Maton. All right, 15 seconds. [cheering] — I lost it. Reset. — Did you reset it? — I guess it reset eventually. — Oh, you know. No, it doesn't reset. You might have hit a reset. Okay. All right. — Keep going a little bit more and we'll stop it somehow. Okay. Good. All right. Stop there. That was 30 seconds. 60%. So better. We don't know what reset. Let's try it just one more time. Marilena, you want to give it a chance? Thank you, Maton. — So, fast, — fast, left, right. Those two keys there. — Let me know when. — Go whenever you want. Okay. Wow. Really good. — Okay, that's 15 seconds. — Damn it. What? — Ew. I hate this. — All right. 30 seconds. Stop. Okay. So, what was it doing as you were trying to be random? It's building a tree of how many times you go left and then right and then left and the right that you can't and it's using that to make a prediction about what you're going to do next. So at the beginning 5050 and then as it learns the patterns that are ingrained in your brain that you don't even know are there it could predict 85% of your next moves which is pretty amazing. So, it's um I just wanted to quit here and uh last slide. We're bad at understanding randomness for these reasons, but it's important in real life and certain for analytics folks and data scientists to understand how uh psychology can affect even the quantitative sciences. So, thank you very much.