Robot golf vs holes that keep getting harder

In my last video, I made this club which always aims at the hole, and it is really cool. Using it feels like magic, but it has a problem, which is that it always aims directly at the hole because it just doesn't know how to do anything else. And according to my wife, if it can't do bounces and banks, it's not really mini golf. So, I decided to fix that. I made some mini golf holes that I thought were legitimately hard. Like, is this bridge even possible? Or this loop where the ball goes out of the top? How does that even work? Then I sat down and programmed. Now, when you swing the club, it thinks really hard, simulating the physics of all the potential futures, so it can aim the ball to thread the needle and do physically pretty incredible stuff. And doing all this turned out to be a lot easier said than done. This turned into months of programming. If I had known what this was going to take, I really don't think I would have done it because there's just no way that this was worth it. But in the end, we do have something really cool, which if I play my cards right, just might impress my wife. My last video was all about making this club and just getting it working at all, which was really hard. It's basically a handheld robot with a motor up here that can make the head of the club turn really quickly and precisely, which I can use to change the angle that it hits the ball. So, I could be swinging this direction, but the club will hit the ball this direction. And I'm surrounded by a set of motion tracking cameras that can tell the computer very precisely where the club and the ball and the hole and everything is, which it can use to calculate how it needs to aim the club, which works surprisingly well, at least for hitting the ball directly into the hole, because that's as far as I got. So, it's time to take things to the next level. And the first type of hole I'm going to try is a bank shot, which means the robot has to figure out how to bounce the ball off a wall. And it might seem like it's just a simple reflection that would be easy to compute, but it is not. A golf ball bouncing off a wall is way more complicated than you would think. Before the bounce, the ball is rolling toward the wall. And when it hits the wall, the spin makes it start to drive up the wall, which makes it jump into the air. But it's still spinning in the direction it was going before it hit the wall, which makes it move sideways and follow a curved path. It is just a mess. So, I think we're going to have to do more of a physics simulation. And my idea is to get a very accurate model of the hole and the ball and everything on the computer, and then we will virtually hit the ball and see how it bounces and skids and rolls, and then somehow, still working on that part, we will use that to aim the club. The first thing we need is an accurate model of the hole that tells us where everything is. And I can't just use the CAD files for this because I did not make these very accurately. So, we need a way to actually measure it. And I was thinking I was going to need some kind of 3D scanner, but I realized that the motion tracking cameras are basically a 3D scanner if you do it right because they tell you very accurately where these reflective tracking markers are in space. So, if you imagine dragging one of these along the wall and recording the path that it follows, it would give me the shape of the walls. Or if I put it down in the hole and move it around the inside edge, I'd know where the hole is. But we also have to measure the ground because if it's not flat, the ball won't go in a straight line. But where would I move this little ball around to measure that? But it's not actually that bad. This is the worst possible curvature we're going to see. So, if we just measured like here, here, and here, and we assume it's smooth, we have a pretty good idea of what the shape of this is. And we can expand that idea into 2D by measuring a grid instead of a line, which will let us recreate this surface pretty accurately. And if you're interested, this is an application of the Nyquist frequency. The only issue is doing this practically. We can't just drag a marker along the walls. They'd block the view of most of the cameras. You know, it also just wear it down. So, I made this kind of goofy looking thing. There's a set of markers, which is called a constellation, attached to this nice steel ball, which is what we'll drag along the walls and the ground. But this makes a new problem. We want to know where the ball is, but we're measuring these markers, which can be in a lot of different locations for the same ball position. But if you think about it, no matter how we turn it, the ball is always in the same relative position to the constellation, over this way and down a little bit. If we knew how it was offset, we can just add that to the position of the constellation, and it will always give us the location of the ball. It's just kind of a pain to figure out what that relative position is. So, I made this little fixture that holds the ball in a fixed position. And if I take a bunch of measurements of where the constellations are over time, I can calculate the point in space that they're rotating about, which is the center of the ball. And from there, I can easily calculate the relative position of that to the constellation. And then if we put a stick on it, we have a thing that we can just walk

around the hole and scan it out. But there's still a real practical problem with this. The computer is tracking this constellation, and I could be scanning a wall, or I could be carrying this through the air, in which case the computer needs to ignore it, otherwise it's going to think the hole is completely crazy. So, I made this thing, which is sort of an optical trigger. It has a constellation that can pivot. When I pull the trigger, it angles up, which tells the computer to record what I'm doing. Otherwise, it just ignores it. And this works great. There's no electronics or wires going to the computer. So, let's get a hole scanned out and see what it looks like. All right, this is really cool to see. I have the raw probe data, and you can see the path that I traced out all over the turf. But what's really interesting is if I exaggerate the scale, you can see that the hole isn't flat at all, which as far as I can tell is from me just building them stupidly, but the physics engine should take this into account and correct for it. We just have to get this data into the physics engine. And here it is. Looks a little different than the real hole, but everything that matters is in the right spot. And the physics engine I'm using is called MuJoCo. And if you're wondering why I didn't write my own physics engine, I It's basically cuz I don't have 20 years. And here's how it works. You tell it, here's all my geometry, like the walls, the ground, the ball, their physical properties like friction and bounce-iness, and then initial conditions like the club hitting the ball. And then it'll simulate everything that happens. Although, if you don't configure it right, it will give you garbage, which I did many times. All right, I finally got through most of the really obvious stupid issues. Now, the question is if it's accurate. And since we have motion capture, we can use recordings of me actually swinging to swing a virtual club and hit a virtual ball. If the simulation is accurate, the virtual ball will go to the same place that the real ball went. And uh they're not even close. The simulation has a bunch of parameters for friction and bounce-iness. If those don't match reality, the simulation will be wrong. For example, if the friction's too high, the ball will stop early. I don't really know what they should be. I made my best guess, but clearly I'm off the mark. And this is challenging because there's a lot of parameters. Some can be ignored, but I think there's about nine that need to be right. But fortunately, we have a lot of motion capture data showing us where the ball went. So, I'm going to write a program that virtually hits the ball the same way I did and compares where the virtual ball goes to the actual ball. Then it'll tweak the parameters and try again, going through all the different combinations of parameters until it finds the right ones. But there is a problem, which is just the number of combinations of parameters. If we tried 100 possibilities for each of the nine parameters, it would be like a quintillion combinations, which would take my computer many lifetimes to test. And we all know that YOLO, so I don't have enough lifetimes for that. So, we have to be a little bit smarter. And I'm doing what's called a stochastic solver. It's kind of a fancy way of saying that you try things randomly, except it's kind of systematic and smart. And all we have to do now is just let the computer think really hard for a while, and then we'll have our answer. All right, that took most of the day, but my solver spit out some numbers. So, let's plug them in and see how the simulation looks. Okay, I figured it out, and it's actually really pretty funny. For each point that the simulated ball goes to, my program calculates how far it is from the actual path, which it adds up for each point the ball goes to. And a smaller score is better. And it turns out that if the ball barely moves, it never gets very far from the actual path that the ball took, and you get a really good score. So, I just cranked up the friction and won on a technicality. And I never really was expecting my programs to maliciously comply. It's kind of like a genie when you ask it for a million bucks and it gives you a bunch of deer, but it should be pretty easy to fix, at least. It should be a lot better than that. That is not right. Something is wrong here. I do not know what, though. The super-duper stochastic solver has just been kicking my butt. I've spent the last couple weeks trying to get it working. Most of that time has been waiting while it just computes things and then tells me the wrong result. So, I decided to do the manly thing and give up. And instead, I'm going to solve it by hand. So, I made this program which runs simulated hits and shows me a 2D version of what happened. It has a super complicated set

of keyboard shortcuts which let me change all the parameters. Whenever I change one, it reruns the simulation and shows me what happened. And I can use my understanding of what they mean to work my way towards good parameters. It only took like an hour to get some quite good settings and uh that's a whole lot better than I can say for the computer. So, we have the entire hole scanned out and loaded into the physics engine. And when I put this ball down, there's something really important that happens and it's how the physics engine knows how to actually aim the ball so that it goes in. Because right now, the physics engine just tells us where the ball will go for a given hit, not where to aim it to make it go in. To figure that out, we have to take the swing the club is doing and simulate rotating the head to a bunch of different orientations until we find the one that makes the ball go in. And we only have a few milliseconds when the club starts swinging to figure this out. And this is computationally expensive, so there's just not enough time. So, instead, when I set the ball down, the computer does an enormous amount of pre-computation. It simulates all the possible hits that the club could do at all the speeds and angles and it records which ones make the ball go in. So, in the end, it has a list of all the possible scoring shots. It's kind of like when Doctor Strange sits down to consider all the potential futures to find the one where the Avengers succeed. But the downside of this is that it is an enormous amount of computation. It takes forever. It initially was taking around 15 to 20 hours and you have to redo it every time the ball moves. It's a little bit too long to wait between shots, I think. I was able to optimize this quite a bit and I got down to about an hour, but that's still too long to wait. So, I also take all the data and send it over to my beast computer. It uses 100% of the CPU's brain, which is really cool to see, and it gets it done in around 2 to 4 minutes. Which, if you're a serious golfer, that's about the time you'd spend crouched down looking at the green and checking out the break. So, as long as I do that, you wouldn't even know that I have a robot computer actually solving it for me. Well, other than the tracking cameras and the wires and all that. But if you ignore that, you'd never know. So, now when I swing the club, we don't do any physics simulations. We just look at how the club is moving and we compare it to all the simulated hits we already did that made the ball go in. We find the one that's most similar and then we angle the club to match that simulated swing. And if everything works out, it goes in. That is so cool. Got to show the wife this. All right, come look at this. Make sure this is on. Did I capture your amazement? All right, you ready? Hit me. I'm going to hit the ball, not you. Oh, that's better. Yes. Good job. That's it? I don't know what to say. Anyone can do that. Put your money where your mouth is, lady. You know, it can do a lot more, so that this is just the this is the appetizer. Wait till you see what's coming next on the menu. This is the modular bridge system. You can put different size bridges on here to test the accuracy of the club. So, I have a relatively challenging bridge that you might see on a mini golf course. And then I have this bridge which I'm not even sure if it can do. We tried it with a regular putter and it's just impossible. I can't even roll the ball across it with my hand. So, let's see what the club can do, starting with the easier one. All right, so just get up here. I don't even need to really aim. Give her the old swing. It's not supposed to do that. All right, yeah, I think I know. Silly mistake. That doesn't make any sense. You don't make any sense. What are you looking at? My handsome husband. Why don't you just call me in a few weeks once it's — worked out. — It's very repeatably missing. Where are you going? Here's an artist depiction of my life for the last 2 weeks. Put the ball there, wait 4 minutes, swing the club, computer says, "Oh, yeah, aim here and it's going to go right here. " And the ball goes here. Why? After a lot of weeping and gnashing of teeth, I think I found at least part of the problem. My simulation very subtly does not line up with reality. Like if I hold the club here, it should be here in my simulation, but it's actually over a little bit and tilted, which just breaks everything. And I think this is happening because of temperature. Most materials have this usually annoying property, which is that they

get bigger when they get warmer, they get smaller when they get colder. It's called thermal expansion. When it gets warm outside, my entire workshop gets bigger. The tripods that are holding the tracking cameras get longer, which can make things appear to change position in the motion capture. The simplest fix would be just to probe the hole every time I use it, but that just sounds terrible. It would take way too long. So, instead, I'm using these three markers as a reference. The computer will look at where they are using the tracking cameras and compare that to where they are in the simulation. And if they're different, it'll calculate whatever shift or rotation is needed to make them line up. So, now if things get knocked or they drift around thermally, the system should automatically compensate for that. All right. Easier bridge, take two. Medium hole. No problem. It is basically getting a hole-in-one just with more steps. But the next thing is not. This has so little margin for error, I just don't know if the club can do it. Here we go. 3 2 1. Oh, that was so close. Right across the bridge though, that was pretty good. All right, here we go. 3 2 1. Pretty surprising how easy this one ended up being. I was fully prepared to suffer immensely. I guess I kind of perfected the art of going in a straight line. Although sometimes the physics solver will use a bounce off the wall to make it go in. If I had a hat, I would take it off. The bridge hole was designed to test the precision of the club and this is the wavy hole, which is designed to test how well it knows physics. Can it predict how the ball will curve as it goes through these bumps and aim just the right way so that it goes into the hole, which is really hard. I was testing it with the regular club trying to hit it to the same spot and it would go to totally different locations. So, let's see what this robot can do. Okay, I didn't hit that hard enough at all. Wow. That's cool. This might not seem too hard unless you've tried it. It would make a great carnival attraction. It's super sensitive to aim, but also speed. And the robot club misses sometimes, too. Little variations dramatically changes path. I suspect there may be some chaotic behavior that makes it almost unpredictable at lower speeds. But my goodness, to see it pick a line through those hills and bounce it off the wall into the hole is just chef's kiss. I think this is meaningful. This is actually hard. If you're too legit to quit, why don't you put a corner in it? Have it bounce off. Do the bounce to this? Sure. Why not? I think it's a lot harder than you think it is. Am I blowing your mind? I just don't know if we can do it. I have faith in you. So, we have to do it. This is the same thing as the wavy hole, but it's so much worse. Not only do you have to pick the right line do the crazy surface, you have to bounce it just right off the wall. Hey! Well done. I take back what I said about the previous hole. This is the chef that I want to kiss. This hole is so hard because the bounce and the hills are very sensitive. When you stack two of these things in a row, it can add a lot of error. If everything isn't calibrated just right, it's not going to work. But when it works, it's so cool. And I thought I was getting hyped up until I saw my wife. Oh, you think I had the sugar. Whoop ow. When I think mini golf, I think loops. And I took the opportunity to completely over-design something based on the idea that objects want to go in straight lines, not curves. So, with a specially designed hole, we should be able to just extract the ball mid-loop. There's no good reason to make this other than I just really wanted to see it. It seemed awesome. But I can tell you confidently it does not work bouncing it off a corner.

The ball always catches air. I think it's from that driving up the wall phenomena, which makes it bounce and lose speed. It's also really hard to hit the ball without giving it a bit of air. And trust me, I tried. So many times. So we're just going to do the straight shot. Uh-oh. It worked once. All right. Well, I obliterated my prototype. Got a new one on there. See if we can do it again. Yes! Into the hole. Where'd the ball go? It's like one of those slots in the medicine cabinet where the razor blades go when you're done. They just Ball's just gone. It's just in the hole now. All right. Do you like my club? I do. Does mini golf? Check. It bounces off walls? Check. Does curvy surfaces? Check. — Rides the wave? Check. Impresses the wife? Check. All right. So, that's all we have to say, I guess. If you had to give it a score out of 10? Nine. Yes. I hope that wasn't German. So, I do have one more idea, which is what if we hit around the loop but not into the hole, and then it landed and went into the hole in the ground. Cuz the simulation does do air resistance, after all. Do it. Everyone says you should. All right. We'll try it. So, you might have noticed the pattern with some of the projects that I do, where I take a robot and somehow combine that with a human and make that human do something superhuman. And I love doing this because I think it's just the coolest demonstration of the power of technology. And I hope it inspires you to get out there and learn, maybe even to make some stuff. And if that's the case, the foundation of all of this is math and programming. That's literally This entire video is math and programming, and it's why I think this video's sponsor is important. It's the best tool that I know for learning technical stuff, and it's called Brilliant. And what makes it so good is how it works. It takes a technical subject like geometric transforms, which made an appearance in this video, and it breaks it down into a series of lessons that are also interactive. So, you're not just watching a video or being taught stuff, you're solving actual problems. And doing that just makes it hit your brain totally differently. Like it just sticks. And having the subjects divided up into smaller lessons keeps it from being overwhelming, and makes it easier to develop a continuous habit where you learn a bit every day. And it also runs on your phone. So, next time you're thinking of scrolling, you can just do a bit of learning. So, if you've had an itch to get better at math or maybe relearn the programming that you forgot from high school, check out Brilliant. You can learn for free for a full 30 days. Just go to brilliant. org/stuffmadehere, or scan the QR code, or click the link in the description. And Brilliant's also giving our viewers 20% off an annual premium subscription, which will give you unlimited daily access to everything on Brilliant. And that's it. Thank you, Brilliant, for sponsoring this video, and thank you for taking the time to check it out. Okay, we have the hit to the loop to the hole. Here we go. This actually does not work very well. You have to hit it at just the right speed, otherwise it's going to bounce out or not go in. Not a great demonstration of the technology, but if it goes in, it's still going to be awesome. Oh, come on! Oh my goodness! That's got to count. Yes! Finally! That is completely unreliable. You have to hit it at just the right speed, and even when it aims it correctly, the bouncing just is unpredictable. But even if that's the case, that is so cool to see. And my wife isn't here to see it because this took way too long to get set up. She's sleeping, but I'm sure she'll be impressed when she wakes up. So, good night.