# NVIDIA’s New Gaming AI Does The Impossible!

## Метаданные

- **Канал:** Two Minute Papers
- **YouTube:** https://www.youtube.com/watch?v=Fv4q4UWyLMQ
- **Дата:** 04.02.2024
- **Длительность:** 7:09
- **Просмотры:** 98,440
- **Источник:** https://ekstraktznaniy.ru/video/12744

## Описание

❤️ Check out Weights & Biases and sign up for a free demo here: https://wandb.me/papers

📝 The paper "Learning Physically Simulated Tennis Skills from Broadcast Videos " is available here:
https://research.nvidia.com/labs/toronto-ai/vid2player3d/

Code: https://github.com/nv-tlabs/vid2player3d

📝 My latest paper on simulations that look almost like reality is available for free here:
https://rdcu.be/cWPfD 

Or this is the orig. Nature Physics link with clickable citations:
https://www.nature.com/articles/s41567-022-01788-5

🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Bret Brizzee, Gaston Ingaramo, Gordon Child, Jace O'Brien, John Le, Kyle Davis, Lukas Biewald, Martin, Michael Albrecht, Michael Tedder, Owen Skarpness, Richard Putra Iskandar, Richard Sundvall, Taras Bobrovytsky, Ted Johnson, Thomas Krcmar, Tybie Fitzhugh, Ueli Gallizzi.
If you wish to appear here or pick up other perks, clic

## Транскрипт

### Segment 1 (00:00 - 05:00) []

wow I can't believe it this new AI can look at a real tennis player and put them in a computer game where we can play with them this looks absolutely amazing and doing this the hard way has so far been absolutely impossible but what about the easy way exciting as this may seem perhaps there is nothing new here you see we can already do something like this through motion capture what is that well you put a human in a studio attach these little balls to them and record their motion then transfer this motion to a virtual character however this paper has an incredible twist and that twist is that this can do the same but without the balls no cameras gyroscopes or any additional Machinery is required just the raw video feed and the learning happens from that sounds incredible but how dear fellow Scholars this is 2minute papers with Dr Caro well there are previous works that try to do something similar but the Precision of these motions was almost never satisfactory especially for tennis where almost hitting a ball is not nearly good enough and I am very proud to say that all three of these papers have been chased here on two minute papers before but this new technique this one can do it even better first we give it the raw video feed of the players and then it will try to estimate their motion so are we done well unfortunately not even close look the motion is still way too jerky and imprecise to be used in practice it is also full of artifacts so what now well now comes the magic here a computer simulation is built that tries to reproduce this motion but this one cannot be jerky because it is within a video game where the laws of physics are intact in reality people don't move like that and with this simulation H this is looking so much better from now on we can take a Target position that we start from and simulate many motion types then we grab a controller and perform these serves Top Spin Back Spin hits slices you name it and they look absolutely incredible yes I hear you asking caroy are you sure that this is a simulation well if you look here is the proof yes simulations today are getting so Advanced that we need to see these 3D models to be absolutely sure that this is in indeed a computer simulation wow and it gets better here is the interesting part this is not just a copying machine although just copying the motion from these videos would also be quite a feat given the occlusions and Tiny But super important wrist motions that are really hard to see from the footage but with this system we can even highlight the region where we would like to hit the ball and the Motions will be syn sized accordingly we can run the same simulation with a number of different Target locations and all of the synthesiz Motions will be believable and accurate loving it or we can also ask the player to hit the ball from the same point from different starting positions and it will work not only for the easier cases but in more difficult scenarios too where you can barely catch that ball so good and now hold on to your papers fellow Scholars because it can not only do all this but it can even learn the individual style of famous tennis players right and left-handed players work too and it can simulate a one or two-handed backhand shot depending on the style of the player at hand and if that is possible at all it gets even better if we can simulate one virtual human why not make a game by simulating two playing against each other I have to say if I look closely I can still see that this is a simulation but if you squint a little it's easy to think that this is real footage that is an incredible achievement both in the graphics and in the animation departments Bravo and styles are particularly interesting here because if it would have only one style two players playing the same St style would not look

### Segment 2 (05:00 - 07:00) [5:00]

nearly as interesting as this footage with one left and one right-handed player what I particularly like is that there is barely any foot sliding a problem that you see in almost every single Paper in this area they usually compete on how little foot sliding there is no foot sliding at all is almost unheard of wow also almost none of the Jitters from the original motions are there in this footage the foot sliding here is less than an inch and the Jitter is almost non-existent all gone and if we remove just one small part of this new proposed system things can go really bad for instance if the rest motions are captured with just a bit of inaccuracy the Heats will go completely wrong you have to get everything nearly perfectly for this to work and the scientists absolutely nailed it so how many hundreds of hours did the AI had to sft through to learn all this well it was not in the order of hundreds of hours but get this only 4 and a half hours of video footage goodness I don't even know what to say when using the four system the heat rates can reach up to 100% which sounds almost impossible to me they absolutely knocked this out of the court with this paper and thus it was published at sigraph perhaps the most prestigious conference in computer Graphics in my view it is like winning the Olympic gold medal of computer Graphics research and I have to say it is very welld deserved Bravo this might be the future of video games experiment tracking model evaluation and production monitoring for your deep learning projects and llm apps this is what weights and biases does and it is the best everyone is using it try it out now at wb. me/ papers or click the link in the description below