NVIDIA’s New AI: A Revolution In 3D Modeling!

I was really looking forward to this paper so let's use an AI to create a 3D virtual world but let's do it in a way that we don't need to be a skilled 3D artist how about just using a text prompt as an input nice it is giving us a list of objects that we need okay that's all well and good but that is text we don't need text we need 3D geometry yep that is the next step wow okay so how about an environment map this will be used as a background and for the lighting of the scene as well looking great now putting it all together and goodness that is fantastic but this still needs something as this is just a bunch of stuff thrown together it needs a nice overarching theme how about a Gold Rush theme there we go loving it and now get this we have a full research paper by the name edify 3D that describes the secret of how all this is possible and turns out it can do even more dear fellow Scholars this is two-minute papers with Dr Caro they call it high quality synthesis I would perhaps tone that down a little however that is still a huge improvement over previous works so what more can it do well as you saw text input works really well but not only TT text can be the input you can assemble a huge scene with just a text prompt and it can do a variety of objects and styles spectacular and can that really be wow just make a photo of something and you get a 3D model of it hm this gives away a small part of the secret which we will take a closer look at in a moment so this gives you a 3D mesh with a quad topology normals everything which in short means that I think you can use this as is in your computer games animated movies virtual avatars anything and if you look closely these are clean topologies something that you don't always get from previous Works thumbs up so how long are we waiting to get all this a scene like this would take me hours and hours to make and the AI does all this not in hours but hold on to your papers fellow Scholars it does it in Just 2 minutes now it's not 2 minutes like 2 minute papers is 2 minutes which it is not it is actual 2 minutes wow I did not find the paper specifying it but I think it is per object by the way the neural network they use here has 2. 7 billion parameters that might sound like a lot but it is a really tiny Network by today's standards if you have a newer phone you might already be running models of similar size on it without even knowing about it so what is the secret well let's pop the hood and fellow skylers now you will witness something really cool one it is a diffusion based model so it starts out from noise but two it generates not one image but a bunch of images then he tries to guess what kind of 3D geometry would look like these images I am loving this and in the meantime the textures are waiting to be put onto the final model to make sure that the quality is as good as it can be it also does its own upscaling super resolution if you will the enhanc thing now you see that of course it can do text to 3D and image to 3d2 it is built into the system it is how it works and finally here is the secret it is also not an accident that multiple views are being generated the whole neural network was trained to be able to understand 3D geometry from a bch of 2D views the more the better for instance if you train it on four views this is what it can do but if you bump it up to eight there is a noticeable difference now limitations we have textures they are up to 4K resolution goodness okay but still no sophisticated material models yet only albos that is at the risk of simplifying it color information for each part of the geometry there are previous works that special I on materials and you can generate absolutely stunning virtual objects with really sophisticated material model and I bet that this is coming just two more papers down the line I met this team a few months ago at the Nvidia headquarters and they are super nice and Brilliant scientists they showed me an earlier version of this work and said that they are working to improve it and might be able to show it a few more months down the line and now

here we are so cool note that we do not have a business relationship with Nvidia and note that this area is subject to intense research attention for instance this one is called mesh GPT that is able to build new geometry right in front of your eyes and is also way better than previous works at that look at how precise these objects are so cool and it can even suggest you possible ways of completing your own work how cool is that so what do you think what would you fellow scholar use this for let me know in the comments below to run your own experiments on an Nvidia GPU check out Lambda I use it myself regularly for these videos H look at that you can generate high quality images in less than a second per image I did a ton more of them and paid less than a dollar for all this crazy seriously try it out now at lamb labs. com SLP ERS or click the link in the description