Google’s New AI Fixes The #1 Problem With Your Photos!

This AI is what Photoshop should have been,   and it’s a bit like reaching into a photograph  to add the sun to it. Brighten it up a little. But this is just an illustration, so at first I  thought this work seriously cannot be real. You   see, if you have a photo and you wish to brighten  it, no problem. Just take Photoshop. Grab one of   those sliders, and there we go. But, if you want  to turn a lamp on or off, you are out of luck.    Not possible. Until today. Now hold on to your  papers Fellow Scholars, and check this out. With this new AI technique, you take  a photo, and if you wish to have   the lamps on or off, you just use this  slider. And this is where it gets wild:   the system doesn’t just brighten or darken things  - it handles reflections and shadows correctly,   even the specular highlights on glass and  metals. You can turn off sunlight from a window,   switch on a lamp, change its color, and the  shadows follow the physics beautifully. It’s   like Photoshop, but actually obeying the  laws of light transport. Absolute insanity. Turn off the lights so the cat  can sleep better? No problem.    Pretend that you’ve been studying the  papers all day and night? No problem. So that was the initial shock, but  wait, it gets even better. Now get this,   it only takes 5 seconds per image. Here  it is shown like this for our pleasure,   but remember, 5 seconds. You know, the  5 second rule holds for this one too. But it goes further than that. It can even do  things beyond all this craziness! Dear Fellow   Scholars, this is Two Minute Papers with Dr.   Károly Zsolnai-Fehér. Dr. Carroll. Get this:   the system can even push intensity  beyond its trained range and still   produce reasonable results, including these  neat edits. It’s like over-cranking a dimmer   past 100% and it still keeps up. And it  even works on stylized or out-of-domain   images - for example turning on a cartoon  desk lamp realistically. Absolutely amazing. Also, you can add an invisible point light and  the scene still shows plausible falloff and   highlights. It’s like placing a ghost flashlight  in mid-air and the room reacts correctly. This is an AI-based technique, so how  was it trained? How is all this possible? Well, first, you can take a bunch of  rendered photos from 3D modeling systems,   in this case, 20 curated scenes from  Blender. Here, you can turn on and off   lights in your rendered synthetic images as  you like, this is a virtual world after all. But here is the catch, they also do real-life  photos with lights on and off. Why is that? Well,   they isolate one lamp’s effect by simple  light arithmetic - take the ON image,   subtract the OFF image, and you get  exactly that lamp’s contribution.    It’s like removing the band from  a song to hear only the singer. This is the secret sauce and it is surprisingly  elegant. We know that diffusion models can get   confused by unrealistic computer graphics. But  they also need a massive amount of data to learn   physics, which is hard to get from real photos.   And the researchers found the perfect balance.    They used a small set of real photograph  pairs to teach the model what real-world cameras,   lenses, and lighting look like. Then, they  used a massive dataset of over half a million   synthetic images to drill in the complex rules  of shadows and reflections in countless different   scenarios. The real data keeps the AI grounded in  reality, while the synthetic data gives it a PhD   in physics. And this is the combination that  makes these results so incredibly plausible. So, the training mixes a small real set of real  photos, about a few hundred, with a huge synthetic   set of 600k rendered images. This enables it to  learn both real camera quirks, and physics at the   same time! It’s like learning from real classrooms  and then drilling in a flight simulator. But that’s actually absolutely crazy.   To understand how monumental this is,   consider the old way of doing this. You would  need to build a full 3D model of the scene from   the single 2D image - an almost impossible  task. You'd have to guess the geometry,   the texture of the couch, the material of the  floor, the roughness of the metal on the lamp.    Only then could you try to re-render it. This  AI sidesteps that entire impossible pipeline and   somehow accomplishes it even better. By training  on a clever mix of real and synthetic data,   it doesn't need to recreate the 3D scene.   It just learns the rules of light transport.    It learns exactly how light reflects off  shiny surfaces and scatters on matte ones,

all from just looking at pixels. It's a  testament to how modern AI can learn the physics   of our world without us having to explicitly  program it to do so. What a time to be alive! But wait, all this still should not work at all!   Here’s why. If light transport were nonlinear,   like many things in physics, that would  mean turning a lamp on would also warp   materials and colors unpredictably, so ON  minus OFF would not isolate the lamp at   all. But the light transport operator in  mathematics IS linear. Thank the Papers!    That means that you can add and subtract  these lights as if they were numbers. That   is not trivial at all, and this property had  to be used in this paper to make all this work. And it works beautifully. The lead author wrote  to me about it that he is a Master student behind   this incredible work and he’d love to be on Two  Minute Papers. Well, there we go, the honor is   mine. And man, so young, with so much talent.   Absolutely incredible. Great job Nadav! And   since almost nobody is talking about this work,  I wanted to make sure that you Fellow Scholars   and the world hears about it. Like, leave a  comment and subscribe if you’d like more of this.