Neural Material Synthesis | Two Minute Papers #88

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. If you are new here, this is a series about research with the name Two Minute Papers, but let's be honest here. It's never two minutes. We are going to talk about two really cool papers that help us create physically based material models from photographs that we can use in our light simulation programs. Just as a note, these authors, Miika and Jaakko have been on a rampage for years now and have popped so many fantastic papers each of which I was blown away by. For instance, earlier, we talked about their work on Gradient Domain Light Transport, brilliant piece of work, I've put a link in the description box, make sure to check it out! So the problem we're trying to solve is very simple to understand: the input is a photograph of a given material somewhere in our vicinity, and the output is a bona fide physical material model that we can use in our photorealistic rendering program. We can import real world materials in our virtual worlds, if you will. Before we proceed, let's define a few mandatory terms: A material is diffuse if incoming light from one direction is reflected equally in all This means that they look the same from all directions. White walls and matte surfaces are excellent examples of that. A material we shall consider specular if incoming light from one direction is reflected back to one direction. This means that if we turn our head a bit, we will see something different. For instance, the windshield of a car, water and reflections in a mirror can be visualized with a specular material model. Of course, materials can also be a combination of both. For instance, car paint, our hair and skin are all combinations of these material models. Glossy materials are midway between the two where the incoming light from one direction is reflected to not everywhere equally, and not in one direction, but a small selected set of directions. They change a bit when we move our head, but not that much. In the Two-Shot Capture paper, a material model is given by how much light is reflected and absorbed by the diffuse and specular components of the material, and something that we call a normal map, which captures the bumpiness of the material. Other factors like glossiness and anisotropy are also recorded, but we shall focus on the diffuse and the specular parts. The authors ask us to grab our phone for two photographs of a material to ensure a high-quality reconstruction procedure: one with flash, and one without. And the question immediately arises: why two images? Well, the image without flash can capture the component that looks the same from all directions, this is the diffuse component, and the photograph with flash can capture the specular component because we can see how the material handles specular reflections. And it is needless to say, the presented results are absolutely fantastic. So, first paper, two images, one material model. And therein lies the problem, which they tried to address in the second paper. If a computer looks at such an image, it doesn't know which part of one photograph is the diffuse and which is the specular reflection. However, I remember sitting in the waiting room of a hospital while reading the first paper, and this waiting room had a tiled glossy wall, and I was thinking that one image should be enough, because if I look at something, I can easily discern what the diffuse colors are, and which part is the specular reflection of something else. I don't need multiple photographs for that. I can also immediately see how bumpy it is, even from one photograph, I don't need to turn my head around. This is because we, humans have not a mathematical, but an intuitive understanding of the materials we see around us. So can we explain the same kind of understanding of materials to a computer somehow? Can we do it with only one image? And the answer is, yes we can, and, hopefully, we already feel the alluring call of neural networks. We can get a neural network that was trained on a lot of different images to try to guess what these material reflectance parameters should look like. However, the output should not be one image, but multiple images with the diffuse and specular reflectance informations, and a normal map to describe the bumpiness of this surface. Merely throwing a neural network at this problem, is however, not sufficient. There needs to be some kind of conspiracy between these images, because real materials are not arbitrarily put together. If one of these images is smooth, or has interesting features somewhere, the others have to follow it in some way. This "some way" is mathematically quite challenging to formulate, which is a really cool part

of the paper. This conspiracy part is a bit like if we had 4 criminals testifying at a trial, where they try to sell their lies, and to maintain the credibility of their made up story, they have previously had to synchronize their lies so they line up correctly. The paper contains neat tricks to control the output of the neural network and create these conspiracies across these multiple image outputs that yield a valid and believable material model. And the results, are again, just fantastic. Second paper, one image, one material model. It doesn't get any better than that. Spectacular, not specular... spectacular piece of work. The first paper is great, but the second is smoking hot, by all that is holy, I am getting goosebumps. If you are interested in hearing a bit more about light transport and are not afraid of some mathematics, we recently recorded my full course on this at the Technical University of Vienna, the entirety of which is freely available for everyone. There is a link for it in the video description box, make sure to check it out! Thanks for watching, and for your generous support, and I'll see you next time!