We Taught an AI To Synthesize Materials 🔮

creating photorealistic materials for light transport algorithms requires carefully fine-tuning a set of material properties to achieve a desired artistic effect this is a lengthy process that involves a trained artist with specialized knowledge in this work we propose a system that only requires basic image processing knowledge and enables users without photorealistic rendering experience to create high quality materials this is highly desirable as human thinking is inherently visual and not based on physically based material parameters in our proposed workflow all the user needs to do is apply a few intuitive transforms to a source image and in the next step our technique produces the closest photorealistic material that approximates this target image one of our key observations is that even though this process target image is often not physically achievable in many cases a photorealistic material model can be found that closely matches this image our method generates results in less than 30 seconds and works in the presence of poorly edited target images like the discoloration of the pedestal or the background of the gold material here this technique is especially useful early in the material design process where the artist seeks to rapidly iterate over a variety of possible artistic effects we also proposed an extension to predict image sequences with a tight budget of one to two seconds per image to achieve this we propose a simple optimization formulation that is able to produce accurate solutions but takes relatively long due to the lack of a useful initial guess our other main observation is that an approximate solution can also be achieved without an optimization step by implementing a simple encoder neural network the main advantage of this method is that it produces a solution within a few milliseconds with the drawback that the provided solution is only approximate we refer to this as the inversion technique both of these solutions suffer from drawbacks the optimization approach provides results that resemble the target image but is impracticable due to the fact that it requires too many function evaluations and get stuck in local minima whereas the inversion technique rapidly produces a solution but is more approximate in nature we show that the best aspects of these two solutions can be fused together into a hybrid method that initializes our optimizer with the prediction of the neural network this hybrid method opens up the possibility of creating novel materials by stitching together the best aspects of two or more materials deleting unwanted features through image in painting contrast enhancement or even fusing together two materials these synthesized materials can also be easily inserted into already existing scenes by the user in this scene we made a material mixture to achieve a richer Nebula effect inside the glass we also show in the paper that this hybrid method not only gives a head start to the optimizer by endowing it with a useful initial guess but provides strictly higher quality outputs than any of the two previous solutions on all of our test cases furthermore if at most a handful of materials are sought the total modelling times reveal that our technique compares favorably to previous work on mascara material synthesis we believe this method will offer an appealing entry point for novices into the world of photorealistic material modeling thank you for your attention