Texture synthesis typically means that we need lots of concrete or gravel road, skin, marble, create unique stripes for zebras, for instance, for a computer game or the post-production of a movie, and we really don’t want to draw miles and miles of these textures by hand. Instead, we give it to an algorithm to continue this small sample, where the output should be a bigger version of this pattern, with the same characteristics. So how do we know if we have a good technique at hand? Well, first, it must not be repetitive, checkmark, and it has to be free of seams. This part means that we should not be able to see any lines or artifacts that would quickly give the trick away. Now, get this, this new paper attempts to do the same with neural cellular automatons. What an insane idea! We like those around here, so let’s give it a try! How? Well, first, by trying to expand this simple checkerboard pattern. The algorithm is starting out from random noise, and as it evolves…well, this is a disaster. We are looking for squares, but we have quadrilaterals. They are also misaligned. And they are also inconsistent. But, luckily, we are not done yet. And now, hold on to your papers, and observe how the grid cells communicate with each other to improve the result. First, the misalignment is taken care of, then, the quadrilaterals become squares, and then, the consistency of their placement is improved. And the end result is, look! In this other example, we can not only see these beautiful bubbles grow out of nowhere. But the density of the bubbles remains roughly the same over the process. Look! As two of them get too close to each other, they coalesce, or pop. Damaged bubbles can also regrow. Very cool! Okay, it can do proper texture synthesis, but so can a ton of other handcrafted computer graphics algorithms, so why is this interesting? Why bother with this? Well, first, you may think that the result of this technique is the same as other techniques, but it isn’t. The output is not necessarily just an image, but can be an animation too! Excellent. Here, it was also able to animate the checkerboard pattern, and, even better, it can not only reproduce the weave pattern, but the animation part extends to this too. And now comes the even more interesting part. Let’s ask why does it output an animation and not an image? The answer lies within these weaving patterns. We just need to carefully observe them. Let’s see. Yes, again, we start out from noise, where some woven patterns emerge, but then, it almost looks like a person who started weaving them until it resembles the initial sample. Yes, that is the key! The neural network learned to create not an image, not an animation, but no less than a computer program to accomplish this kind of texture synthesis! How cool is that? So, armed with all that knowledge, do you remember the regenerating iguana project? Let’s try to destroy these textures too and see if it can use these computer programs to recover and get us a seamless texture. First, we delete parts of the texture, then, it fills in the gap with noise, and now, let’s run that program! Wow! Resilient, self-healing texture synthesis. And in every case, it starts out from a solution that is completely wrong, improves it to be just kind of wrong, and after further improvement, there you go. Fantastic! What a time to be alive! And, note that this is a paper in the wonderful Distill journal, which not only means that it is excellent, but also interactive, so you can run many of these experiments yourself right in your web browser. The link is available in the video description. Thanks for watching and for your generous support, and I'll see you next time!
