These Are Pixels Made of Wood! 🌲🧩

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Everybody loves style transfer.   This is a task typically done   with neural networks where we have  two images, one for content, and one   for style, and the output is the content image  reimagined with this new style. The cool thing   is that the style can be a different photo,  a famous painting, or even, wooden patterns. Feast your eyes on these majestic images of  this cat reimagined with wooden parquetry   with these previous methods. And now, look at  the result of this new technique, that looks   way nicer. Everything is in order here, except one  thing. And now, hold on to your papers, because   this is not style transfer. Not at all. This is  not a synthetic photo made by a neural network,   this is a reproduction of this cat image  by cutting wood slabs into tiny pieces   and putting them together carefully. This  is computational parquetry. And here,   the key requirement is that if we look from afar,  it looks like the target image, but if we zoom in,   it gets abundantly clear that the puzzle  pieces here are indeed made of real wood. And that is an excellent intuition for this  work. It is kind of like image stylization,   but done in the real world. Now that is extremely challenging. Why is that?   Well, first, there are lots of different kinds   of wood types. Second, if this piece was not a  physical object but an image, this job would not   be that hard because we could add to it, clone it,  and do all kinds of pixel magic to it. However,   these are real, physical pieces of wood,  so we can do exactly none of that. The only   thing we can do is take away from it,  and we have limitations even on that,   because we have to design it in a way that  a CNC device should be able to cut these   pieces. And third, you will see that  initially, nothing seems to work well.    However, this technique does this with flying  colors, so I wonder, how does this really work? First, we can take a photo of the wood panels that  we have our disposal, decide how and where to cut,   give these instructions to the CNC  machine to perform the cutting,   and now, we have to assemble them in a way that  it resembles the target image. Well, still, that’s   easier said than done. For instance, imagine  that we have this target image, and we have   these wood panels. This doesn’t look anything like  that, so how could we possibly approximate it? If we try to match the colors of the two, we  get something that is too much in the middle,   and the colors don’t resemble any  of the original inputs. Not good.    Instead, the authors opted to transform both  of them to grayscale, and match not the colors,   but the intensities of the colors instead. This  seems a little more usable…until we realize   that we still don’t know what  pieces to use and where. Look. Here, on the left, you see how the image  is being reproduced with the wood pieces,   but we have to mind the fact that as  soon as we cut out one piece of wood,   it is not available anymore, so it has to be  subtracted from our wood panel repository here.    As our resources are constrained, depending on  what order we put the pieces together, we may   get a completely different result. But look. There  is still a problem…the left part of the suit gets   a lot of detail, while the right part, not so  much. I cannot judge which solution is better,   less or more detail, but it needs to be  a little more consistent over the image.    Now you see that whatever we do, nothing  seems to work well in the general case. Now, we could get a much better solution  if we would run the algorithm with   every possible starting point in the image, and  with every possible ordering of the wood pieces,   but that would take longer  than our lifetime to finish,   so what do we do? Well, the authors have two  really cool heuristics to address this problem.    First, we can start from the middle, that usually  gives us a reasonably good solution, since the   object of interest is often in the middle of the  image and the good pieces are still available for   it. Or, even better, if that does not work  too well, we can look for salient regions,   these are the places where there is a lot going  on, and try to fill them in first. As you see,   both of these tricks seem to work quite well  most of the time. Finally, something that works. And if you have been holding on to your papers,  now squeeze that paper, because this technique   not only works, but provides us a great deal of  artistic control over the results. Look at that!

And that’s not all, we can even  control the resolution of the output,   or, we can create a hand-drawn geometry  ourselves. I love how the authors took   a really challenging problem,  where nothing really worked well,   and still, they didn’t stop until they  absolutely nailed the solution. Congratulations! Thanks for watching and for your generous  support, and I'll see you next time!