AI Learns To Create User Interfaces (pix2code) | Two Minute Papers #161

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. Creating applications for mobile Android and iOS devices is a laborious endeavor which most of the time, includes creating a graphical user interface. These are the shiny front-end interfaces that enable the user to interact with the back-end of our applications. So what about an algorithm that learns how to create these graphical user interfaces and automates part of this process? This piece of work takes one single input image that we can trivially obtain by making a screenshot of the user interface, and it almost immediately provides us with the code that is required to recreate it. What an amazing idea! The algorithm supports several different target platforms. For instance, it can give us code for iOS and Android devices. This code we can hand over to a compiler which will create an executable application. This technique also supports html as well for creating websites with the desired user interface. Under the hood, a domain specific language is being learned, and using this, it is possible to have a concise text representation of a user interface. Note that's by no means the only use of domain specific languages. The image of the graphical user interface is learned by a classical convolutional neural network, and this text representation is learned by a technique machine learning researchers like to call Long Short Term Memory. LSTM in short. This is a neural network variant that is able to learn sequences of data and is typically used for language translation, music composition, or learning all the novels of Shakespeare and writing new ones in his style. If you were wondering why these examples are suspiciously specific, we've had an earlier episode about this, I've put a link to it in the video description. Make sure to have a look, you are going to love it. Also, this year it will have its twentieth year anniversary. Live long and prosper, little LSTM! Now, I already see the forums go up in flames. Sweeping generalizations, far-reaching statements on front end developers around the world getting fired and all that. I'll start out by saying that I highly doubt that this work would mean the end of front end development jobs in the industry. However, what I do think is that with a few improvements, it can quickly prove its worth by augmenting human labor and cutting down the costs of implementing graphical user interfaces in the future. This is another testament to the variety of tasks modern learning algorithms can take care of. The author also has a GitHub repository with a few more clarifications, stating that the source code of the project and the dataset will be available soon. Tinkerers rejoice! Thanks for watching and for your generous support, and I'll see you next time!