AI Builders: Building an AI agent for interior design

Hi, I'm Russ and this is AI Builders, a video series for developers who love to build and are now incorporating AI into pretty much everything we do. One of my responsibilities at Weights and Biases is to build sales demos. And because I want to make these demos as close to real life applications as possible, I'm never content with anything too simple or that doesn't really do all the things it's supposed to do. Today I'm going to walk you through this application that I built that a couple of years ago would have definitely been a pretty significant time-consuming undertaking without the help of a solid engineering

team. Now, if you've done any furniture shopping in the past couple of years and have tried to scope out how a certain lamp or sofa might look in your house, this W& B Home demo interface probably looks somewhat familiar. lets you upload a photo presumably of some room in your house and then swap out any of the lamps in the uploaded photo with lamps available for purchase from a store catalog. What you're looking at here is really the final web interface for the application. So, let me quickly walk you through where I started and how I got

here. I built my application prototype here in a Mimo notebook. The notebook's included in the GitHub repo for this project. You can find the link in the video description. Now, let's quickly walk through the workflow cell by cell. Up top, I've got my library imports. And we've got our agent class. And you can see the weave model reference here because I'm using W& B weave for observability. We'll get to that in a minute. My class includes a couple of helper functions here. And then the invoke function that lets me pass in a bunch of local or remote images and my instructions as part of the prompt. Calling invoke calls the Gemini image model which combines my bedroom photo with the lamp photo. And when it's all done, I return a results dictionary with my inputs and outputs. In the next cell, I start by calling weave so I can send all my traces and results to my weave account. You can see all the local image paths here and my prompt here. I just specify the images that I'm passing in and ask Gemini to replace the current lamps in the photo with the lamp from the input image. And to get the most real life depiction possible, I'm also passing in a photo of my cat, Max. — We'll get them all set up curled up on a bench at the foot of the bed. And this is where I instantiate the class and call the invoke function. And once we've got that all set up, we just execute the cell and we're off and running.

You can see the output here below generated from our print statements and from weave as it traces our agent call. And when it's done, we have the results here below in this tabbed image viewer cell that I've created. MIMO's got a ton of cool features that basically turn your notebooks into applications. Here I can pop this up full screen and check out my input images and my final combined image. Lamps look pretty good and Max looks very comfortable. Now, if you've been paying attention to all the AI marketing, you're well aware that building prototypes is the easy part. That building something ready for production is much more difficult. And as developers, we know that this has always been the case. But the tools we use now to build a stable, bulletproof application have changed. And again, I'm using Weave for my observability and evaluations. So, let's take a look at what adding Weave to my prototype got me. Here's my

Weave account, and here's the traces page with all the inputs and outputs of every call I've made. You can see all the images we've created and we can even add feedback directly in the interface to help us keep track of which prompts and which models gave us better results and which didn't do quite as well. There's really so much we can do with all this data, but here at the start, the most important thing is just that we've got it stored somewhere safe so we can come back to it later. The notebook is of course just the beginning. Now it's time to iterate and optimize our

agent so we feel comfortable putting it under the hood of our W& B home application. And just as we did with the notebook prototype, here we're collecting the W andB home traces. And here we're running evaluations using different image models to see which perform the best in terms of accuracy, latency, and cost. And that's it. Check out the links in the video description to the project blog post in the GitHub repo with the notebooks and the images I used. I've also included a link to the Gemini API quick start so you can create an API key if you don't have one already. Thanks for your time and happy building.