Introducing TanStack AI Code Mode

Now that we've worked at LLMs for a while, we know that there are things that they're very efficient at and other things that they're not so efficient at. And one of the at is tool calling, which is a bummer because in order to actually make your agent do anything cool, it needs to have tools. So how do we fix and optimize how we call our LMS to use their advantages and get away from their disadvantages? Well, at TanStack, we've been building out this thing called code mode and I want to show you the difference between regular tooling and code mode so you can understand why we're so excited about this. Lemme show you just how inefficient tool calling is and how code mode fixes it. This is an example page. We've got a shoe product list here. So you've got two tools. getProductListPage, which gives the list of IDs as well as the page count, and then getProductById. It roughly maps to REST N+1 API. And a lot of folks do this kind of thing with their LLMs, and it's horribly inefficient. Let's ask the question. Let's say, what's the average cost of a shoe? Now we're gonna run that with regular tools, and we can see over here that it's making a bunch of tool calls. Now what happens with those tool calls is that the LLM responds to us with a request to run a tool, and then we run that tool and then we start a whole new request with all of the messages that we had before and all of the tool calls and all the results, and we send that back to the server for processing. And in this case, just to answer that simple question and get back the average price of the shoes, we've had to make four round trips between us and the server to service all of those tool calls, even though a lot of these tool calls were done in parallel. So this is about the most efficient that you're gonna get with this N+1 pattern when it comes to normal tool use. And we can see that the context that we've sent back and forth to the server has really compounded. We've got 9. 8 k worth of use of context going back and forth to the server. So even the simple question is actually pretty inefficient. Now to fix this, we lean on something where we know that the LLM is really good and that's in writing TypeScript. So let's take a look at this exact same scenario, but with code mode. I'm gonna click on the code mode button here. And we can see that instead of running those tools directly and instead generated some TypeScript code that it then sent to execute_typescript and that Typescript code was executed inside of an isolate. That isolate had access to getProductListPage and getProductById. And so it could just go and do things like Promise. all, map, reduce all that stuff to go and figure out what's the average cost of a shoe. So it literally wrote the code to do that. And we can see a huge performance win instead of four LLM calls. Now we've got two instead of 9. 8 kilobytes of context, we've got 1. 7. And it's even faster to do it. All the tool calls were 27 seconds, and the duration of code mode was eight seconds. And the next thing we know about LLMs is that they're not particularly good at math. Check out the average price. All of the shoes coming out of the regular tools versions. A hundred thirty, four fifty, is that right? Well, actually, no, it's not. The code mode version gets you the actual price because it's just using TypeScript to do the calculation and the actual average price is 137 and 75 cents. So now let's go over to the app and see how this is actually working. Now this is the TanStack AI Monorepo, and I'm over in the examples ts-code-mode-web directory. That's the application that we were just looking at. And in there is the routes for home. That's the homepage that we were just looking at. And inside that is the API for product code mode. So this is what the code mode side of the chat calls to get anything done. And the really interesting thing here is this async function get code mode tools. So code mode is actually modeled as just a tool. You just call createCodeMode. You give it a driver, that's the isolate driver that's actually gonna be used to run the TypeScript. And you can use QuickJS, node or a CloudFlare worker currently. You create that isolate driver, you give that driver over to code mode, and then you give it the tools that are injected into that isolate. And what we get back is a standard TanStack AI tool that we just give off the chat and a system prompt. And the system prompt has all of the TypeScript typings for the injected tools as well as all the information that the LLM will need to know about how to call execute_typescript. And then we just add that system prompt onto our existing system prompt. And that's it. That's all it takes to do code mode. But if you're thinking that's pretty cool, well, I agree with you, but it actually gets better. So check this out. We also have code mode skills, and that's an additional library that you can layer on top of code mode and it allows the LLM to actually store TypeScript that thinks is valuable. I mean, if you think about it for a second. This generated code over here is basically a tool. So the next time we ask for an average, it could just reuse this code, right? Let's try that. Let's turn on with skills and then run it again.

And now, in addition to giving us the output, it's also registering a skill. That skill is get_average_product_price. And it's got the input schema. The output schema, as well as the code, a description, and we can take a look at it over here and actually see it. Now this is persisted currently in this demo on disc, but you could put it anywhere you want. in a database, you could put it anywhere. Now let's refresh the page. Click on with skills again, go for average cost and see how much faster is that? Well, it just used the skill that had just created. And look how fast and efficient it is. Two calls the LLM, half a K of context, a duration of three seconds. It's astounding. But that's not all. We've actually been working on code mode for a little while, and so we've kind of gone a little hog wild on it. We've got a cool database demo over here where you can see that we've just connected code mode straight to a database, and it's gonna build the TypeScript as well as all the SQL to go and do database work for you. This you can just use right outta the box on your database, get some cool reports. It's very exciting. Just go and put it onto your database. You're gonna love it. We've got a much more in depth example called MPM GitHub Chat. It shows that you can have regular tools as well as code mode tools all together in the same system. It actually allows you to go and select between the different isolates and it shows you how often various tools are used in real time. It even shows you how to use custom messages to go and put up UI like this NPM data over here. But if you think that's cool, well hang onto your hats. If you go over here to Dynamic Reports, this is where it really gets exciting. Let's take a look at hot state libraries. This is exactly the same set of tools as we had before, but we've added on some UI tools and now code mode can actually go and build out a UI dynamically for this report. It is so cool. All I asked it for was this report. It figured everything out for me. Now this is using code to build out the UI and I've seen a lot of JSON specified UI stuff out there for LLMs, but why not just use code? This works really, really well. I could go on and on about code mode, but I really encourage you to check it out. Go clone the TanStack AI monorepo, run ts-code-mode-web. Try it out for yourself, try it out on your own data. Can't wait to hear what you're experiencing. Of course, leave that in the comments right down below, and if you have any questions, I'll try to answer them down there. In the meantime, if you like this video, hit that like button. If you really like the video, hit the subscribe button and click on that bell and you'll be notified. The next time I new Blue Collar Coder comes out.