Studio Update #07: AI Agent With Own Gmail + Slack, How to Run DeepSeek / OpenRouter LLMs in n8n

all where are we going we're in a forest oh we went a bit off track hey everyone I'm Max the original flow gramar and this is the studio update the AI and automation show documenting flow gmers around the world doing awesome things with a i in automation I got a fantastic episode this week had a great chat with Devon kierin from Custom AI Studio who shows me his mallerie AI agent I've also got an update on the project I'm working with nidn support team to tag incoming support tickets with AI I've also got a cheeky sneak peek of an upcoming end feature and I didn't get permission to share it but it starts with e and ends with valuations we've also got an update from the esteemed Angel Menendez and sneaky little life hack on how you can use various different LM nodes in and at end without us natively supporting them all right Kang you know the drill it's K things off with some studio project

updates all right first off people have been loving the building AI agents tutorial series part three is out last week focuses on prompt engineering I'm going to pause that series for a little bit get some feedback as I work on other projects and then kick it back off so do be leaving comments on those videos on what you want me to see next and then we broke ground on the auto tag incoming sport tickets with AI project this is a collaboration between me and nidn support team and what we've done so far Ria started a work for that pulls relevant examples uh because if you recall what we're going to do first is try a multi-shot approach so we're going to take previous tickets that they've tagged manually and use those as examples in our prompting and see if that's good enough so if that works we're going to ship it great and if that doesn't work we're going to try fine-tuning a model on those same examples so either way we need high quality example data and I've got it to where I've got a Json file outputting from an nn workfl it's a beastly 17 megab file but it's got six examples for each tag so the way it's organized is the top level item is tags then within each tag there's the six different tickets the nice thing about that is the way I have the pipeline set up is if I need 12 tickets I can just rerun that take a couple minutes and that'll be done and the good thing is with the number of examples that I have is I can use some for the examples and some for testing because it's basically a supervised training set right humans have already tagged all these tickets so if I show the AI three examples and then run in the three others that I haven't and those work we've got a basis of like a working example and then we can test it on more tickets of course this week I'm continuing on building that and with any luck this time next week in an update I've got green lights it's all working we're shipping it and handing it off to the support team with a bunch of happy campers if that doesn't work we then fine tune that figure out how to do that cuz I'm not sure and share that process either way I'm taking little Vlogs as I go and outside of this series I'm going to probably cut down a little mini Dock of the whole process show you from sort of Discovery through to deployment how we solved a real problem at NN with NN intuous all right next up let's get a

cheeky little update from Angel Menendez and see what he's been cooking up so max queue up that fun little Arizona transition flyover thingy hey Max hope you're doing well and hey everyone one of the challenges that small cyber security teams face is a clear lack of direction from the alerts that they're given they use a system called a Sim to generate alerts from a tool that logs everything that is connected to their environment this doesn't always give them the information they need to fix these cyber security attacks we've built an AI power tool that extracts actionable insights groups attack types and guides teams through remediation these raw Sim alerts provide a lot of technical information but they don't help you figure out how to remediate these issues additionally these alerts aren't grouped together so it makes patterns hard to spot and worst case if you have a new team they may not know where to even begin the workflow that we've built tags these alerts with TTP or tags that allow you to group these types of attacks together it uses a vector store that has been trained on the miter attack frameware and it provides detailed remediation steps for each alert that comes in this also will help teams spot different Trends or patterns in the types of attacks that are coming in automatically this won't replace cyber security teams though it's going to give them the confidence they need to jump in and start working on these tickets and help clear out the queue I'm currently working on two different videos one that shows you how to build your own Vector store so you can deploy it internally I'm also going to be providing a copy of the clean data that I'm using for the miter attack framework training and lastly I'm going to provide a workflow that already has prompts in it to make it easier to deploy this in your organization and let's take a quick look at a demo of how this works I want you to imagine a small Healthcare organization with a limited cyber security team on the morning of November 26th 20124 their Network triggered multiple alerts indicating suspicious activity now these alert its flag potential issues but the cyber security team didn't know where to start and to give you an idea of what they see this is essentially an example of some sample similar data as you can imagine looking at this it's not particularly useful information unless you have a very deep understanding of cyber security these can flow into an organization's ticketing system so if we open it up we can see we have the ticket name and we have the ticketing information this is not very useful so without more cont actual information it's very difficult for a small cyber security team to be able to just jump right in and start working on this they're going to have to do some additional backend data to try to understand what these alerts mean why they're flagging them they provide technical data but no actionable guidance to essentially walk them through what the next steps should be let's go ahead and take a look at what we've built this workflow trains your vector store so here what we have is we have a Google Drive pool so of a binary file so this pulls in the Json file that contains the miter attack data that catalog of different malicious attacks that a hacker might utilize to try to gain entry into an organization once we receive that Json data we go ahead and extract it from the file from there it becomes Json which we then split out into individual Json objects and then we use those objects to train our quadrant Vector store this allows us to program a collection within quadrant with the attack data now once this is complete we're able to essentially chat with our Vector store so if we go ahead and pull one of these alerts we can actually get some useful data here so I can just pull up the chat and go into the message input and we can go ahead and submit it and as you can see it's going ahead and processing it in the agent it's querying the vector store passing that data back to the agent and essentially processing this multiple times until it gets a final output so as you can see here it ran two queries within the vector store and here we go TTP information extracted so this looks like it's a command and control tactic using Dynamic resolution and here we even have the miter ID technique ID so adversaries May establish connections to command and control infrastructure using Dynamic resolution techniques to evade detection this involves using algorithms to dynamically adjust parameters such as domain names IP addresses or port numbers used for command and control so this is a lot more clear already than the alert that I fed into here this at least gives a cyber security team a starting point to start working on this ticket and here we have the remediation steps deploy intrusion detection and prevention systems to monitor for suspicious traffic threat Intel integration update intelligence feeds and we even link to the miter attack resource to get even more information and even see what hacker groups might be using these types of techniques against a particular organization so now that we have the chat set up that's all well and good but again most organizations utilize titing systems for this so what we can do is we can go ahead and take a look at the next step which is essentially embedding this within our zenes ticketing system so if we go back to our zenes ticket here we have our ticket again it doesn't have much information but if we run this workflow let's go ahead and test it it's going to pull the tickets and it's going to process each one using the quadrant Vector store and once it receives the information it then passes that information into the ticket itself to update it so let's go ahead and run this and see essentially what happens on the back end so there we go so we got one of the tickets updated this is ticket 24 let's take a look there we go so as you can see we have all the data it's very easy to read it's easy to pull up the recommended resources to get a better idea of how to educate our team to remediate this one of the cool things is that you don't have to run this using open AI like I'm running here if we go down here you'll see that we can actually change the model that it's running to something else so if I disconnect this I can go ahead and hit the plus and here we have multiple different models including olama if I wanted to run it locally especially if I'm working for a cyber security organization I don't want to expose my information to external third parties such as opening I so here we can run this model locally and still have access to this wonderful data store that stores the miter attack framework so there we go all the tickets have now been updated if we head back to I think it was ticket 30 that we were looking at perfect identified techniques so this is process hollowing and task scheduling this right now is just updating a note so a cyber security analyst can come in here read it and process it however you can structure the output for the AI chatbot to essentially output in a structured format in Json so that we can feed these into custom fields in zenes jira or in whatever platform you're utilizing and pass this information in a format that allows for things like graphing or creating dashboards I hope that you can see the benefit of this type of tool so not only can we train it on cyber security information but we can train it on just about anything that is Json data by utilizing the vector store you're essentially performing an automated search on a back end that only gives you back the objects within that database that are relevant to your query this not only reduces the cost of running the llm but it also increases the accuracy because you're dealing with much smaller context windows and that's the demo back to you Max hey thanks a lot as always Angel appreciate that looking forward to your next update this next one is an absolute

life hack for anyone to use llm models that NM doesn't officially support tldr is that most llms especially open source ones conform to open ai's API spec basically you can change a few settings and have in and in node think it's talking to open AI when it's talking to some arbitrary service for example deep seek and a big thank you to Anthony Lee he a guest we had on the show recently U because I knew about this but he dm'd me about this and he was so excited as if uh he just got his Christmas presents that I realized this is the kind of thing that's really quick to know but super helpful and difficult to figure out if you don't know about it so big shout out Anthony for sending me that because again that's why everyone's going to get to see this and if there's at least one person that found it valuable that's good enough for me although probably not good enough for my boss Lu GM please do uh let me know if you like that so the scenario is I click on chat model in N ATN and I go oh no we don't have that cool new llm that I'd like to try what do you do so the neat thing is that open ai's uh schema for their chat completion API so it has the structure of a base URL and then the version and then SL chat completions they have an expected input structure here you can see they have a model key for specifying the model name how they expect chat messages to be sent between the different roles and then the expected response the neat thing is that a lot of the open source Community other companies have adopted their structure for this API endpoint so if you know how and where to swap basically this base URL you can basically access like dozens or maybe hundreds of really cool open-source models all right how to do it there's going to be multiple different vendors that support this format one that's good is open router so you do need to put a credit card on there I think they give you about a dollar of credit before you do that so you can test it out and they've access to all these different types of models if we have a look there's deep seek R1 and I did just test it we're going to show how to set that one up so deep seek it's a Chinese model that's kind of in the hype cycle right now you can try that out but I think what was really cool is seeing all these Niche models for specific use cases like this one here Rogue Rose is just for role playing and storytelling and I saw there ones kind of focus a bit more on legal and whatnot I think what's really cool about open router is the categories that they have as you can see here there's categories for you know on hugging face your categories are very technical it's like text classification multimodal image this that what's nice here is these are focused on use cases because we're starting to see with all these great Foundation models folks taking those as bases fine-tuning them or we starting independently but again Niche models for Niche use cases right so let's say we pick deep seek R1 and I want to add this the first thing you're going to want to do is head over to the API Tab and you're going to want to create an API key and then you'll click create API key give it a name give it a credit limit if you like okay so you've got your API key in your pasteboard right you go over to n8n and then you add the open AI chat model to your AI agent or any other AI node like a basic chain summar summarization chain that kind of thing now in here what we're going to want to do is you see I've already created an open router account but let's create another one I'll create that and let's just call this my open router account so we'll paste in that API key you don't need an organization ID and you do need to swap this base URL because this is the default for open AI so if we go back to open router you're going to want to go back to a model page then in the API section of a model you're going to scroll down and you're going to see this base URL in the using open AI SDK so this is what we want to copy here now one thing when I was trying this is that URL this versions kind of this URL because that that's the base so you're going to see in other places like here/ chat completions make sure that you just have SL V1 without the leading slash off to it if you're seeing this later down the line and you go on this website says V2 that means they've updated it but yet do that don't include SL chat completion to end it and handles that for you so we save this and yep connection test so it's working so now the one difference is this list isn't going to autoload so even if I refresh it it's failing so what you're going to have to do is set this to an expression go over here to whichever model you want to add and then here from the top of the page for example here we can just copy that so then we'll paste that in here so this should be it again because it's using open AI API spec all of these different options should work if the model supports it but let's just stick with the defaults right now and let's give it a quick chat so who are you now in testing this the model did seem a little slow so open router is a great place to try these things I'm not sure if it's very performant here they have an uptime thing here because I think they're perhaps aggregating from yeah different deep seek providers because basically open route is an aggregator right so there's different providers under the hood they're probably taking a little margin on top and then you know handling that for you so I would say this is probably good for like when you're testing and building if you're going to use one of these models you may need to like pick a specific provider cuz here we can see these different providers you know 15 tokens per second is not that much and here we're seeing even latencies of you know multiple seconds even up to a minute now this deep seek model is super popular right now it's getting it's in the hype cycle it's why I'm doing this video so probably it's going to take a little longer there too but we see it returned who are you think greetings I'm deep seek da da da cool so again I haven't played with this model much but I'm going to run and actually go play with all these other models if you dig through here and up and rou and find something neat do let me know because I loaded up some credits right now thanks Louise that's pretty cool right knowledge is power God I love computers all right what's next I had an awesome chat with Devin

Ken the co-founder of custom AI Studio they work on AI agents exclusively basically for their clients and one thing I really appreciate about him and his team is how much thinking they're doing about how to structure the AI AG so that they can deploy in for them and their customers and actually sort of manage it and have high quality outputs in fact deon's one of these guys where after I get on an interview with him I'm usually hitting up the internet team like hey guys check out this pattern this is how our power users are using our tool we should support that better we should consider making that a native feature and so he showed me the mallerie AI assistant which helps him and his team with various things like booking appointments and stuff what I think is really freaking cool about it is the ux of how it works so mallerie has her own and slack account when she's in email thread you just say hey mallerie can you book the appointment and mie reply yes I booked the appointment so there's no onboarding it's basically just like working with a colleague now to be fair Devon and his team have not figured out what they're going to do as soon as they have a client or an employee named mallerie but I have all the faith that they'll figure it out at that juncture in time in any case I think Devon's going to do a much better job of explaining it so check this one out hey Devon welcome to the show hey Max good to be here how you doing I'm doing well good to have you man first off how was your 2024 man it was great I mean like obviously the back half of the Year things kind of took off in the AI space which helped us and uh it seems like everyone's excited for 25 yeah I think that was the sentiment over the holidays was just like new stuff coming out people sharing their rtps by having on good authority that you got a pretty cool use case to show us before we jump into that could you quickly introduce yourself please yeah sure my name is Devin Kars I'm co-founder of custom AI Studio we focus on building AI agents using the nadn platform our main focus with our agents is kind of business Ops a little bit of augmenting an existing product that an entrepreneur is building or building custom like agentic SAS vertical SAS YC would call them vertical agents building those has been part of what we do as well but we're just solving like day-to-day industry agnostic business operations you know pain points thanks for sharing and to give folks a bit of context like how many clients are we talking about well over two dozen at this point wow congrats that that's awesome growth in 2024 yeah know it's been great we've had some Growing Pains along the way but have learned a lot so far and especially I've learned a lot on how to build these agents we learned something new every single week which makes us want to rip up what we had done the week prior and redo it all again from scratch the power of iteration right so for everyone's context Devon he also runs a YouTube channel where he's like putting out some really cool stuff on AI agents I started watching it because I think a lot of people out there are building these kind of oneone videos because they don't really know like they're not actually building I think the nice thing about your agency work is like all of your content is so clear how it's inspired by like actually solving some real requirement from real customer actually make it succeed to get your invoice paid right so that's why I really appreciate your content but I heard that you've got an AI agent use case to show off for us could you give us a little bit of context on it and then why you also built it yeah so we had been dealing with a problem where multiple people would be on an email thread and we'd be trying to coordinate times to get a meetings on the books and we didn't have any virtual assistants or executive assistant on team or team and a lot of those conversations would either go on for days or weeks or would just die in the middle because everyone's busy and trying to get back with your availability and fitting a time slot in between for five people is just a challenge and so we built an agent that was initially designed to be just helping us coordinate meetings and we continued to add some tools to it and some prompts and now it kind of acts as you know our internal rag agent kind of knowledge based agent we have a bunch of transcripts and emails and SL messages and portal chats being fed into it it's in our inboxes booking meetings coordinating calendar events just kind of handling some stuff like as an assistant pretty much got you so it's got like multiple tasks that it can do what are like maybe one or two of the most popular most useful ones you've got it running already yeah so the first is like I said coordinating meetings and the second major one is retrieving Knowledge from the knowledge base and then there are supporting kind of tools and actions that it takes right which is creating updating deleting calendar events or labeling emails categorizing them correctly as they come in you know sending emails replying to emails Etc right and then taking actions whenever directed by the user so we have her in slack as well if we just say hey email this person and ask them if they're available to me tomorrow and then she does it right and then go ahead and book the meeting at that time stuff like that okay very cool I could imagine like it's a lot of time saved especially on its this kind of op stuff that doesn't create business value in and of itself would you mind sharing your screen and showing us an example of it running I'll okay so I'm going to shoot my colleague Welden an email here from a cc mallerie we'll make this a team meeting and is mallerie the name of your AI agent mallerie is the name of the AI agent yep okay got you I'm G say hey won let's get on a team meet tomorrow afternoon mallerie can you check my calendar for a good time have that oh wow so you chat with the AI agent as if it's your VA like there's no like ux or onboarding whatever you just talk with it in line with the conversation with the other parties exactly she's triggered by that's super cool she's triggered by her name so if you mention her name that's how it'll get we use a filter which I'll show in a second that filters out any messages emails that don't mention her name so that's how you kind of like call her essentially that's super cool have you guys discussed what happens when you get your first client named mallerie we no okay so you've sent off the email what's happening right now so let's check executions here M's running very cool and how many times a day would you say that you use maler how many times a day I would say right now it's not super high usage it's ramping up because we've been basically iterating on it and fixing it up day by day right okay gotcha okay so she got back to us she said hey guys I checked Evan's calendar for tomorrow afternoon and I found these time slots please let me know which time works best for you I'll respond to her and say hey mallerie 1 p. m. looks good I talk to well then and he said that time words for him as well go ahead and schedule it okay so the reason why I said this I talked to Welden and he said that time works for him as well is mallerie would have said okay you know Devon's good at 1M now let me reach out to Welden and ask Welden if he's going to be available right so she would waited until he got back with his availability before actually scheduling it so I'm telling you just go ahead and schedule it don't worry about his availability got you so it's like understanding that there's multiple parties that they have to sort of accept it exactly so far I'm just loving how it's literally like interacting with a colleague I don't I think this is definitely the direction for a lot of AI agents if we're going to weave them into our workplace so it kind of feels like you're working with a colleague right I completely agree it's interesting I've been the idea of in inate end having one AI agent node and having all of our separate agents mallerie our executive agent our developer agent our sales agent Etc having all of our agents not be Standalone agents but just be prompts and have a orchestration layer that identifies which prompt to run based on the given scenario and passing that through the shell and the shell just has all the tools available to just run it so that way it can be a much more efficient system I'm seeing quite a few sophisticated NN users that I respect separating their prompts out let's say in an air table whatever and hot loading them in on execution and doing some stuff like that I think there's definitely some benefits especially if you're iterating on this prompt often like as you know you change a couple words it could totally change it having that management in somewhere could be pretty helpful for that yeah by the way mallerie got back to us 17 meeting 1 P pm. to 2 pm tomorrow and they gave us the link right that's really awesome could you show me the calendar can we just see that it's in there as well yeah for sure have it in here team meet tomorrow Saturday 2m we have well then scheduled by mallerie wow very cool so I love how because Mallie has an email as well this happens a lot in automations it's like the Fantom of the person do it you have like traceability of that too right mallerie did this mallerie's an AI agent so we know that our AI agent did this thing exactly Devin could you walk us through how this all happened because this is like super impressive yeah sure thing the way I organized my Inn is by triggers agents and tools so we'll go to triggers so in this instance she was received a new email so we have new email trigger for Aller M so a new email would come in we're basically filtering for just inbox unread emails only she has her own email account we're going to instantly mark it as red and then this is where I mentioned that we are filtering for maler being mentioned because if she's not mentioned then she's not going to get it basically it it stops there then we're basically setting the values for the email and extracting the email thread and cleaning it up so it's easy to understand for the agent itself then we're just doing some data manipulation and sending it off to mallerie and then in here we do have the prompt rather than in the agent because in this case we wanted to easily be able to manipulate it specifically like somebody who's non-technical needed to be able to easily manipulate it and so telling them to just go in here open this up and change the prompt was kind of like much easier yeah the data gets passed through here right we have a specific way that we pass it through the data no matter which data source it's coming from Context which is the me message history content which is the actual email that just came in and then in this case we're dynamically passing through the prompt and when you say you do this structure everywhere is this just for mallerie or you saying for all of your agents across your organization you have this like structure for from the trigger to the agent for all of them we have the structure yeah I can imagine there's a lot of benefits to that what was your sort of main reason for doing that yeah the main reason was to make it easy to set up the data Pipeline and the trigger events and then separately set up the agents and the prompts and be able to use that input data for any agent really instead of adding the new a new uh Trigger or web hook or something in the same workflow or in a separate workflow and attaching it to various different agents if you already have an agent that exists basically and you want to have it be triggered off of multiple different events over time and you want to keep adding those events it makes it easier to scale that yeah makes a lot of sense and I think you're going to be really happy to hear there's an update to execute workflow sub workflow stuff and it allows you to define a schema in the trigger so when you consume it there's like a typed like list of what it expects yeah it sounds like it yeah that would be super helpful yeah because the main challenge is passing data from one place to another and the llms interacting with deterministic key value pairs is sometimes a challenge right so yeah makes sense so we have this payload of data that's from sent from trigger workflow into mallerie could you walk me through like the different tools and stuff that you've got access to mallerie here yeah so we have slack agent email agent and calendar agent and then we have our knowledge base which is basically getting fed our meeting transcripts our client portal chats our emails and our slack messages and these agents here are basically what I used to call Foundation agents I now call them tool agents but they're just agents that have access to the apis that are available for that specific platform so this email agent has Gmail API end points available to it and then calendar as Google Calendar right so I can show you those it's nice it's like separation of concerns there right like the that agent it doesn't have the context of like the job to be done like the users's job it's just like interact very well with Gmail okay wow so really all the different actions of Gmail basically exactly and it's not even supposed to generate any data to be filled in the apis its job is to call the apis sequentially if it's supposed to be in some kind of order right like Mark this email is readed and then reply to it or label this email as a client email and then notify the owner of this contact so multiple different steps can be taken inside the Inbox and so that's what this one will do but it's not generating any data okay got you and I guess the nice thing is if you port this over to a client and if you guys change your email or whatever you got one place where you're updating obviously you know this isn't going to be a five minute jump to swap that out but it's in one place because you kind of decoupled that right like mallerie is interacting with the email agent it doesn't know what service you use or it's not relevant to it requesting that t to be done in that service right very cool exactly we've been thinking about using HTTP nodes instead of the built-in nodes because we can dynamically pass through the credentials which would make it easier to swap out that might be something we do instead later okay makes sense so this is and I'm guessing the other tool agenes you're calling and the email agent the other ones kind of look basically like this it's like they got a bunch of tools all the different actions for that service they don't create data they just go perform that action well there's a series of actions exactly yeah sometimes you got to build your own custom tools right to get exactly what you need from common scenarios like getting open meeting slots this one was used recently by mallerie oh okay because the Google Calendar returns like booked slots yeah so you have like a separate tool to is that because you were getting like variable responses cuz I made like a simple one right without a dedicated tool and it works 90% of the time but it sometimes gets that wrong like inferring the Delta between like what's free and what's books yeah I have it as a separate tool and yeah I the same exact problem right so this one right here just shows when you're actually booked and we were trying to find the time distance between those B times and have it presented in times that are reasonable right like if you're not going to book an hourlong meeting when you have a 30 minute Gap and you're not going to book a 30-minute meeting when you have availability from 9:00 a. m. to 12:00 p. m. you're not going to book a 30-minute meeting at 12:00 p. m. right like people understand when you talk about time ranges that there's a 30 minute buffer time typically you have to kind of like think through those scenarios when you're building a tool like that uh for the agent yeah so let's go back to the execution so we can see the day they get past year and then would you mind loading this execution into the editor sure okay so this came through my email hey let's get on to meeting malerie can you check my calendar for a good time to chat then it hit mallerie inside of here we have the context okay and then we have prompt being dynamically inserted in here and then just like some important instructions to not forget right and then we have email agent we have the content and contacts being passed through as well typically try to just pass through as much as possible and then mallerie's job is to pass through the email parameters for the given action and the also the action that needs to be taken so the email agent knows which action to take it's this is the email agent it's got the calendar agent so it can book the actual appointment it can communicate what it's done be it through slack be it through email I'm guessing the context from the workflow trigger that you sent in that's how it knows whether it needs to be replying in slack or email right yeah exactly after the AI agent step you've got this activity log air table step can you just quickly walk me through that yeah so I like to just track the outputs of all of the agents that we have even the sub agents so all of these have the log as well basically tracking that here in this activity log this is where I keep track of all of the activity of our agents oh nice so you got like a redundant source of Truth as well so Deon first off thanks so much for showing this obviously this is not the V1 of malie right you've a couple versions in you're also like with your agency like thinking about this daily but if someone's getting started out and this is maybe a few versions away what would your advice be for when they're starting out when they're thinking about scoping their own agents it's a good question I would say that where I started was trying to just build a personal agent so I set up a telegram bot and then I grabbed a tools agent and hooked up the llm and then the first tool that I added was Wikipedia and I went into the prompt and just said hey you are my knowledge agent you have access to Wikipedia answer all of my questions and then the output was telegram so I just started chatting with it via telegram back and forth looking at the execution seeing how the data is being passed through and then it was like okay let's say I wanted to tell it to uh schedule a calendar event for me that was the first thing I wanted to do so I added a tool I hadn't built the workflow yet I got an error so I followed the path of troubleshooting the error by building out the workflow more I just worked through the errors basically like adding a node making a change pressing test workflow getting an error copying that error putting in chbt getting chbt bonds making the fix and iterating over time that's the only way so you would say working on iterating through those errors and fixing them was a big part of how you learned all this kind of stuff would it okay good to know so gang when you got a little bug and you're troubleshooting your use case think of Devon's words of inspiration here and so Devin what's next for mallerie yeah so mallerie we're trying to make her more scalable so right now she has access to my calendar and weldon's calendar and uh a couple other people on the sales team but we need our whole team to share their calendar with mallerie and we also want to add some personalization so one is to make it scalable so everyone on the team can reach out to her directly because not everybody's in our mallerie channel on slack but can reach out to her directly she can have access to their calendar to check their availability same thing with an easy place to add personalized aspects to The Prompt meaning you know I typically work from these hours to these hours make sure you only find availability here right things that are kind of like personalized parts of the prompt whereas we would still have the main parts of the prompt for how she performs and takes actions so those two things are the next steps for mallerie that's super powerful I'm just kind of like seeing like The Logical conclusion of that right like as your organization grows you've got folks like obviously who dedicated own mallerie know a lot of details on that and you got everyone else on the edge adding in there two three to 5% of their own hypers specific context but that's awesome dude I'd love to tell that story and show mallerie evolve into this vision of yours yeah let's stay along for the journey let's keep doing this fantastic awesome well Devin I want to say thanks so much man for coming on the show I know it's beginning of the year and for agencies usually it's super busy time if folks like what they see where can they follow along your builds your content and what you're working on yeah you can go to YouTube it's Devin kar's custom AI Studio you can go to our website book a call with us you can hit me up on LinkedIn we're pretty active online you can shoot us an email shoot mallerie an email Maller customi studio. you have to say hey mallerie but that's pretty much it that's really cool I didn't realize that I could email Mario as well but yeah makes sense so everyone go check Che out Devon and the awesome stuff him and his team are working on again every time he puts a video out or post something I'm immediately like saving it or watching it right there so I highly recommend you do the same dein keep crushing it in 205 really excited to see where mallerie and all your other awesome work goes and again thanks so much for finding some time to come on the show thanks man have a good one appreciate it you too how cool is that pattern I told you guys right Devon is serious business I'm very excited to check in with him over the next months and see how mallerie is

developing and before we wrap up I want to give a quick sneak peek on an upcoming NN feature cuz I'm super excited about this one and to be honest any company that says they have an AI agent builder needs to have this to be taken seriously now the thing is I did not get approval to show you this so Martin That's the AI Squad PM sorry about this one mate but it's so awesome and I can't wait so yeah here it is the this is a pretty cool view by the way so the feature is designed with llm evaluations in mind but it's being made to where you can basically set up arbitrary workflow tests for any workflow so you can set up one to many tests and you can even set up evaluation workflows that take the outputs from your workflows and run them through eval so again that could be some basic logic like checking if something was true or false or it could be llm is Judge right you can have an AI agent itself analyzing the outputs and scoring that and whatnot then having it all in a nice little view so super exciting stuff and I think it's going to be a real game changer God leg same game changing and I'll definitely be giving a walkthrough of that one as soon as it's actually

ready all right gang well that is a wrap for this week if you've got an awesome flow gramming use case you'd like to show me make sure to send me a DM on LinkedIn I'm always looking to Showcase hot stuff from flow gramers around the world it's time for me to get back to the support ticket use case but before that I'm going to catch some views up here in toyburg if you were a biner and didn't guess where I am by now you need to get out more but yeah if you're ever in bin check this spot out the really cool thing about toyburg is it's a relic of the Cold War this is where the CIA monitored East Germany so you know not the proudest moment in human history but as you can see graffiti artists have reclaimed it and turned it into what I think a really beautiful example of human expression unfortunately it's not after hours otherwise this is a perfect spot for so until next week and as always happy flow gramming hey hey hey