Build Hour: Agent Memory Patterns

Hi everyone, welcome back to another build hour. I'm Michaela on the startup marketing team and I'm here today with two members of our solution architecture team. Emry live in the studio and Brian joining virtually to help address Q& A throughout the hour. — Hi, I'm Emry. I work as a solution architect at OpenAI supporting digital native customers on building various of AI use cases including longunning AI agents. So today's topic is agent memory patterns which is a very exciting topic in Emry and I's first ever build hour. So if you've been following along, we started with how to build agents from scratch using responses API, then moved into agent RFT and today exploring agent memory patterns. All of the sessions are up on our YouTube channel, so definitely check them out if you want to catch up or revisit earlier builds. Though the focus on the build hour is to empower you with the best practices, tools and AI expertise to scale your company using open AI APIs and models. So for today's build hour, we'll start with an introduction to context engineering, the foundation for agent memory, and then Emry will walk through several live demos covering memory patterns like reshape and fit, isolate and route, and extract and retrieve. We'll end with best practices, resources, and of course, live Q& A. On the right hand side of the screen, you can drop questions into the Q& A box any time during the session. Our team is monitoring both in the room and virtually to help answer throughout, and we'll save a few for the end to go through live. All right, with that, I'll hand it over to Emry to kick things off. — Thanks, Michaela. Hi, everyone. Um so I'll start the first part of this um the session with context engineering definition. So this nice definitions from Andre Karpati. I'll start by emphasizing that the context engineering is both an art and a science. So it's art because it involves judgment. So you have to decide what matters most at a given step of uh a reasoning or action processes. It's science because there are concrete patterns, methods, and measurable impacts to make context management more systematic and repeatable. So, I'll highlight that modern LLMs don't just perform based on the model quality, but they context you give them. In this slide, I want to talk about different disciplines that comes together to basically present the context engineering. Uh so it is a broader discipline than any single technique like prompt engineering or retrieval. So the diagram visually represents the ecosystem of context optimization layers that together shape uh what the model sees and understands. So you see prompt engineering uh as a core principle structured output rack state and history management memory is also a crucial part. So using persistent or semi-persistent storage like files, databases or memory tools to upload and retrieve key information and all of this is contained inside the larger sphere of context engineering. Um so we can also connect these capabilities into different uh product capabilities. Here is a nice summarization slide um that talks about the core principles um like why it matters because longunning tools longunning and tool heavy agents blood tokens and degrade quality uh via uh poisoning noise and and confusion and bursting. We have three core strategies uh as we discussed in the beginning like reshape and fit uh to the context window, isolate and route the right amount of context to the right agent and extract high quality memories to retrieve uh in the right time. We also have prompt and tool hygiene as a core principle. So keeping system prompts lean, clear and well structured. uh use a small canonical set of uh fusion examples and minimize overlap in tools and get the tool selection. And our goal in Northstar is basically aiming for the smallest high signal context that maximize the likelihood of the desired uh outcome. And then in this slide is where our transition from why context injuring matters to how to actually do it in practice. So I'll frame this as a toolkit of techniques. So these are not mutually exclusive. So most real world agent architectures combine multiple strategies depending on the use case uh and the context budget. So the first technique is reshape and fit. We can apply context trimming, compaction and summarization. The second one is isolate and route. uh we can offload context uh and tools to specific sub aents with a selective handoff and the last bucket is extract uh and retrieve um we can talk about memory extraction state management and memory retrieval in that last um bucket when we talk about context engineing it's essential to distinguish between short-term and long-term memory because they solve very different problems uh I think we group uh the first two bucket as short-term memory which we also call as insession techniques and then the last bucket is long-term memory uh we call it cross session. So that means you can um collect different information from multiple sessions uh and you can retrieve back in the next session or other sessions uh in the future. So short-term memory is all about like making the most of the context window uh during an active interaction and active conversation. Uh and then in contrast long-term memory is about building continuity uh across the sessions. Cool. So we often get excited about how powerful our agents are becoming how AI models are getting them better and better. They can handle complex tasks uh route between tools. They can plan multi-step workflows. But underneath all that there is a core bottleneck because context is finite. So every piece of information we add to the prompt instructions um conversation history tool outputs competes for a space in a fixed uh token budget. And this is why uh it matters slide. Uh I want to make the problem concrete here. So I'll frame it around before and after contrast. So you see two conversations. Um what happens without memory on the left and what happens with memory on the right. So on the left hand side um the user started with the issues like Wi-Fi, battery and over overheating in it troubleshooting agent. After many turns the agent has forgotten the earlier context. it falls back to reasking for information that the user already gave. Right? But on the right hand side, the agent remembers the original issues even after many turns. It can pick up the unresolved thread. Uh it references previous actions like firmware update, background sync which makes it feel intelligent uh and reliable. So this is such a stateful behavior which is the foundation of uh a longunning agent. Now I'll switch gears to failure mode. So we can group these failure modes into four categories. The first one is context burst. So you can imagine it as a sudden token spike in one of or multiple components um to do the limited external control or increased calls. Context conflict if there's any contradictory instructions or information in your context. Context poisoning uh if there's an incorrect information uh that enters the context and propagates over the turns. It can be via summaries or memory objects, state objects you're injecting into the context. And then finally context noise. So you can imagine it as multiple um tool definitions or like way more many tool definitions uh coming into your context at the same time. This can be redundant or overly similar items. So that can make a noise uh in the context. Uh here's a nice um visualization of context burst uh in tool heavy workflows. So you'll see that um there will be like a specific increase uh in one specific turn and you'll be injecting uh like large amount of tool tokens here. And then the next one is context conflict. So we can easily visualize it here. So you can imagine in one of the turns uh there's specific tool call and here in this tool call you see that in the system instructions I never issue a refund if warranty status is not active. But in the middle of the turn you're also uh saying that it's eligible refund for VIP customers. But at the end of the turn and your agent is responding, hey, you know, given your urgent travel, I can issue a full refund. So, this is a nice vis visualization or example for the specific um context conflict that can be coming from uh one of the tool results. And then last one is context poisoning. uh so you can imagine it as a hallucination or something inaccurate mixed into the context in any step and propagates across different terms. So here we have a couple possible pitfalls here. Uh losses summarization edits can be causing this. If you're using a free form uh note that accumulates over time that can be contradict and then finally uh older summaries override the newer ones and you'll be basically uh causing any hallucinations uh because of the summary logic and you'll be injecting that hallucination into the context and propagating uh over time. Cool. Now I'll stop sharing my screen and switch to the demo that I prepared for you. Uh and then I'll go over all of these some of these challenges um to show you like how it actually works in a real world scenario. Okay, let me share my screen.

Cool. So here I prepared a demo app for this build hour. It is an ID troubleshooting agent for software solving issues uh for issues related to both software and hardware. Uh and this is a dual agent demo that lets you run two agents side by side. So backend logic states inside the nextJS app and I'll be using OpenAI agents SDK. So we have two tools connected to that agent. One of them is get orders and other one is get policy. So here I can start sending a message uh and saying hi to both of the agents, right? Uh and then you'll see that both of the agents are responding to my message and then here I can say hey my laptop fan is making weird noises while I playing games. Is it normal? So here you see that the configuration for both of the agents are same. They're at the same model, same reasoning level. um but I'm sending the same message and there is no memory um basically configuration yet. So here you also see the context usage bars at the top. So you're seeing different type of components uh that is already in the context now. Um and I can say hey before I want to see my orders my order number is 1 2 3 uh four five and so now I'm expecting the model to make a tool call and show me uh the orders I have. So here you see that the order status the items I have and it's powered by a specific tool called. So as you see over time um it will accumulating different tokens and different type of tokens here and [clears throat] in the context life cycles I'm visualizing what is happening under the hood across multiple turns. So here you see that I have uh 84 tokens of system instructions. My user input is increasing uh slowly. Uh but the core component here uh is agent output that will be generated by a model. Cool. Um so this is a typical real world scenario. So now I also want to showcase like how context burst is happening. So I can still uh start with high um and I can say hey this time I'm having an overheating issue on my laptop uh and then the model is responding to my to my message to my issue basically uh and then here it's telling me like specific um instructions and it's asking me some questions to better understand uh what's happening right and then I can say hey thanks before that I want to see the refund policy of my MacBook Pro 2014. So while I'm sending this message, I also want to quickly show you the code and core concept of or how it's working. So it's powered by open agent SDK and here you see um the agent definition. So it's a customer support assistant. Uh I'm having specific instructions here um that I'm adding. Uh I'm using like different models here. Um, and also I can show you the system prompt really quickly instructions. Um, so it's basically I'm saying that hey you you're a customer support assistant for devices. Uh, and then I'm using like very slight prompting and instructions for that specific uh agent here. So let's go back to um the response. So, since I asked about the specific refund policy for from a MacBook Pro 2014, uh it's made a tool call get refund and it's basically returning uh a specific um refund policy that I added before. So, here you see that in between turn two and turn three there is a specific spike um in the context window. So, in turn two, I I had maybe around like 300 400 tokens, but now I have more than 3,000 tokens because I am just dumping lots of information into the context. And then this is a nice example for context burst here. um instead of maybe just dumping all of these information into the context uh as a refund policy, I can be more careful about my tool definitions and tool outputs um to basically make a decision about like what should I inject into the context. So maybe not all of these information are valuable, but as you see that I'll be injecting lots of information into the context that's visualized here uh in this context life cycle tab. Cool. Now I'll stop sharing my screen and go back to the deck. So I'll continue uh with the next uh steps here. Okay, nice. So, we talked about challenges and what's going on under the hood and a specific example about context. So, now let's talk about the solution, right? Um so the solution is basically uh managing context efficiently using different techniques such as um tming compaction uh state management memories and make the natural step beyond prompt engineer. Uh again this is also another visualization about different components in the context. Uh so you see that across the times it's increasing the token counts are increasing and these tokens can be coming from the system message user message maybe you might be injecting uh memories or different uh type of specific tokens that can be added into your context. Uh here I want to group uh AI agents in terms of context profiles. So we can group them into three categories. So first one is rack heavy assistance. So you can imagine reports policy QA agents. Um in these type of agents context is most dominated by retrieved knowledge and citations. The second one is tool heavy workflows. Uh context is mostly dominated by frequent tool calls and returned payloads. And last one is conversational concier. So you can think about planning agents, coaching agents. uh and in this case context is mostly dominated by uh growing dialogue history. There'll be lots of tokens in conversation history like assistant usage tokens that scales with uh with session lengths and then to better understand the solution and uh the techniques uh we can go over what is fixed in our context and what is dynamic and a variable. So here you see different type of components like usually system instructions tool definitions and examples unless you're doing a rack uh approach is mostly static in the context. What is dynamic is tool results retrieved knowledge memories and conversation history. So these are the nice examples for dynamic u and static context and tokens. uh so you have a control on dynamic tokens and you can apply different techniques to control it efficiently. So I would like to start with prompting best practices to aid avoid context conflict here. Um you can also find it from our prompting guides and cookbooks. Uh the first rule is being explicit and structured. We suggest you to use clear direct language and specific enough to guide action. uh you should give room for planning and self-reflection. I think this is becoming more and more important with uh reasoning models like GPT5. Uh and you should avoid conflicts. So keep the tool set small and non-over overlapping. Don't use uh ambiguous uh definitions. Even if a human can pick a tool, the model won't either. So be careful with conflicting instructions and tool definitions. So for context noise, we talked about like many tool definitions and many tools attached into the context as an example situation. Um again, you should be explicit and structured in your prompts. Um more tools isn't always equal to better outcomes. Uh so favor targeted tools with clear tool decision boundaries and then return meaningful context from your tools. So in the demo um you show that specific uh example of context burst. So we suggest you to control basically what is the tool output uh and then return high signal semantically useful fields um and prefer human readable uh identifiers. Nice. So now I'll switch gears to engineering techniques and I'll start with the first one which is reshape and

fit. So the first technique here is context trimming. So it is a pretty basic technique. It basically means that dropping older turns while keeping the last uh end turns. Uh here in the turn we have like limited context. Uh it's getting noisy. There are lots of information in the context. It can be coming from a tool user message or different sources. And there's higher likelihood of losing track because uh we are getting close to context limit. But once we trim the older uh conversations, older messages uh now we have fresh context. It has better attention uh and you'll see that uh it will also increase the latency uh that you're using. So it basically keeps the last end messages and trips the previous older messages. Uh and these are the some parameters and knob we have control over in context trimming. The second technique is context compaction. uh it basically means just dropping tool calls or tool call results from the older terms while keeping the rest of the messages. So if you have a tool heavy agent, you can consider these techniques. Um you'll see that your context will be most dominated by tool results. It will be noisy. There will be maybe some context noise and lots of information coming from different tools. And after compaction you'll see that there will be fresh context uh better attention and faster uh processing uh and you will also be keeping the tool placeholders uh intact after even after the context compaction here and a question you might have uh would be like okay how can I decide heristics about trimming and compaction uh so here I can share a couple suggestions so first you can analyze your sessions uh you can collect like context snapshots from production or from your users. You can collect times down and dislike context to see what's going wrong there. Think about the average token size of a context. Uh what type of task do you have in one session? Secondly, um like do not trim mid turn and break turn blocks. So a turn basically means that a user message and all the other message until the next user message, right? Uh so if you just break or don't respect these turns uh there will be higher likelihood of losing track. And then finally don't wait to hit context window limits. Uh so keep track of context allocation. You can set thresholds like 40 or 80%. So if you're getting closer to hitting the context window limits, uh these thresholds will help you to better understand basically uh when you should trigger some of these operations. You can control tool outputs uh and you can also keep track of token saves. So these techniques are also really nice for cost um cost reducing cost purposes. Uh, and you can always keep track of like how much token you're saving while you're increasing the overall capability of the of your agent. And then the next technique we have is context summarization. Um, so it basically means compressing prior messages into structured summaries and you can inject into the context history. So here uh in turn n you see lots of messages uh like noisy context again and you're basically keeping the last n messages and just summarizing or compressing the previous ones um so that you'll have fresh context uh better attention faster processing and at the end of the day you'll have a golden summary. So this will be like a valuable information because you'll be basically compressing all of the valuable informations. Uh and at the end of today it will have like a very dense um object that you can keep track of and that will be also useful for you to better understand what happened in the conversation. And here's a nice visualization uh in the context life cycle uh about the summarization. So let's say you perform the summarization a specific turn. So you'll see that you are compressing uh all the previous information and injecting it back uh into as into the context as a memory object. So you see there is a new component um called as memory after the summarization performed. Nice. Um so here is a comparison of summarization versus trimming. Uh so there are different dimensions that you can consider while you're designing uh a memory pattern for your agent. Um you can see that in trimming uh you'll just keep last entrance and you'll be dropping the oldest ones. So basically it'll be pretty straightforward operation. So it'll be very fast uh you know and there's no latency. Uh but the trade-off is that you might be losing some information uh that is already there. So I think this is the main trade-off that we have. Uh it's really best for tool heavy ops and short uh workflows. And then in summarizing, you're basically just keeping track of all the information. So you you're not throwing away any anything. Uh this can add a little bit of um latency and cost because you'll be doing another summarization call to a model. Uh but you'll be just collecting all the information. So you can think about a uh an agent or use case. If you have like multiple tasks in your longunning agent that are independent from each other, you can definitely consider u trimming because probably the trimmed or throwed away information is not important for the agent for the next turns. But if you're collecting useful information across multiple turns and tasks are dependent to each other, then you can definitely consider uh summarization here. Nice. Uh so now I'll stop sharing my screen and go back to the demo. I'll show you couple examples of these techniques that we already covered here. Let me quickly share my screen. Nice. So here let's go to configurations page uh on the demo. So for agent P I

want to uh enable trimming and I can basically set max turns S3 to trigger uh trimming operation. I want to keep recent turns uh as S3. So here I can start again to test my agent and saying hi. So this time I want to understand the refund policy um that I want to check. Maybe I want to refund the laptop I just bought. I can say, "Hey, I want this refund policy for my MacBook about a month ago. Uh, and I want to understand like what's happening here um in terms of refund. So if I'm eligible or if I'm not eligible. So the model is now making uh a tool call uh and calling the get refund tool. Here as you see it's returning that specific information. I have 30 return window for uh for returning that specific laptop. And I also want to check my order. So I changed my mind and I'm saying, "Hey, can we also check my order? My order number is 1 2 3 4 5. " Uh and I want to see like if it's on the way, if it's coming. So you see that the model is doing another tool call as get order. And now I want to switch gears. I can say, "Hey, thanks. I'm having an issue with the internet connection. So until this turn, you see that I have lots of tool tokens here in the context in context life cycle. So it's getting uh accumulating over turns and in that specific turn now it's telling me that hey let's sort it out. It's asking me a couple questions um about my device and I can say hey I tried to load an internet page and still see 40 404 error. Um I basically share a couple uh important information about the situation and you see still see that the across the turns it's accumulating lots of tokens and even agent output is increasing. Okay. So let's say it's still happening on Safari. Um and then this is the last message probably uh that I want to share about the current situation and I'm waiting to have more guidance and instructions uh here. And here you see that at the end of turn six the context is trimmed. Uh if I go back to here to visualize what's happening. So when I hit the turn six, you see that it trimmed the context. So it basically removed all these tool outputs and tool tokens. So now I have a fresh context here. Um and then now I can continue talking about the same specific issue or I can continue to talk about like different information. Nice. So let's go back and then here I also want to show you how summarization works. So also the compaction is also works in a similar way as a trimming. So here I can set uh like compaction trigger as for and keep recent turns too. So you'll see that at the end of turn two, it'll be compacting and removing all these tool outputs and I'll have a fresh context similar to the trimming approach. But now I want to be a little bit more advanced here. So here I want to enable summarization. Um I want to set the summarization trigger as five and I want to keep the recent three turns. So here I clicked save and now I want to just see how it's summarizing all these information. So here I'm sending hey I I'm having internet connection again. So this time I decided to share more and more information about my situation. So I can say where I bought this computer um what is the model. So I can say, "Hey, I have a 2014 MacBook Pro 14inch and I live in the US, but I bought it from Amsterdam. I received it from battery change service and just updated the OS version last week and they asked me to update the OS version to Mac me OS Seoia. So as you see, I'm sharing like many information uh and these informations are really available for an IT trouble troubleshooting agent, right? Uh and here I can go back and say okay these are nice clarify the problem uh like what I need from you and I can say hey I already tried hard reset after checking the FAQ docs but it didn't work. So this is still in a form of memory because I'm sharing like which steps I tried which worked and which didn't work. I can go back um and continue talking with the agent. So now it's reasoning itself and providing me like more detailed guidance uh and instructions for specific to a MacBook. So I go back I went back to my computer and I saw that the Wi-Fi icon is not active. Um and then I'm thinking maybe it's related to Wi-Fi or a specific um software issue. So, as you see across the turns, it's getting more complex. The the agent needs to reason and the agent also needs to keep track of what's in its context and make sure that it's not uh there's no burst, there's no conflict. Um there's no poisoning and other type of failure modes. It's telling me uh some specific steps and I can say, "Hey, I tried it already and I'm wondering if it's a specific software issue, right? " Um and then here I'm waiting for the response from the agent. So again, I shared lots of information here. Uh lots of available information. So now the agent knows my device, where I bought this device, which steps I tried. Um and basically what I what type of um steps I performed before uh doing that. Cool. Uh so now you see that it's responded to me like a very well structured instructions given what you described. Wi-Fi icon is not active blah blah. And then here I see that the context is summarized and I notice that there is an orange uh component we count as memory item and the memory item is basically um the summary uh that we had. So here between turn four and turn five you see that I'm condensing some part of the context and I'm injecting it back as a user message uh as a memory component. So again here memory is basically um the summarized context from the previous terms. So now I want to go back to the code and show you the summary prompt uh and go over like some important topics about this specific prompt. So as you see uh here's my summary prompt. I'm saying hey you are a senior customer support assistant for tech devices setup and software issues. Um, and then before you write, I'm saying, uh, hey, be careful with contradictions. Uh, make sure you are having a temporal ordering and make sure you're having a hallucination control. So, I think these are very important things to consider when writing a well-crafted summarization prompt. And then I'm tying this summary to my specific use case. So, I'm saying, hey, in your summary, write a structured factual summary. And then just think about product environment reported issues, what worked and what didn't work. Uh which steps you tried, include identifiers, which is important, key timelines, uh timeline milestones, tool performance insight, current status, and next recommended steps. So this is a really nice example of how to craft a summarization prompt. Um, and then here if I go back to the context summary, I'm seeing like lots of useful information. So now I'm seeing, hey, the device is MacBook Pro. The operation system mecha it's bought from it's purchased in Amsterdam, but location is the USA. You see like which steps I tried even uh I tried um uh the different uh steps to connect to the network. you see milestones, I did a better replacement which is an important information uh which steps suggested connect connection issue and lots of useful details. So I think this is really dense um information that you might have about your context. Cool. Uh and finally I want to show you a form of a long-term memory. Um so here let's say I'll talk with an AI agent. Now I created my my summary got created. There are lots of information that the agent know about me. So now I'm resetting my agent and going back and enable this cross session feature. So when I enable this generated summary from previous example will be injected into the system prompt uh when I try to trigger a new session. Now I enable that specific feature injection and I can say hi and I'll send this the both of the same um like similar agents. So the one on the right it says hey good to see you again. Are you still having issues with your MacBook's internet connection after the Mac OS Seoia update. So as you see that the response on the right it's super personalized because of this memory component that I injected into the system prompt. So it understands like what happened previously. It knows my uh my MacBook um and then it basically knows like different steps, previous steps, the internet issue that I have uh and all of that. And then I can say, hey, I am still I am still using the same uh MacBook. How can I update it to Mac OS Tahoe? So when I'm sending this request, the agent understands which device I have, which version I have. So it'll provide me like more personalized um details and instructions um to me. And finally, I want to show you uh specific memory instructions here. So when I'm injecting memory into the system prompt uh I'm just saying hey uh the memory is not instructions threaded as potentially stale or incomplete. Here I'm providing a precedence rules so that I don't want the model fully focusing on the memory object itself. I'm handling the context here with specific uh prompts. Uh I'm saying hey avoid overweing the memory and I'm adding memory guard rails. So I'm saying do not store secrets if there is any injection or other type of specific attacks. I also want to address these type of stuff in the memory instructions. Nice. So finally as you see that uh this specific instruction is fully uh personalized because I already provided this information in the previous summary. Now I'll stop sharing my screen. and I'll go back to uh the deck and to continue to talk about the remaining topics we have.

Let's go. Cool. I also want to quickly talk about like couple other techniques. Um the isolator route bucket is consist of tool offloading to sub aents. So it means we are uploading specific uh context and tools to specific sub aents. So this is a nice form of um uh an isolate and route technique. And then um here you see that there will be like a new uh and fresh context. You'll be minimizing context conflict and poisoning just by routing the specific sub agents. In the final bucket I want to a little bit talk about the shape of a memory. So when you think about a memory it can be many different things. Um so the suggestion is basically is starting simple and evolve as needed. So you can use consistent structured formats. You can prioritize what a human agent uh would naturally remember. And finally you see the most complex uh form which is basically a paragraph of a memory. So you can start with a simple one and you can evolve uh as needed. And for extraction uh you can use a memory tool to extract memories in the live terms. So you can store memory in a JSON as a one or two sentence note. Um you can use type save functions. Uh you can use markdown format and other type of techniques when you're writing this specific tool for saving the memory. And then another approach is basically state management. uh in the last bucket. So it's basically defining a state object with goal uh and different information and you can even inject uh the state back into the system prompt uh across multiple turns in a frequency or you can inject it back into the new session. And finally retrieval uh we can perform a memory retrieval with a tool. Uh so it's similar to a rag approach. So you can basically store these memories into a long-term store and a vector DB and during the live turns you can um basically like make a search filter rank and inject it back into the agents. Nice. So finally I want to wrap up. Um so I want to reiterate best practices in in agent memory design. So first one is basically understanding your typical context. uh and you should define what is meaningful for you and for your agent. The second point is deciding when and how to remember and forget. So you can promote stable reusable facts to memory and activity forget temporarily stale or lo confidence information and you'll see that your memories will be evolving uh over time uh and you can continuously clean merge and consolidate memories uh and you can optimize these steps uh in iterations and finally evals is also super important. So you can run your own evals to see if there is any uh improvement with memory on and off. You can even build your memory specific evals for long running task and long context. — Awesome. With that, let's move on to some Q& A. We've had a ton of great questions come in. So why don't we re refresh the presentation and we'll pull up the next slide and get into a few.

— Nice. Okay. So let me go back to the Q& A session and jump into the questions we have. So let me quickly share it again. Nice. Okay. Yeah. Let's start with the first questions. Yeah. So there are li any libraries or packages uh to recommend for context engineering. So this demo is built by using uh open agents SDK package and library. It gives you a really good flexibility um to implement your own sessions. Uh and in these sessions you can easily implement like trimming, compaction, summarization and all of that type of uh techniques easily here. Um so I see many different libraries uh that are evolving really fast to basically um make your life easier for context engineering. So as you see we have too many um techniques and each technique has different parameters to tune. So I also see that uh there is an uh evolving uh part of the all of these um libraries but I can suggest open agents SDK as a starting point uh to basically start implementing specific uh context engineering techniques uh and go from there. Nice. Next one. So how do you evaluate or measure uh the memory feature is evolving uh the memory is improving performance. So this is a really nice question. Um yeah after this the session you might think about hey you know I implemented a specific memory approach but I don't know if it's good or not how it's performing well. So we can maybe split this into couple um uh portions. So first one is basically just running your regular evals with memory and without memory. So I think this is a really nice way to start thinking about if memory feature works or not. So if you have some specific eval metrics like completeness uh like upwards downloads or that type of uh numeric metrics you can see if there's an increase or decrease or if there's any statistically significant uh uplift coming from the memory uh and then maybe your evals might not be capturing well that type of um memory based boost or improvement then I suggest you to think about more memory based eval. So what I mean by memory based eval is basically uh evaluating the model on long running tasks and long context. So if you are not hitting any um context thresholds maybe uh your agent uh doesn't need any of these memory um improvements at all. So again you can start with your core evals if you have already and then secondly you can start creating your own memory based evals. So you can even evaluate the quality of the summary, you can evaluate the injection time, prompt. So there are different ways to evaluate it. Uh but of course in most EVAs you also might need to prepare a golden data set first and think about maybe like couple 50 examples like golden examples of a good summary or you can try different horistics that I mentioned before to basically find the right balance of trimming uh and compacting. So I think we can just group this into three different buckets. First one is running your own evals to see if there's an uplift. Second one is building memory specific evals. And the third one is basically finding the right heristics uh and parameters to apply in the context engineering techniques. Next one. So should we use hierarchical context like entire project context for immediate task and context for immediate file edit in questions? Um so yes the qu the answer is yes but it's mostly dependent on the use case. So we also have a concept called memory scope. So you can think about this memory scope as a global scope that means if you have a customer or user of your agent probably there are some information that you should always remember about that specific user. Maybe this user likes more friendly tones. Maybe this user lives in the US. So these are some examples for global memory. Um but you can also have um uh a scope based on the specific session. So let's say I want to uh book a travel uh and then this time I prefer window seats because I want to sleep. Uh so this also a nice example about the session scope and session memories. So I think this is a good practice to maybe separate these into two buckets and you can keep track of uh session memories with session scope and over time you can graduate session memories into global memories and you can keep track of like what is really important uh about uh the specific user. So in travel concier example if user is always saying hey this time I want window seat uh like maybe multiple times and you can finally graduate that memory into global memories and keep track uh keep it in agent's mind basically and remember that for the next uh next bookings. Nice. Okay. So what strategies do you use to keep memory flash or prune so the agent doesn't become overloaded with stale or yeah this is a really um this is another good question uh and in the real world you see that memories are evolving really fast so after some time you'll see that there are some memories that you need to prune and the agent needs to forget. So in that specific case there are a couple techniques to apply. So first of them first of it is basically keeping a temporal tax. Okay, I learned this memory uh from the user but I learned it maybe like two months ago. So if you can keep track of these um timestamps uh or temporal tags, the model will understand what is old and what is new. So if I say I like dogs, if I said I like dogs like two months ago and today I say I like cats. So you'll see that the model is going to understand my favorite animal now is maybe cats and it will override the memory um with the right instructions. So this also falls into a little bit to memor consolidation. So how to prune uh stale memories, how to basically update uh and override the new ones into the existing ones. So temporal text is one technique that you can apply. Uh the other one is you can use a way decay or um basically a window function uh and you can basically uh focus more on the recent memories uh and you can basically downgrade the oldest ones. So it really depends on the nature of your use case. So if you think that what I said a year ago is not important for your agent, uh you can definitely prune these old ones and implement a weighted average probably for all your memories. But if you think that all of this memory is equally important for your agent, then you can consider maybe like memory consolidation and memory override with temporal text. So we can talk about two different uh techniques um to manage the overloaded and stale uh memories. Nice. Okay. So how do you manage scaling agent memory systems when you have many users with individual and shared memory pools? Yeah, this also another good example from real world. Um so once you see the memories are uh evolving over time and you'll see that you're collecting tons of memories from um from your users. Um so there are different ways to scale it. I think the first path or first decision criteria starts with if you are basically performing uh a retrieval or search base long long-term memory approach or you're just using um basically summarizing the context. So if it's the second one that means you're just storing all of this information and persisted into a disk. So you can think about some scaling methods about like data management how to manage like a large amount of memory nodes uh as a text in a text format or you can basically think about um scaling the first approach which is basically you have to think about like how to scale a search uh and retrieval system. you might be storing all of this information um into a vector database and then in this vector database you can try to scaling uh the storage you can scale all these vectors filtering ranking system and all of that so I think the first bucket is mostly about this long-term memory so we talked about like memory as a tool so if you can think about extracting memories with a tool and retrieving back during the live turns probably this is the situation where you're going to uh hit this question about scaling uh for many users. Uh in this case you can think about u scaling techniques for vector databases. Uh you can use shorting you can optimize your embeddings model probably if you're using like customized embedding model. Um and you can basically optimize retrieval process similar to a rag approach. Uh again the first one uh is mostly about scaling a retrieval system. Uh the second one is mostly about basically like data storage, how to store specific data, how to manage uh like tons of information and sentences. I think to wrap up uh we can basically put it into two buckets. One of them is uh scaling um and optimizing a retrieval system. Uh the second one is also u making more efficient for storing and persistent uh in the disk. So this is also a common question that I hear from my customers. Um I think you can maybe follow like a like pilot approach and you can turn on this new uh memory techniques for for a subgroup of your users and you can think about okay how it's evolving over time. Maybe you'll see that most of the memories that your users are saying are pretty limited. So think about this travel concier agent. So probably I'll just sharing my memories about my seat preference. Maybe if you want to book a hotel, I like maybe higher floors. Maybe I like the specific menu or breakfast. Uh so I think this is more limited type of groups uh type of memory possibilities I can say and that type of agent. But you are if you're building a life coach or life coach agent. So there are tons of memories that you need to remember uh about me my life. uh and you'll see that these type of memories and memory pools are evolving really fast. So yeah the third point is that try to understand the evolution of memory and possibilities of memory in your AI agent. So we have two examples here travel concierge memories and then life coach memories. So yeah as you see in the second one you'll be collecting tons of information that is valuable for uh yeah for my life. Um and then my dreams, my goals, uh what I was thinking a month ago or a year ago. So the second one is mostly like super advanced and complex and sophisticated memorable that requires uh lots of scaling uh for sure. Okay. Um so yeah, that was the end of the question the Q& A session probably. — Yeah. — Okay. Um yeah, and then we can just switch to resources. So um all right, this has been awesome. To wrap things up, um we're we've linked a few great resources here, including the context engineering cookbook, which was referenced, and the context summarization cookbook and our agents Python SDK. I know we've gotten a lot of questions on is this available in GitHub. So you can explore all of these links on the right and the full build hour repo is available on GitHub. Um, so good news. We're likely going to squeeze one or two more of these in before the end of the year. So keep an eye out on our build hours page linked here. And um, a big thank you all so much for tuning in and a big thanks to Emmery who's did an amazing job with this session. — Yeah, thanks everyone. Uh, we hope you enjoyed uh, this build hour on an agent member patterns. I know we covered like lots of different techniques uh lots of different information about memory, how to think design memory. So, so overall as you see there are too many options but the core uh idea is basically better understanding what your agent should remember and how it should forget. So you can think about these three things when you're designing your own agent u memory. Uh and this is still an evolving field. So you might see like some um like new features coming uh about memory overall. Uh but yeah, I just wanted to show you like different design tradeoffs uh and guide you with the best option. So finding the right balance between these techniques are uh usually like related to your specific uh use case. And then you can keep track of all the news and cookbooks uh in the resources section. So, I'll be also upload uploading this um demo page uh so demo application to our build hours GitHub. Uh and then yeah, thank you for your time and thank you for listening uh all of this. — Yeah, have a great rest of your day and we'll see you next time.