OpenAI DevDay 2024 | Community Spotlight | Genmab

we are uh so excited to be here today um to talk about how we used AI agents to help the clinical trial process get a lot faster um I'm Scott I lead our AI Innovation team at genmab and uh the only reason I got the time to come here is because my 5-year-old was super excited I got to meet Chach BT in person uh cuz he tells way better stories than daddy does so Sam I'm Sam Wagner uh Scott maybe after the talk we can meet Mr chat GPT I think we should um so jenm we are a biotech company we are an innovation company but focused on biology um but that culture has permeated and now um genmab is committed to not only being the best at biology in any bodies but also to not just adopt AI but try to push it forward so this is where our framework comes in to help something we think is really important the clinical trial process uh for those who don't know is super long and super expensive eight years or more billions of dollars for one medicine in one disease there something has to give to scale this and we think that AI is well is way well positioned to help here so we're going to tell you a story about how we did this for very specific use case for document generation regulatory documents that you have to submit to the government one of these things you can think of as the patient story so um for every patient on every trial you have to for every day that they're on your trial you have to generate a very specific clinical document that takes a skilled Workforce a significant amount of time to triangulate they have to go to hundreds of different documents or P hundreds of different pages of documents thousands of data points compile that all together with their skills as clinicians right and generate this summary this is one of many documents by the way we can go into more if you want um for thousands of patients this takes significant amount of time and often it's not just your internal stakeholders and Company and data but it's external Partners too that you have to work with and you know gbd4 just prompting can't get you that we for a regulatory document you need to be 100% accurate 99% is not good enough um so we're going to talk about how our framework which we call Kelly um go can get us that last mile uh to do this so we uh our high level architecture here sort of explains how and Sam's going to show you this in a minute our um language model takes in sort of real world real natural language the user story of the task um plans out the future in context knows what step 10 is going to be when it's executing step one um Can self-correct Has a guideline has a um a way of evaluating how each step um executes and performs and sort of is able to tailor and customize the plan going forward now it takes that and everything it does all the tools that it calls in Step One is the input is the Step Zero of the next step and we can sort of iterate this kind of ad nauseum and we show that you can Converge on that 100% accuracy that we're looking for um we've also shown and this is going to be published soon that it can solve generic problems where you know how to evaluate the solution um but we can go into that later but I think right now I'm super excited to hand the floor to Sam to show us uh Kelly in action thanks Scott all right so I'm going to kick off Kelly here as it's loading um what I'm going to show you here is um the points that Scott was talking about which are um that we're able to learn the patient as we go along that it's drafting step by step section by section and as it's doing that it has a retrieval process to get the information that it needs so the first thing that it does is give us a system message here um the way that we initialize this is that we have a series of prompts uh specifying the job right so the first thing that we do is and this is a compilation of the job description which you write beforehand so the first thing that we do is we set up a role and objective to say that you're going to draft this particular document we set some criteria for what the tasks are so this is a list of tasks that need to be done sequentially so you can think of this as a checklist right so it's going to complete the first task move on to

the second third and so on and as it's doing that if it's not able to do a task that it's able to uh solve that um the medical writers and the clinicians give us guidelines to how these things are supposed to be drafted so that's included in the instructions in the prompts we also have some prompt completion mechanics uh which is an essential part of Kelly this is really part of the secret sauce so what we tell it is that as it progresses that it needs to tell us what's been completed what it's working on and what it's going to work on next so first thing that happens here is we sent this to this system message to GPT it responds it knows that the first thing that it does needs to do here is do a function call for the particular identifier for the clinical trial what's interesting here is that any kind of IDs or keys that you retrieve it's going to stay in context so you can do any kind of key value pair lookups in your function calls at any time so we sent that to GPT it response tells us that the first task has been completed it's working on task two proceed to task two so it's going to instruct itself the next time that a call is made so it knows to keep going here so it does a function call it retrieves the table here those results are appended to the context so the contexts keep building up here now we have two tasks that are done so the interesting points here um is that it's that has a evolving narrative here it tells us what kind of information that it's gleaning from the particular retrievals and the other part of this too is that it knows how this information is going to be used later on so this is Task two right now and it's telling us later it needs this information for drafting so it can see ahead to task 10 because all of that information is inside of the system prompt altogether so just as a reminder every time that this is uh done sent to GPT the system message comes along with that so it has that blueprint the entire way so we're going to skip ahead to where it starts to draft at this point it has all of the tables that it needs so it starts to write here's the background section here it writes about day one and it keeps doing this keeps writing every section until it's done with that and it's time to compile it it's compiled it Al together and one of the reasons that we do that is that we're very accurate when we cut this up into small smaller sections that it writes and it's able to glean all the information that it needs from the context so here we save that there is a monitoring agent at the end of the process that confirms that it's been saved everything's been done up to that point in sequence like it's supposed to and then we have a complete draft which looks like this all right back to you Scott cool so as I'm pulling this up um what Sam showed uh could take hours but his process takes minutes um and we that we're talking about thousands of patients on many different trials for many different days lots of other documents too and um and if you think about it like the ultimate goal here is if you shave a month off of a trial let's just say a month that's literally hundreds or thousands of people that with serious disease they can have access to our drug they would not have been able to because of how serious the disease is so that's what like gets us up in the morning and to come to work right is that problem um and if you're interested in working on those problems with us like QR codes uh open source Kelly as a problem solver as a document generator is generic we think um also uh on the right if you're interested of helping people uh with serious disease uh that's for you so thank you very much we're we've been so excited to be here today thank you