Real-Time Medical Transcription Analysis Using AI - Python Tutorial

in this video we'll build a medical transcription analysis app which will be able to take in medical spoken data in real time and also understand key information within it let's take a look at exactly how our application will work I'm going to click the button to start recording and I'm going to pretend to be a doctor talking about a recent patient visit that I've just had today I met with patient Mr Garcia who is a 45-year-old man he has been dealing with a cough for about 2 weeks and he mentioned some mild chest pain he did not have any fever and his breathing was fine however I've prescribed a course of antibiotics to cover any possible bacterial infection I've also ordered a chest x-ray and some blood work we'll be meeting again next week to follow up to see how he's doing so our application transcribes everything that I'm saying in real time and it also identifies key medical information such as medical conditions uh any medicines which are being prescribed or any tests and procedures that are needed to build this all we need to do is make use of assembly ai's realtime transcription as well as its large language model framework lemur which enables us to use different large language models for example clot 3. 5 Sonet there's two things you need to do to start building this application first off you want to download the GitHub repo for this project and I'll be leaving the link for this GitHub repository in the description box below and the second

thing you want to do is sign up for a free assembly AI API key the link in the description box below will allow you to do that and it also gives you $50 worth of free credits to get started there are three important libraries that we need to download for this project first of all it's assembly AI so you want to run this command pip install assembly AI extras in your terminal and also you want to install Port audio as well so let's actually go ahead and copy this and head on over to terminal I've went ahead and created a virtual python environment where I will be downloading all of these libraries so once I'm there I'm just installing the libraries that I needed so I've installed assembly AI extras and also now I'm going to install Port audio finally we also want to install flask now that you've installed these three libraries we can head on over to visual studio codee open up that project folder that you've downloaded from GitHub and we can start writing our code once you're in Visual Studio code in the project folder that you've just

downloaded you should see three main components first of all it's app. py where we write the main logic of our code and then next off is index. html which has the HTML code for our UI and also the communications between our app. py file as well as the front end we also have styles. css which just contains the CSS for our UI now let's go ahead and open app. py I've broken down our code into six different steps 1 2 and three are already written and we will be writing steps four to six before we start that let me actually walk you through the first three steps so in Step One what we're doing is importing all of the Python libraries that we require for this project and that namely includes flask and assembly AI next we also want to Define our assembly AI API key so here's exactly where you should be doing that next we have a few Global variables that we want to Define at the very top which is our transcriber object as well as our session ID in Step number three we have our prompt so this prompt is really important and what it

does is it essentially tells the large language model that you are a medical transcript analyzer your task is to detect and format words and phrases that fit into the following five categories and these five categories are essentially what we're looking out for in the transcript and what we have stated in our UI as well so that includes protected health information Anatomy uh medicines and also tests and procedures so for each of this category when the large language model is identifying them it should format the text that it returns to us when it receives a transcript and it processes them it also makes sure us to format that text with html text and that's essentially how we are able to display the highlights or different formattings on our application now let's move on to step number four which is realtime transcription first off we are going to start off by defining the real time transcriber so this is the code for defining our realtime transcriber object in this we Define our sample rate as well as what actions we require it to do when it receives data when an error occurs when we first open the real time transcriber and when we close it so these are all methods which are right here and we'll be filling them out right now in the onop method we want to define the behavior of our realtime transcriber when we start a session here we want to define the session ID next is the on data method in we want to Define what we want to do with the real time transcript which is coming in so let's actually start writing that out so what we have just defined is that if we don't receive any text or if we receive empty text don't do anything next in the event that we receive a real-time final transcript which actually refers to a fully uttered sentence or whatever sentence that you have said before you took a break of at least 700 milliseconds so all of that text what we want to do is send that full final transcript right into this method called analyze transcript now in the else Loop what we're saying is hey if you didn't receive a final transcript so you're getting a partial transcript and in this case a partial transcript is word by word what you're saying so individual words of what you're saying what we want to do in that case is just print that out into our UI in these two methods we're defining what we want to do in the case of an error as well as when we close our transcriber now let's go back over to transcribe real time and complete this method so now we can do transcriber do connect and then we can also start a microphone stream so here we connecting the transcriber to our microphone and we're streaming it next in Step number five we're going to be analyzing our transcript every time we receive a realtime final transcript we send it over to this

method which will then send out transcript over to the large language model to get it analyzed for the step we'll be making use of assembly AI lemur framework and lur allows you to use a bunch of different large language models and in this case we'll be using 3. 5 on it in this case first off we're calling on Lima's task method and to do that we need to pass in three variables which is our prompt which is the prompt that we've written here at the very beginning the second parameter that we're passing in is the input text which in this case is our transcript for every single sentence that we're uttering we're going to be passing that in and lastly we're defining that we want to make use of claw 3. 5 Sonet in order to do this at the end of this once we get our result back from lemur we want to pass that to our front end so again we're going to be making use of socket IO to pass that result. response which is the text back into our front end now we're in the final step where we'll put all of these pieces together to define the logic of our overall app for for so first off we are rendering our HTML after that we're creating a method called handle toggle transcription and essentially what this does is every time we click on the button to start transcribing in real time it first checks if a transcriber is already in place if so it makes sure that it closes that and then restarts a new one and

also ensures that it is thread safe after saving everything that we've done I'm heading on back into terminal to run our code once I open it on the local address this

is what our application looks like so let's start recording today I met with a patient Jane Smith who is 32 years old she has been experiencing frequent headaches and some dizziness over the past few days I've prescribed some ibuprofen to deal with her pain and I've told her to keep a headache diar so she keeps track of whenever she gets headaches we'll review her symptoms in 2 weeks to determine the next steps so here's our application which does medical transcription analysis in real time and it's also able to identify key medical information in real time which is extremely helpful for healthcare Professionals in the next part of our application we'll take a look at how we can save this key information directly into Google Sheets and how we can deploy this application to the cloud if you're interested in watching the next part of the medical transcription destion analysis project check out the video above