Named entity recognition on audio files with Python

hey and welcome today let's learn how to do entity detection on our audio files using python in a very simple way so let's see what the result is going to look like once we're done with this project we are going to get a list of entities in a dictionary format for each entity we are going to have a start and end time stamp in the audio file of when this word was uttered the word itself and the type of entity that it is so for example we have the entity type event which is ted so the ted talks for example we have the entity type occupation and the text was psychologist we have the entity type person name and it was dan gilbert was mentioned for example so let's build this we are going to use assembly ai's api to achieve this because assembly ai on top of the transcription that it does has some audio intelligence features and entity detection is one of them so for this project we are going to have two python files one of them is called main. pi and the other one is called configure. pi in main. pi we are going to have our actual code for this program and on compu pi we are going to have the assembly ai api token that we got from assembly ai so it's very simple to get a api token

from assembly ai all you have to do is either go click the link in the description that we have or go to assemblyai. com and once you click get started you will be taken to a page where you can create a free account and once you do that you will already be assigned a free api token for using assembly ai's api so here i just need to click this and i will save this authentication key in my configure. pi file and what i need to do is after that is to import it here to talk to assembly ai we are going to use the requests library of python so i will just import the request library now and what we want to do is to send an audio file to assembly ai right and we're going to do that with the request library with a post request in this post request we are going to need three things the first one is the transcription endpoint for from assembly ai and let's write it down here this is the transcription endpoint of assembly ai it is basically api. assemblyai. com version2 transcript the next thing that we need is the data that we want to send to assembly ai it is going to be in json format so i'll just call it json in this data we are going to have two main things one of them is the url linked to where the audio file lives i already uploaded it to a location now i'm going to pass the url of this audio file to assembly ai and the second thing is what we want assembly ai to do on top of just normally transcribing the audio file as i mentioned assembly ai has audio intelligence features on top of just transcription and here we want to use the entity detection feature of assembly ai so at first i will call the audio url here i will pass the audio url and then i will mention that i would like entity detection to be true because audio intelligence features are by default normally are set to false next thing we want to pass to assembly ai is headers so again i'm going to simply call this headers are going to help us authenticate ourselves and tell assembly ai who we are and who's trying to start this transcription so let's call it headers the first thing is authorization and this authorization is basically the auth key that we saved in our configure. pi file and the second thing that we want to pass is the information of what kind of data that we're passing to assemble the ai and that is mentioned by content type and we have a content type of application dash slash json so this will be already enough to start a transcription at assembly ai once we submit a transcription job we will get a response from assembly ai so let's print this and see what kind of a response we get all right so we get a response 200 that means that things are going well but if you want to understand more of what's happening which we do we are need we are going to need to look into more inside this response so let me run this again all right that's more like it so the response we get gives us some idea about the transcription job that we just submitted it tells us the id that was assigned to this job which language model is being used and some other information but it also mentions to us which features of assembly ai are being turned on as you can see we are not using auto chapters for example that is set to false or sentiment analysis whereas we want entity detection that's why we have it as true here from this point on what we want to do is to continuously ask assembly ai whether the transcription job is complete or not so once it's complete we can get the information for that i'm going to create a while true loop and in here what i want to do is ask assembly ai whether transcription is complete um if yes we're going to print the information about the entity section and if not then we are going to run the loop again so to ask assembly ai means basically polling assembly ai and to poll assembly ai we are going to need to create a

polling endpoint so here we have the generic transcription endpoint but we have to create an endpoint that is specific to our transcription job so to do that we need the transcription id that we got from assembly ai the one you might remember from here so i will extract the transcript id from the response we got from assembly ai and it will be in response. json id and to create the polling endpoint we are going to need to combine the transcription endpoint a slash and transcript id and once we have this we can send a get request to assembly ai we are using get this time because we're not sending any data we are only asking for information that's why we send a get request to the polling endpoint and we also need to pass the headers to assembly ai of course and we are going to need to do this a bunch of times so i'm just going to wrap this into a function poll let's say and it's going to need the transcript id and as a result it will return the polling response all right now that it's done i can actually just call it here every time i need it so basically every time we run this um loop and let's run this already and see what kind of result that we get i'm just going to call this data what the response that we get from the polling and we can print it to see what kind of response we are dealing with clean this up quickly and just to see more clearly maybe i will stop printing the transcription submission response all right we keep printing it so i forgot to add a little break here so we only want it to run once let me just stop this clear again and then one again all right so this is a response that we get uh when we pull assembly ai again it's basically the same thing actually we get the id and some other information about this transcription job but one thing that we have to pay attention here is the status at first when you submit a job you get the status as queued and immediately after that it turns into processing and what we want to see in the status is completed basically so how we can check that is i'll just remove these ones because we don't actually need them for now uh if data status

is completed then we can basically print the data which includes the entity detection uh results so we did this already we also did this uh we do not have to run write the bits specifically where like if it doesn't happen run the loop again because obviously it's going to happen automatically but one thing we can add is if the transcription job errors out we might want to handle that too so i can say if data status is error then we can print there was an error and the response will have more information about what kind of error happened so we can print that too but once that happens i want to break this loop or this one happens all right so basically this is it this is all we have to do to get the entity detection results from assembly ai one thing i want to do here is to instead of just print the results i want to pretty print them so it's easier to see them and i also want to add a way to wait for 30 seconds every time before we ask assembly ai if the transcription is done or not so we ask if it's not completed or if it has not errored out we can say print waiting for 30 seconds before asking again because very likely that asking every second is not going to help us so then i can say time that sleep 30 seconds here now import time and that's basically all that we need to do so let's run this and see what our results look like all right our program is done running as you can see we waited for 30 seconds a couple of times and then we got the full completed response from assembly ai here we have a lot of information not only entities we can scroll down and see we also get the actual transcript and some other information with this transcription as you can see here but what we are interested in is the entities and here is what the entities look like as we showed as we saw also in the beginning we get a start and end timestamp for this entity and also the type of entity and the text of entity what was mentioned in this audio file so let's see some examples maybe it says the entity type is person name and the text is thomas uh another one could be let's see um money amount 3 000 here an occupation for example is mentioned as studio musician so there are many different types of entities that assembly ai can detect if you want a full list of what is possible you can go to assemblyai. com and in there we have a documentation and through the documentation you can see all possible entity types that can be detected using assembly ai's api i hope everything was clear don't forget to let us know if you have any questions about anything that we did in this project and don't forget to go get your free assembly ai api token using the link in the description have a great rest of your day and i will see you in the next video