Auto-generating meeting notes with Python

hey welcome in this video we will build a application that will automatically summarize your meetings for you so the

use case is basically let's say you missed a meeting and you don't want to listen to the whole thing all over again because it might be a long meeting right it could be like a 40 minute 50 minute one hour or maybe even longer than an hour meeting and if you just missed it you will probably just want the main concepts that was talked about in this meeting so that's why we made this app so to basically quickly go through it what's going to happen is that once you get the recording of this meeting you can upload the audio file here this audio file will also be displayed here so you can just go over it and listen to any parts that you're interested in

we will extract the main themes that was talked about in this meeting so in this specific one i am using the reunion of the lord of the rings cast where they have a zoom meeting together and in this specific one of course are talking a lot about movies technology food and drink television family and relationships etc so these were the main themes that was talked about in this meeting and we will also get some different chunks different chapters in this meeting so at some point apparently they were talking about how this video is sponsored by cheerios and um a part where the theme was a trip into the past a lord of the rings reunion etc and then we will also get for each of these chapters a summary of what was talked about in the specific part and we can also click to you know if you are specifically interested in this little chapter you can click and the player will jump to that place and you can start listening please welcome to reunited apart lord of the rings edition the fire so let's learn how to make this so let's first set up the streamlet

structure of this whole application and then we can add the machine learning capabilities on top of it the first thing that i want to do is to import streamlit and then i will ask the user to upload a file i will just call this uploaded file for simplicity normally if there is no file that was uploaded our streamlight application the streamlet function here will return a none but we can say if uploaded file is not none then we can do something here and the thing that i'm going to do is to display the audio file that was just uploaded for now i will set the start time to be zero but as i said after we have the chapters we can change this to dynamically start at when the chapter starts next i'm going to create two new

python files this one will be to communicate with assembly ai's speech to text api and the next one will be the configure file this is where i will keep my assembly ai api token basically so in here what i'm going to do is to import the requests library because that is a library that's going to help me communicate with assembly ai's api and also paste the function that will help me send the file that i get from the user to assembly ai and get a transcription plus some of the analysis that we saw so for example the categories and the auto chapters back from assembly ai so let's go through this line by line first of all what we're doing is we are

defining the transcription and the upload endpoints for assembly ai the upload endpoint is where we're going to upload our file the file that we get from the user the audio file and the transcript endpoint is where we're going to send the request for the transcription to happen but also the extra nlp capabilities to happen it's the same uh url that we're going to use after that what we're creating is a post request to assembly ai's upload endpoint with the audio file that we're going to get from the user and we will define it here we will basically use the uploaded file so i can already actually include it here so if i say from get results import all i can already call the or the name of the function upload to assembly ai function and to that i will send the uploaded file so basically this is how we're going to do this um and what we also need here are the

headers the header consists of two things one of them is the api authentication token that you have and the other one is the content type that you're going for i will define the header here so as you can see right now we're missing the authentication token so what i'm going to do is go here and configure that pi file and then i will define an authentication token and in here i will paste my authentication token that i get from assembly ai and that's quite simple uh all you have to do is after you make an account at assemblyai. com you can sign in and then you will have access to your token and just by clicking it you'll copy it and then you can paste it here after i save that i can again come here and say from configure import all or you can just say import authentication token so this one but yeah it's the same thing so it works so all right after we create the request the post request to the assembly ai's upload endpoint uh we will also get a response from it so after it is uploaded we will get a json response and this response will have the url of the file that we just uploaded so that we have access to it and i will keep it in this uh variable called audio url and when we want to start a transcription what we're going to do is first we're going to create this json file or json variable that includes the audio url so the link or the address of the audio that we just uploaded to assembly ai and some extra things here so basically these two things mean iab categories if you said it the true what you're saying is i also want on top of the transcription that you're sending me the categories that this piece of audio belongs to and also you are saying by saying auto chapters is true that you want this audio to order the strength transcription to be divided into chapters well you will get a heading for them you will also get a longer summary for them and the gist so basically a very a couple of word summarization or headline of this chapter this is what we're going to get from it same as last time like the upload we are again creating a post request we are this time sending it to transcript endpoint using the json variable that we just created and also again the headers for authentication purposes and from this again we get a response a json response and from using this response we can keep track of how the transcription going because it's not going to happen instantly we're going to have to wait a little bit but using this transcription endpoint that we already know and id that god we got from the response when we created the transcription request we can create a

polling endpoint and this polling endpoint will help us go back to assembly ai and ask hey is my are my results ready so that's why in the main streamled application file i will also say this is returning me the polling endpoint from this point on what we want to do is to check to see when the transcription is done and then visualize or display the results to the user so to understand what kind of thing that we need to do let's first go and take a look at the documentation from assembly ai and understand what kind of results we're going to get so here is the results for auto chapter summarization these are the descriptions for the auto chapter or summarization capabilities of assembly ai and if you look here this is the kind of response that we're going to get again this is the idea of this transcription and the status will be completed when it's done and here is how we're going to get the chapters so basically the chapters will be a list of dictionaries and each line or each dictionary inside this list will have a summary as we said it will be a longish description of what is happening in this chapter a headline that is again a bit of a longer description of or the biggest headline of what is happening what is being talked about in this chapter and the gist is just a couple of word long um basically the gist of this chapter what people are talking about we will also get the start and end of this chapter in milliseconds so then we can turn it into seconds and then display it to the user so let's build the logic in streamlit to display the results when we get it in a json format like this we might need to ask assembly ai if the transcription or the analysis is done or not a bunch of times so that's why what i'm going to say is while the status is not completed just keep going with this loop and before that i'm going to set the status to be submitted because that's will be the status at first and all right so how we're going to get the status is by using the polling endpoint so again we're going to have to send a request to assembly ai to tell us how the transcription is going this time this will be a get response because we're not posting anything we're going to send it to the polling endpoint and we are using the same headers as you might remember i defined the header in this other file but because i'm saying import everything from the get results python file they are able to have access to the headers and the response to this sorry this is not response this should be requests and i also need to import requests library here and the response to this will be called the polling response and from this polling response i can extract the status how i'm going to do this is i'm going to make it into a readable json format and in there i can find the status and then i will say well this is the status so i will update the status variable with this so if it's still not completed in the next round if it's it will not enter this loop again but for now what i want to know is if status is completed i want to enter into this condition and start displaying the results there are two things that i want to display the first one is the categories and the second one is the summaries we can also just create headers inside our app like we had before here to kind of you know make it easier for people to use it make it more user-friendly so for that i'm going to use subheader this will be the main themes of this call summary notes of this meeting so as i showed you before i basically want to put everything in an expender because if you start having a lot of text one after another it's going to start getting really messy so that's why i want to keep everything in what in stream what they call an expander especially the themes because you know these themes are just going to be like a good to know they're not that vital often information so that's why i will just list them one after another in an expander and how i'm going to do that is with this i'm going to use the width keyword here and i will create a streamlet expender and this expander will just say these are the themes and in there what i want to do is basically create a markdown for all of the themes that assemble the ai was able to capture from this call to create a list i can easily in markdown at least i can easily add a asterisk and then leave a blank space just one space and then it will automatically create the list view like this but of course to display them first i need to extract them from the response we get the transcription response so here we got the polling response right and if the polling response was has the status of completed that means that we can extract these categories from them so that's exactly what i'm going to do pollingresponse. json and again let's take a look at the assembly ai's documentation to see what it's going to look like so we need to go

to topic detection and here is what the results are going to look like so basically we need to go to iab categories results uh here is a more extensive list of all the uh categories that they were able to capture but i want more of like a summary all of them together so it's easier to read we don't have to deal with this whole dictionary sort of structure i can just read them here and that's what i'm going to do i can actually copy and paste it will probably be easier my av categories result because i am very prone to making uh typos and then summary and this is already a list here as you can see so it will be very easy to parse it also once i put this into a categories list i can just say in order to display them while i'm sorry for cat in categories create a markdown list and show me old categories for chapters i'm going to need to do a little bit more work it's not that much more work because here you can see for categories we're going to have more information and i want to actually use most of this information i don't want to just extract the summary and let the other ones go so that's why what i'm going to do is to put all of this the response for the chapter summaries into a data frame i like using pandas data frames because it makes everything a bit more manageable so before i do that let me import pandas as pd and put the results for the chapters from the polling response into a first just a variable and then to a data frame luckily it's very easy once you have a list of dictionaries to create a data frame from it so i will just call this a chapters data frame and then pandas data frame and i just need to chuck this in there and then it will create a data frame for me so maybe let's stop here and then see what we have so far and then we can go ahead and display all the chapters in a more structured way all right it looks good for now so let me just browse my files and upload the meeting audio all right let's see if this works i want to ask about some of the competitive relationships in this film all right it looks like it is actually working we are able to upload an audio file and also listen to the audio file so let's see so far what we built is um displaying the themes and also we wanted to see the data frame actually let me also just print um sd data frame just display this data frame to see what kind of data frame that we have and then we will be able to understand it better alright this all looks good we have the themes already here that was the easier one of the two and for the summary tab chapter summaries we have it all in a data frame as you can see the start and the end are noted in milliseconds so it's one of the first things that i want to do is to turn them into seconds and minutes i'm going to put this extra function here in the get results python file because i don't really want to have a lot of helper function etc in my main application file so this is a very simple function actually patrick the other youtube maker youtube video maker on our channel used this function before on a similar application uh tutorial so it's very simply just converts the milliseconds into seconds and minutes and returns the text version so what i'm going to do here is to just apply this to the start and the end columns i will create new columns basically because i don't want to mess with the milliseconds i'm going to need it in the future when i want to change the where the audio starts playing so that's why i will create new columns setting or keeping their names what their string name needs to be in terms of minutes and seconds and we can display it again and see how it looks all right we see that we still have the start and end in milliseconds now we also string formats with minutes and seconds so let's go ahead now and create the actual expanders that will show us the chapters just and also the full summary so i don't really want to show the data frame anymore i just wanted to see you know what it looked like so from this point on i would have to iterate through the data frame to show the summaries for each chapter and chapter is f this way i will get each row of this data frame and every iteration of this for loop and for i'm going to create an expander that will have the gist of this chapter and inside i want two things i want the summary so i will use that in just normal text format inside street streamlet and i also want a button that is going to be clickable to bring us to the beginning of this chapter that you're looking at so for the summary it's very simple again we already have the summary inside this data frame so just need to say row and for the button it's again very simple you need to say row start scr so it's not clickable just yet i just want to show you what it looks like and then we can do the dynamically changing where the audio starts so let's start around this again all right so we already have the chapters here as you can see maybe let's look into a bunch to see like if they make sense so for example a trip into the past the cast of the lord of the rings is on a reunion call for from the cast of the classic trilogy joan and josh are trying to reunite some of the cast behind the iconic trilogy based on the books orlando bloom is now on the call but he is not a fan of the series interesting uh another one could be lord of the rings the union okay this is a good summary of elijah wood sean ashton dominic moynihan blah blah blah blah they reunite and elijah wood uh says this iconic line and let's bring on the mastermind behind the magic peter jackson apparently he also joins uh what else theft from your set that's interesting if you have a prop from the lord of the rings pull it out now i'm very impressed by the fact that everybody got something i guess everyone pulled something out this is called theft that's really funny or another one for example coming out with the score fellowship theme and the shire theme are the other half of the film howard shore is a composer of the iconic ring theme peter and philip first recorded the blah blah so basically here they're talking about how they came up with this core of this movie so that's quite interesting we actually got some really nice chapters here so the next thing that i want to do is to make these clickable so when i click this audio file is going to click to the 0 6 11 point and then i can start playing it so for that actually i'm going to use one of my favorite things about streamlight which is the session

states and the callback functions they make a gigantic difference in your app to make them a little bit more dynamic uh sophisticated other than you know just an app that runs from beginning to end so the first thing that i need to do for that is to create a session state and then i will call that session state the start time or something like that let's call it i don't know start point and we can initialize it to be zero it doesn't really matter what it is in the beginning and i will also create a

callback function and this callback function will be the thing that will update the session state when i click the button you can call it update start we will pass the milliseconds to this function and then we will convert it into seconds jump using the session state of streamlit and let's do this first i will update the session state that i've just created so it's called start point and take the milliseconds divide down by a thousand then i have seconds but i also need to make this into an integer so now after this happens the session state of start point will be updated but of course this will yet not mean anything if you remember when i have the audio widget of streamlight here i set the start time to be zero instead i will the starting point but yet still nothing will happen because i am not calling this update start function anywhere yet so for that i'm going to go over here and in the button i'm going to create a on click behavior and that will be the update start function as you can see i am passing a argument to the update start function and to be able to do that i'm going to say args row start and that way i will be passing the start milliseconds to the update start function will update the session state called the parameter in start point and this will affect our audio widget and it will update when this audio file starts playing let's do it one more time so right now it's playing them or is it just this group but we obviously talked to howard early because we have the themes we have old chapters and the descriptions let's say we choose this one and if i click this yes it has jumped to 37 minutes 35 seconds and then let's say you're interested you can start listening to the chapter here please welcome mr howard shore and i'm guessing from that point on howard shore joins the call and they start talking about the uh score or the music behind lord of the rings which is an amazing movie if you haven't watched it i'll definitely recommend that you do but that is not the point in this video of this video i hope you enjoyed this tutorial i hope everything was clear before you leave don't forget to give us a like and maybe even subscribe to our channel to show us your support we were also love to hear any of your questions or comments about this tutorial in the comment section below and don't forget that you can get a free api token for assembly ai using the link in the description but for now thanks for watching again i hope you enjoyed it and i will see you in the next video