I created a Python App to study FASTER

hi everyone in this tutorial we build an app together that allows us to go faster through video lectures or zoom calls basically any kind of video material we have the app is built with python and streamlit and it looks like this so first of all we can watch our video here

and then it uses machine learning to do two things the first one is it extracts highlights automatically so for example here i'm using the nlp with deep learning lecture from stanford that you can find for free on youtube and here it extracted these highlights for example we can see word vectors word vector learning context words and a lot of other highlights together with these timestamps so we can click on any of these timestamps and then the video jumps to this position and starts from here so we can basically skip the boring parts and just jump to the highlights so this is the first thing it's doing and the second one is it automatically generates chapters and chapter summaries also with timestamps so we can see here all the timestamps for each chapter and then for each chapter we get a little summary for example here this time stamp and then we get gpt3 is a huge new model that was released by openai so let's click on this and watch the video the single biggest development in nlp for the last year certainly in the popular media was gpt 3 which was a huge new model that was released by openai so yeah this works pretty well and this is done with a deep learning model in the background for us which just blows my mind so let's implement this so in order to analyze our video we

use the assembly ai speech to text api and don't worry you can get started for free so we just need to create an account and then we grab our api key and that's it and then we use the requests module in python to send the api requests and we need two api endpoints the first one is the upload endpoint and then we need the transcript to send three api requests to do all this so i will go over them here very briefly because i already explained them step by step in another video that you can find here and i will also put the link on the code on github so you can go through it again by yourself so the first thing to do is grab the file so you can download either the video or the audio file in this case i downloaded the mp3 file with a youtube downloader and then we need to upload the file so here we send a post request to the upload endpoint and this will give us a upload url back so if the file is already hosted somewhere in the web we can skip this part but in this case we upload it ourselves so this is step number one then step number two is to start the transcription so we send a post request to the transcript endpoint now and here we need to give it the audio url and then in order to use the chapters and the highlights feature we just need to turn it on here in our request so we say auto chapters is true and auto highlights is true and then we send this together in our request and then we get a response back and then the last thing to do is just to grab their results so now we again send a request to the transcript endpoint now together with this transcript id that we got from the step number two and in this case we send a get request to this endpoint and then we can check if the status is completed and then we can save all the results so i saved the transcript in a text file i also saved the chapters in a json file and i saved the highlights again in a separate json file and that's all so then we put all this together in a pipeline function so we upload it we transcribe it and then we get the results and save it and that's all we need to do so now let's run the file and wait for the result all right so it's done and it took a few minutes because this was a 90 minute lecture but now you can see we have three new files in our folder so we have the transcript then we have the chapters and the highlights file and now we can start implementing

our app so let's create a new file app. pi and as i said we use streamlit for this so we say import streamlet sst and of course we need to install this so in order to do this we just say pip install and then streamlit and hit enter and this is already installed here and then we also use a third-party widget in this case with which is called streamlet player so we say um from streamlet player we want to import the st player so this just gives us a couple of more features for our video widget for example this automatic starting of the player so yeah we also can install this by saying pip install streamlet dash player and hit enter and that's all the libraries we need then in this case we also use json but this is a built-in python module and now we can start coding our app so the first thing to do is give our app a title so we say let's call this simply learn and then let's um copy the url and the file names so here we have the youtube url then highlights json file and the chapters json file so this is the file name of this here and now let's start implementing our app so the first thing i want to do is add the player and for this i use a placeholder and i explain why in a second and for this i say placeholder equals streamlets dot mt and then i say with placeholder. container and now i set the actual widget by saying st underscore player and this gets the url it also gets the arguments playing equals false in the beginning and also units equals false and i use this placeholder because later when i click on a button i want to again set this player but now then with playing equals true and also muted equals true but i don't want to set another widget i want to use the same widget at the same position so that's why i need to use this placeholder so now we have this and then let's also create our sidebar so let's call this mode and then we say streamlets dot site bar and then we use a select box and it gives it gets a title so let's say this is the summary mode and then it needs a tuple with all the options so the first thing we call this is the highlights mode and the second one this is the chapters mode so now we have this and

then here we can use a if else statement so we say if mode equals and then we need the same string as here so if this is the highlights mode then we do the highlights showing and otherwise else we do the chapters so for now let's run this and um save this so here we say in order to run our streamled application we say streamlet run and then the file names in this case app. pi and yeah here it's starting our app so here we can see we already have the video player and here we have the two modes we can choose highlights and chapters so yeah this is working pretty cool and now let's continue with this implementation so for the highlights mode i want to read the highlights files so i say with open and then i get the file highlights and i want to open it in read mode as file and then i want to load this so i say data equals json dot load i can simply load the file and then i want to extract the results so i can say results equals data and then i use the results key and the results key let's quickly have a look at the highlights file here so here you can see this is a dictionary where you get the status and the results and then here this is a list or an array and then for each highlight you get another dictionary so here you can see we get the text and the timestamp so we can have more timestamps for each highlight so yeah this is how the highlight file looks so let's continue so now we have the results and now we want to go over the results and here i want to store them in three different columns so i say calls equals streamlets dot columns and i want to have three columns then i also want to count the number of buttons i create so this is zero in the beginning and then let's go over the results and so i want to keep track of the results index and of the result itself so i say for result index and results in and now i use the enumerate function and then here the results and now i want to display the text and i also timestamps so i say text equals results and then again the key text so remember here this is another dictionary with the text key and the timestamp key so we get the text and we want to get the time stamps equals results with the key time stamps then i want to choose which column index i'm currently using and for this i simply say the current results index modulo 3. so this will be one two three uh sorry zero one two and so on so then i assign this so i say with calls and then this index so i write it in this column and now i want to write the text so we say streamlets write text and then i also want to write the timestamps so i will go over them for t in time stamps and then i want to write them in a nice format so first of all we need to extract your only the start time so again this is a dictionary so we say the start in milliseconds equals t with the key start and then i want to add a button so i say add button and this gets the start milliseconds and the current number of buttons so let's say n button with an s and then we increase the number of buttons so we say this plus equals one so now the add button will be a helper function because we use the same later again in the chapters else pass so now let's create this helper function up here so we say define at button and this gets the start in milliseconds and it gets a key for the widget so here first of all we want to get these start in seconds also so we say this is the start in milliseconds divided by 1000 and then we say if streamlet. button and here is a label we want to use another helper function and here we want to display this in a nice looking format so let me simply copy and paste this for you so here i basically convert the milliseconds in a string that shows seconds minutes and hours and if there are no hours then it only shows minutes and seconds so yeah i will leave this here for you can go over this again by yourself and so here as a text for the button we now call this get button text and this gets the start in milliseconds start milliseconds and yeah so this if statement it creates the button and also when this button is clicked then it will execute this code here so now what we want to do is we want to start the player at this time so we can simply set the youtube url with this time as an option so we say url time equals and then we use the base url plus and then as string and ampersand and then t for time and then um i think it's equals and then it needs the string the time as a string in seconds that way that's why we used this here so now we say string and then start on in seconds and then it also needs the s in the end so the time equals this in seconds and then we again set the player widget so we do the same that we are doing here and that that's again why we use this container so now here we say with placeholder container and then again we set the player now with the url time and this time we say playing equals true and muted equals true so i think now we should have everything for the highlights mode so we can again go to our app and check this out so yeah and the this plane works so let's click on one and we see it jump to 8 28 minutes and 34 seconds so this is working um it's still muted or did i do this myself uh let's quickly check ah and of course now i need to say muted equals false and then i also spot a warning from streamlet so i need to have a unique key for each button so that's why i also use this argument here so for this widget i want to give it the key because by default it uses the label as key but here we can have multiple buttons with the same timestamp so that's why we can't need the text as key but we need to use this button number as key so yeah now it should be fixed so let's restart our app and check if it works now so let's go again to our page and click on this and now it's no longer muted so yeah this is working

so now the only thing left to do is implement the um chapters um displaying so in this case we say with open and here we want to use the chapters file again in read mode as s f and then again we use json so we say chapters equals json dot load the file and let's quickly have a look at the summary of this summary an even shorter version and here we have start and end time so we only want the start time in this example so let's extract these information so we have the chapters right away this is our list and then we can say for chapter in chapters and then again we want to get the start in milliseconds by saying chapter with the key start and then we get the button with our helper function and this gets the start in milliseconds and as key in this case we can use none because um this is now unique so each chapter has a unique timestamp and then we also want to get the text by saying chapter and then it has the key summary let's have a quick look um yeah we use the summary here and then we simply say streamlets dots write the text and that's it so let's save it and go to our app again and reload this and now let's select the chapters and yeah so this works here we have the auto chapters here again is our summary cheap gpt3 is a huge model that was released by openai and if we click on it jumps to this time step and that's it yeah this is really cool and i hope you enjoyed this tutorial and if so then please leave us a like and consider subscribing to our channel and then i hope to see you next time bye