ChatGPT + Code Just Changed Everything! (Again)

chat GPT code interpreter starts a whole new chapter in the development of this little mirror club and that is because this feature started a journey that will continue throughout the next months namely we're moving from just receiving information to actually taking action that's right by openai putting a little mini computer into gpt4 with the code interpreter this starts to open up a whole new world of possibilities because now this interface can perform tasks for you and while it can do a lot already most of the possibilities are still walled off this release starts to move it away on the scale from a Google search competitor more towards Auto GPT competitor so let's dive in and let me show you what is possible and what's not

well first things first as per usual just go to settings beta features and enable the code interpreter and this will allow you to select it here which adds a brand new button and this is a big deal you're now able to upload files so as mentioned we're moving more towards the action bot and these actions can be performed on multimedia right so you can upload PDFs you can upload jpegs you can upload the gifs even basic video files and perform all kinds of operations on them alright so we'll just

pick this majority image off a Milky Way as our starting point and as mentioned we'll actually make it do something not just rewrite or research we're going to perform action and the one in this case is going to be turn this image into a video by slowly zooming into it that's right gpt4 can now process images and videos but first we gotta answer the request here what duration do you want five seconds focus on the top left alright and the way this works is that they put a code interpreter and a small runtime environment into gpt4 what that means in practice is that you have a mini computer with very limited specs but still a mini computer inside of this web interface because up until now 10gbt and gpd4 was really good at writing code right debugging too but it could not run the code it just gave it to you and said hey go ahead and write it on your own computer like yes my turn open up your own vs code and use the code in there no need for that anymore look at that if I open up this drop down and don't worry I'll make this non-coder friendly it brings in these packages where pill is one that allows you to manipulate images and the cool thing about python which is the language that this is capable of is that it has a plethora of packages so for example pill allows for all kinds of image editing and as I told you it's a mini computer in here and because we uploaded a file to the mini computer it can now access the image path to the file no problem right it opens the image it defines the video properties right I got a 30 FPS video that is 5 Seconds long and then with the CV2 Library up here it can actually create a video then it just goes frame by frame and finally it writes the video you don't need to understand any of this it's happening in here for you all you get to do is click this button that says download the zoom video Let's just save it to the desktop and have a look and that is not correct but it's a good start so here it's obviously zooming out but I'll use this opportunity to teach you how to work with these things that don't work right away because that's going to be the case more often than not I'm gonna take the learnings from the conversation and I'm gonna update my initial prompt turn this image into a video by slowly zooming in on the top left and remember it also asks for a duration so I'll just include that too okay so now we have everything in one and we'll resubmit this and by using this edit feature and a workflow like this you will get the higher quality prompts every time you do this which will allow you to save some of these to reuse in the future so again the workflow is try stuff answer its requests and once you arrive at a result take all of those answers and incorporate it back into your original prompt which you can install for future use and as you can see now in one prompt we got the whole thing download the video and let's just have a look if this worked there you go now it worked and I'm just gonna try something fancy as I do have a video background and I'm gonna say use a bezier curve for the zoom to ease in let's see if I can do this what that means is that the zoom will not be regular but it will start fast and then slowly ease into stopping oh that's looking good it's creating custom control points AK keyframes where it depicts the zoom in position at various points in the video okay but as you can see now the previous approach doesn't work anymore it needs to use the castle jows algorithm I don't even know what that means but that's the beauty of this it's self-correct while it takes the action so as I said less of a Google more of a auto GPT where you give it a task and then it doesn't just give you the code and run it also checks if the result is desirable and runs it over and over again until it works guys I hope you realize what a big leap we just made here because this is a big deal and as you can see with these more challenging tasks it really needs to think step by step ah there it is download the video I'm excited for this one let's have a look not bad that's so much better do you see the difference if I were to edit this into one of my videos it's not even close I always want the version that eases in okay Twitter is filled with examples of other great use cases for this for example you could ask it to create a gif that looks like the Matrix animation and you could just use this on your website or any other place that takes a gif let's try and push it and say add following Matrix letters on top of the video you just created alright and it finished generating that's the first frame wow not bad this is actually impressive but it did make a mistake it didn't provide move the download link here so I'll just ask where's the video link all right there we go download the video there we go it's something I mean we didn't specify how fast the letters are supposed to go by but this actually worked okay but there's one more massive

point that we need to touch on and that is its ability to perform data analysis and before you phase out because you might say yeah but that's not something I need regularly well I would argue that most people can find a use for this if they know what to look for because no matter your career or occupation producing graphs and learning from your past Behavior can be useful for everyone if you just know what to look for so what do we look for what are we going to generate graphs from where get these insights from well you gotta look into all the internet based applications that you already have and most of them will allow you to download the data set for example if you're any sort of entrepreneur you could take any Excel sheet from your business feed it into here and then start interacting with it and asking hard questions more of a personal use case would be taking the data from your bank account and then performing tasks on top of that you could ask questions like where the most of my money go or categorize all my expenses but I feel like showing that step by step would be a separate video what you would need to do there is export a CS file of your bank statements and every Bank offers this by the way CSV is just the most Bare Bones data format it's literally all the numbers just separated by commas and then you would just upload it to the code interpreter and you might start by asking something like what sort of data analysis can you perform on my personal bank statements and it will tell you and trust me if I tell you that's just where it starts where you have some more Advanced Data like sales or maybe Market data from your industry you can do something like Leo Spartan Halls here on Twitter and he just fed it to the code interpreter and it created this very informative box plot of 12 months moved growth rates for single family homes with a detailed explanation below for what this graph exactly means but look this is really a big deal because we never had a more user-friendly way to interact with large sets of data every time a normal person heard data they kind of phased out because it's something that the big tech companies deal with right are you running no now you literally switch tabs and you can ask hard questions on top of all of your personal data and be careful with this I always equate sharing stuff with openai to sharing stuff on social media maybe it's not quite there but it's a good mental framework at the end of the day it's totally up to you to decide if you're comfortable with sharing your personal transactional information with chat GPT I'm just here to tell you it can be really useful but I enough talking for

the last point today I want to show you something a little more exciting and that is combining the two things we talked about image analysis and its ability to interact with multimedia files and I actually played with this all day yesterday and I pushed it to its limits in multiple ways it can do a lot but not everything let me explain here's what I found so first up I need to make you aware of this technology called OCR which stands for optical character recognition what that means is that you can give it a picture and it can analyze and read what's on top of the picture and then you can use that information to move forward so look let's start with something challenging move to code interpreter so I'm going to upload this Slovak receipt right here where the total is 5 Euros and 20 eight cents okay and then I'm gonna try to use gpt4 to retrieve that number from the image so it says it doesn't have the specific language for the data but the powerful thing here is that gpt4 does so it can read it in English and then it can use the gbt4 capability of translating very well and look I'll just be frank with you when I tested this worked perfectly right now it shows me 228 euros while it's 520 so it reads the 5 like a 2. I'll try to rerun this once yep second try didn't get it either badly scanned foreign language receipts might be pushing this too far so as you can see this opens up a world of opportunities but a lot of it goes beyond its compute abilities and another part of it works but the results are not satisfactory that's why I'm saying this is a real glimpse of the future as we're starting to move towards this action-oriented programs but that's because they're putting so many limitations on top of us right limited memory limited context window only 25 messages when we got Bing it kind of worked but then a lot of the times it didn't all of these show up vision and a path to something extremely powerful so as I showed you this point you need to know what you want to get and you need to know how to ask to get that result but in my estimation that it's going to take months not years for it to advance and for all of these capabilities to merge into one super powerful application alright I hope you learned something and if you want to learn more about the technical capabilities of chat GPT check out this video I'll see you there