How Microsoft ML Researcher Actually Uses ChatGPT

all right so in today's video I'll be talking to a principal researcher at Microsoft and I thought this conversation was quite interesting because we bring such different viewpoints to it as you know I always come from the perspective of power user a tech enthusiastic consumer that obsesses over finding what's possible with these tools and she comes from the perspective of a machine learning researcher that works for Microsoft and actually publishes research regularly and as you know I usually talk about using chat GPT in the web interface and how to find use cases for your everyday life and she's particularly interested and how to use it for code generation and debugging so this is a very interesting conversation in the beginning we go into the basics of prompt engineering and how to enhance your prompts with techniques like following up or multi-shot prompting and she even talks about Shain of thought reasoning then I asked her a question that I was super curious to find out the answer to and that is do you use custom instructions and if you do what are they then after she reveals that and I give my take on what I personally use we move into the more technical questions of how do you use large language models for code generation and debug now before we hop right into the conversation I want to point out two more things and that is that obviously my specialty is not coding in development as you know I know my way around python but I've never built production ready applications whereas that is the use case that bmir actually deals with regularly and the last point is that I actually came into this conversation with the expectation of being a guest and being interviewed and then it turned out that I had to take on the role of actually directing the conversation too so sometimes I found myself in a spot where I was trying to move the conversation forward while providing value which was a little tricky but nevertheless I thought this was a very valuable conversation with a lot of little tips and tricks and a completely different perspective than you usually get from this Channel and I hope you enjoy this conversation between me and Bez me facilitated by Microsoft let's get into it so prompt engineering

Basics I think I'm big on Frameworks because it's a little abstract to teach people you know what to type into that magical little um text box essentially because the possibilities are endless right like the question of what can I ask is an impossible question because the amount of answers is infinite so I love to break prompts down into two parts right no matter how complex The Prompt problem you're tackling or trying to solve it always comes down to two things the core concept that I always teach at the very beginning is think about prompting in terms of instructions plus context instructions are super easy to grasp right it's kind of the task that you're tackling it's the thing that you're trying to solve you know write me an email write me this piece of code and then the context is the second part to the magic to the formula and that's where it gets it can get complicated right it can be as simple as write me python codes for game XYZ right that's kind of the context you're narrowing down the possibilities by telling it hey I'm not looking for C++ I'm looking for python here and then you can go super deep with that right and that it can also come in like a thousand Shades because if you provided a database code base if you provided access to a GitHub repo all of that is going to be taken in as the context and you're all of a sudden going to be talking to this personal assistant that has read the context that you provided to it and this is like essentially to me all I like when we talk teaching prompt engineering that's what it's all about it's about defining the context because getting the instructions right is super simple you can get a bunch of templates ebooks you can go on Twitter you'll find a bunch of instructions that are super useful you know do this do that but the context is where it gets well it can get tricky but that's also where you unlock all the abilities so yeah what do you make of that base framework vamir instructions plus context would you like to add anything to that I would just like you know go one step further and maybe like think about what one would want this model to sort of help with and what is the situation um there are cases where You' want the model to do something that you already know how to do very well but you're kind of just like you know lazy to write it down all that bulky code and you want to kind of enhance your productivity and then there is like situations where maybe you're not proficient with that new problem it may be a language it may be like a new library and then you're just like looking for ideas so that you grasp like what the new functionalities are or a new way of solving a problem and then the third use case can be just like browsing documentation which we know can be very painful um often times so I guess what I'm trying to say is that often depending on like the use case and scenario things may have to tweak a little bit and um having a more conversational Style on things that you don't know is perhaps um nicer and then you get to learn more and then when it comes to documentation um really you know uh giving a prompt that can explore different types of libraries this is something that I'm using day-to-day actually uh for many things that maybe I'm not familiar with just to have a nice comparison between the different libraries uh over the there rather than reading the documentation of all of them and then you can dig deeper into things that maybe you're more interested in I really like that I think you touched on one point um you mentioned following up while learning about a new topic I think that is a core concept that a lot of beginners kind of miss and it's that you don't have to get it right in your first shot right you can it's a conversational interface so you can really start out with you know what you think you're looking for and then as the conversation progresses as long as you respect the uh the token window the context window you can add layers on top of that and what I always Advocate is you know if you start with hey I'm looking for this and that and then maybe you add in the examples like wait a second I was actually looking for this type of output or like here's a similar app like can you make it more like that later on you can make it more specific but I think it's just important to kind of like accept the fact that you don't have to get it right with your first attempt matter of fact you can have hundreds of attempts there's like the cost to it is virtually zero so I would encourage people to have an actual conversation and when you want to H when you know that you're going to be having that conversation multiple times maybe even daily as you mentioned in your case you can start saving some of those um conversation Snippets and you can start combining like multiple prompts or inputs that you had into like one right and the next time you have you face the same problem you can just like copy paste that one larger prompt that original resulted from you maybe going back and forth 10 15 times the way you arrive at that is just by following your intuition using your social skills because we all know how to talk and communicate right the fact that can understand each other right now is proof of that and then just use the social skills and talk to it and then you can piece the different inputs together and arrive at something that you will be reusing regularly and that makes it super easy to begin let's get into some Advanced Techniques here so if you go beyond the basics of asking it what package you could use or feeding it some code of yours and asking what you could add let's talk about I think zero shot multi-shot prompting is a great place to start at the core of all these models um it's just a Transformer that like repeats the patterns it has been trained on so if a pattern is not present in the model itself you could tell it this is the pattern that I'm looking for here and extending your prompts like that is simpler than most people think so how could we integrate multi-shot prompting into advanced use cases when it comes to code generation yeah it's an um in fact you know F short prompting for code generation is still an open research problem in that I um I don't think we really you know have gotten it right about like how to optimize the examples um and the reason for that is because um f short prompting for um previous problems like for example for prediction problems has been used as a way to give different instances of exactly the same problem and that for example if you are doing simply just like multiplication then You' give some examples of multiplication to really condition the conversation but in this case if you say do not know you are in the second scenario when you don't know the solution to the problem if you knew an example about like how that would work then you wouldn't need to generate new code right so this is where this um you know chicken and egg problem starts with but there are other ways you know how you could um kind of blend the techniques between uh Chain of Thought prompting and fot prompting to sort of give an idea to the model of like what they could do so you're kind of like uh facing a dynamic programming um question and you don't know exactly the solution right uh and there are many Dynamic uh programming approaches out there but you also don't know which one of them is going to be like most um efficient but you want to tell the model that there are dynamic programming approaches that it could use one idea to give a fut short prompting there would be to find a similar problem but give a maybe even a pseudo code solution that is not in the right um language but just like pseudo code on how to form dynamic programming would make that conditioning uh work well or giving it if you have implemented it in another language or you found a solution in another language and want to give it to the model so that it adapts it in another language that could be another technique and on the all this like Chain of Thought reasoning is something that could really help the model to pin down on the parts of the solution that are more important but it's very important that um that Chain of Thought comes before the solution itself so this is something that is very inherent in um these Technologies in that they always feed upon what has come previously in the conversation and that is super important and the closer it is to the conversation the more important it gets so um if we want the model to do Chain of Thought reasoning that it has to be prompted so that reason comes first and then the solution comes next so that the reasoning impacts the um solution yeah Ju Just if I may briefly expand on that I think it beautifully links into what we talked about before because the conversation that you're having it's very important to be cognizant of the fact that all that is being now stored and respected it's like bringing a new assistant into your office and then work starting to work with her so all the work that you've done throughout the day will be stored and will be remembered and if you open up a new chat you basically wipe the memory right it's like you give the AI Amnesia and you go from the beginning and you might find that often you get very different results when you send message number 15 as opposed to sending it as message number one it's just a whole different conversation right so yeah just realizing how the context works there is important follow-up question there vamir do you use the custom instructions feature when you're using the web interface or how you know how much do you use let's say the system prompt if you're going through the API how much information do you provide to it in advance before you even start the conversation because for anybody who doesn't know you can do that you can pretty much set it up with um a base with a base knowledge right without even accessing exal knowledge basis or anything like that you can pretty much teach it about yourself your approach your preferences do you do that and your day-to-day approach cuz I certainly do yeah honestly I only have two of them like one is for just like general information I've kind of like customize something that is more basically temperature zero and Beyond so that it only gives facts uh and doesn't get to creative with um you know with a generation and then just like one for code that just uh focuses on the main um coding problems that I work on um but when it comes to you know experimentation for research that is a whole you know completely different game in the sense that um one needs to be uh very specific in those instructions and even come up with these ideas that ENC code diversity depending on you know the problem that we're working on yeah absolutely so the temperature obviously is a super useful one um if you know I often teach people through the web interface just because it's so user friendly and there you don't really get temperature controls but that's huge because you can completely eliminate um you know the creativity there and just make it except uh just make it a little more Down to Earth but for myself I include some like interesting Parts like for example I like to specify um that parts to it like priming it uh to only give me you know verified data and it obviously can't do that perfectly but it attempts to do it and by you telling it something like that in Advan in advance that hey every time you reference something give me the link it will attempt to do it so you know it's kind of like a coin flip if you do that but when doing research or when learning something new I found it super helpful to prime it with some of my own preferences and when I'm doing research concretely for new content I love for it to be a little more forough give me a bullet point summarization at the end of every message I love when it tries to give me the references when it at least attempts to do it and sometimes I even like a little summary uh in the voice of a 5-year-old you know at the end just so just like a super concise summary so you can set it up to really fit your very own needs um providing examples let's just try to dissect that a little more because what I found in my experience is really that is one of the most powerful Concepts you have as a user as a end user that is trying to get something done if you provided multiple examples specific examples of outcome that you're looking for it will follow that closely like for the learning use case right when you're engaging in a new language or library or just trying like to explore like unexplored territory um there this is not so relevant right but when you're extending your app which is the other use case that you outlined in that case uh where would you go what are some places to look for examples right what would be some places some maybe some places across the internet where you would be looking for such examples if you're developing app yeah I also wanted to add a little bit on uh what you're saying is that um you know you're doing two things at the same time which is great like you're giving examples but you're also providing structure which is so important in that like you know you're saying like this is the structure that I'm looking for in the output and this is like how the form should look like which is like super helpful for automating certain things that are maybe like repetitive like you wouldn't want to copy paste all those things and modify them slightly like you you'd want the tool to do it for you so structure is the other thing that goes like side by side with the examples

right um it is very impressive how these tools can now add new lines of code without editing the previous lines like you know it will not break the structure or anything but it will just add that piece that maybe you were missing or that you wanted to get another perspective on how much do you go back and forth with chat GPT like just to give our viewers a bit of an a bit of a feeling for what that might look like is that like a is that a conversation that spans let's say like two to four messages or you know are you like really stretching it and going deep into it I know it depends on the use case but generally speaking when you're like diving into a new topic new language new package whatever it might be um how long are your conversations right off the bed yeah often you know it can be a very long conversation and again it's impressive I that we're able to have these long conversations right now um often it starts with just picking a library so basically there is a lot of choice out there um choosing it's a commitment every time you pick a library and then you include it in your base code it's a commitment because you continue to use it at least for the next few months and you need to maintain that code and read it over and over again um so maybe some often I even um ask the these tools to basic basically give me like similar Snippets but using different libraries and then I kind of compare them side by side and to see like which one has the right style for whatever I it may mean uh you know based on the context and see whether that is something convenient to use um it's important though for the audience to kind of like every time this there is like a library that is being mentioned in code to check that Library really exists um CU you know sometimes it can happen that there is like this model blend many different worlds in a way like text worlds and um some of them may be pretty close to what you want but some may not exist so it's good to do like that first check that the library is there and it's maintained and once you know that then you can do a little bit of this experimentation giving like easy tasks to each of the libraries to see which is the right one to pick absolutely I think that is huge you know trust but verify like it's a it's an excellent and very well educated and highly intelligent assistant but it's still just an assistant so you know respect that kind of boundary and I would also say um I remember like recently I had to generate a code snippet for me where it I used the pill library inside of python to interact with images and uh it used an outdated version um and as soon as I FedEd like the documentation to the updated version it could it worked with that and it just updated it but if I just took it at face value and included in into my app then you know I would run into I ran into problems pretty quick quickly so you know trust but verify yeah and you can also ask follow-up questions again like you know follow-up questions are so important with these things if there is any you know aspect of the code that you really care about then asking again on like well you know can you think about any uh sort of gaps in this code or uh if we're talking about things like um authentication libraries or uh memory management in maybe like CN C++ programs asking for up questions of like how do you think this would work in a case where this and this happens uh just to sort of push the boundaries of what the model is supposed to do in this case is always good and um you know it's a very natural question to ask like why wouldn't the tool do this in the first place like why does it wait for me to ask that follow-up question um one needs to understand that you know a lot of the knowledge that comes out of uh these models is always condition based on the prompt based on the input that we are specifying and um if that is not part of the consideration on the first place that knowledge will not be surfaced immediately so you kind of have to have that condition there for that knowledge to appear and um that's what the follow-up question is doing

basically this is one of the best parts of the entire model this is what got me uh super excited when I started like experimenting with um with especially like gp4 it's just so good right and it's something that we just didn't have a few years ago I'm just so thankful for it especially somebody who hasn't spent his entire life um you know developing apps the ability to just take something you built maybe together with GPT and to feed it back and to ask it like why is this not running working why am I getting this error message like often with gp4 the very simplest thing you can do is just copy the error message you get in the terminal or in the console and just throw it back at it and it will tell you what's going on and it will give you a step-by-step tutorial on how to fix that so how often do you use it for debugging in your day-to-day and what would maybe be some recommendations for people trying to fix some mistakes which inevitably will occur cuz that's just how it goes I mean definitely for checking error messages um in that and there is it is a good question like why one would use web search or uh just a conversational agent to search for these error messages um I guess the main reason is because uh conversational agents can be a lot more concise uh and then in the web search regime you kind of have to go through different links and see which one answers your question um and that's why that scenario is becoming so popular among developers um but when it comes to sort of like debugging a whole bunch of code that where you exactly do not know where the error is coming from and so on um I've been surprised to how the usual practices in software engineering kind of remain the same in that sense in that you still have to modularize your code and separate it in different functions that are self-contained and documented um for better debugging and that is valid also for code that is generated by gp4 and other tools and um and that's just for you as a developer to sort of like know where the error is coming from and um really isolate you know where the information is coming and uh what are the possible error messages that are coming from that function so that you can also ask a more self-contained question for debugging later on to this tools like you can just like copy paste that function and you can say well tell me what can go wrong like this is the eror message that I'm getting this is the function tell me what can go wrong and it is this is much easier than pasting thousands lines of codes to the tool and having it you know try many different things that can happen there so it will help us as developers but it will also help the model to focus on the right things okay so tip of the day like keep your code um like compartmentalized and um if you're using it for debugging like try and tackle it one problem at a time I think this is one of the most common pitfalls I think you outlined that perfectly people just do a you know command or control a contrl c and then they paste the whole thing and like if they're writing in one file if it if the app might not be that large with people starting out building rappers often it's just like one um it's just one document they copy paste the whole thing and then they start asking questions that's really conf confusing to the model A lot of times so from the ground up like from the from just from the very first step thinking in terms of like code blocks or I like to always think of it as buckets um you know just like start multiple buckets and like populate them step by step will allow you um to fix the errors that might occur down the line all right so if you

enjoy this conversation and you might want to check out BME LinkedIn Microsoft for startups on X and Linkedin and if you want to learn more about the basics of using the openi API to build your very own chatbots I have a playlist on all that here where I lead you for it step by step and that's all I got for today see you there