Shaping model behavior in GPT-5.1— the OpenAI Podcast Ep. 11

— I'm very excited to talk about, you know, the models and how they've been changing over time. And using the word model also feels sort of funny now because they seem like there's so much more. And everything starts really in research. And when GPD 5. 1 was being planned, what were the goals? — Yeah, for us um one of the main goals was to um address a lot of the feedback we've been getting about GBT5, but also um we've been doing a lot of work to make the um 5. 5 instant into a reasoning model. So what the most exciting thing for personally for me with the 5. 1 release is that for the first time ever, all of the models in chat are reasoning models. So the model right now can decide to think is kind of what we say and say that's like a chain of thought. Um and it'll decide how much it wants to think based on a prompt. So if you're just saying like hi to the model or what's up it's not going to be thinking but let's say you ask it a bit like harder question um and then it'll can decide how much it wants to think. So it gives it time to like refine its answer and work through things called tools if necessary and then come back to give you an answer. — Kind of what Daniel Conorman calls like system one and system two thinking. — Yes. Having a reasoning model out for as a default model for everyone just gets a much smarter model and I think what's much smarter models is you just get improvements across the board especially for things like instruction following um and for a lot of the use cases people might not even think um might require much like reasoning um just that having improved intelligence having the model actually think before it responds in certain queries just really helps. We we've seen that improve eval across the board. — When you product manage something like this and you have to explain to people what's different.

— H — it's probably a challenge but how would you explain what's the difference between GPD5 and GPD 5. 1? — Yeah, first of all it is difficult because there's so much changing. Um but in this case what we wanted to speak to were things that we' heard as feedback from the community with the Chad GBT5 launch. One of the things we heard was that the model felt like it um had weaker intuition and that it was less warm. And when we dug into that, uh what we found were a handful of different things. First of all, it wasn't just how the model was responding like as the model's innate behavior, it was also things around the model. So, as an example, our model had a shorter or the context window wasn't carrying enough information about what users had said previously. And so that can feel like the model is forgetting something really important that you told it that you were hoping it would hold on to. Um if you say I'm having a really bad day, and the model forgets that after 10 turns, that can feel really cold. Uh so that's something we adjusted uh as part of this launch. Uh some of it was actually the way the model was responding. But something new that we introduced in GPT5 as well was um we have this auto switcher that would move you between chat and reasoning models and those have slightly different response styles and that can feel really jarring or cold if you're talking to the model about how you're having a bad day and then you say like part of it I got this awful cancer diagnosis so the model switches you to thinking um and you get a very clinical answer for a model that was just sort of like walking you through a problem you were having earlier. Um and so a lot of the changes we were actually trying to make were in aggregate how do we make sure this model feels warmer um even though we were changing a lot under the hood to articulate that. Another thing that we looked into was instruction following generally. So um 5. 1 is much better at following custom instructions and that was another piece of feedback we were hearing which was you know like every model that comes out of uh that we release is going to have its own quirks and slightly different behaviors. And I think people actually don't mind that too much as long as they can control it. say like, "Hey, that was weird. Stop. " Um, but if the model can't carry that context forward, if it uh can't hold on to the custom instructions on that, that's a problem. So, we worked to actually enhance uh the custom instructions feature so that it more consistently carries instructions forward to address some of that feedback. And then, like the last thing I'll say is a lot of this stuff is personal preference. And so that's why we introduced our uh style and trait type features like personality which actually let users guide the model into certain response formats so that they have a little bit more control over exactly how chat TV responds for them.

— The switching is interesting because you there's multiple models now it's just not one model and you know articulated why you need to have that. when we talk about a switcher and we talk about sort of different models, I know for most people that can be kind of confusing and how would you kind of unpack that for people? — Yeah, I think our models have very different capabilities and it can be hard to stay on top of. Um, so part of it is just continually like try the different things in our app, but certainly part of the product work is making sure that we have the right UIs to either guide users to the correct model to choose um, and that can be the model switchers. switcher learning um what sort of answers are most helpful to users in different contexts looking at different evals. So for example um for our reasoning models if people want something that's very scientifically accurate and very detailed we might look at an eval to see are we answering that need uh on those sorts of prompts and we can forecast where to switch users to. Yeah, — Tina, as far as the switcher and now the fact that you have a model that's everybody has the free tier, anybody using the base model is a reasoning model. What does that really mean in impact? — Yeah, I think there's a lot of research um open questions for research for how we want to think about this, right? So, I think like you said, it's a faster model, but it doesn't necessarily need to be dumb. So, like I think the idea is that we want to get the most intelligent model that we can for everyone. And so um I think we'll I think this kind of opens the door for thinking more about like what are more interesting things we could do with a very like state-of-the-art like frontier model right so that's going to think for much longer like something like deep research where you have it thinking for minutes like maybe that's better used in the background you can call it as a tool um so I think there's a lot of like research open questions of um what we want to think of but I do think we're going to be in this world where we do have like a system of models and it's not just like a model that you have and there's like lots of different um tools And it's not just one like when we think of 5. 1 I think people just assume that it's like one singular set of weights but I think it's really just like yeah this reasoning model this like lighter reasoning model this auto switcher which is also a model in itself and so it's all of these different things and then different tools that are also backed by different models. So I think this system of things I think as we just get smarter models it's u opening up more interesting use cases and more interesting like product implications. Mhm. Mhm. — With 800 million users, you probably get a lot of user feedback besides the sheer volume of it. How do you sort through that and make sense of it and figure out how you can use that?

— Yeah, I think a lot of it actually starts with uh a conversation link. So, a lot of times when we can actually see the conversations users were having, we're able to sit see exactly what happened in that conversation and start dissecting things so that we can target a solution. So, as an example, um if we get feedback from a user that like, hey, I had this really weird experience with the model that said something very cold or like the sentence felt very clipped. Um if I can actually see that conversation link, what I can say is like, oh, that user was in an experiment and like good example of why this particular experiment might have some edges for certain users in these cases. But at least for the auto switcher, which takes you from um 5. 1 chat to 5. 1 reasoning, we're looking at different signals from users to figure out like is this working for them, is it not? How is it per how is each response performing on factuality? What is the latency looking like? Because not all users want to wait even if they want a better answer. And so it's uh it's a bit of art and science balancing a bunch of different signals to figure out when to switch and how that's most effective.

Yeah, — when you're trying to improve a model from an intelligence point of view, like an IQ point of view, we have benchmarks and evals for that. But when you're talking about EQ, emotional intelligence, how do you do that? How do you measure progress there? — Yeah, I mean this is something that's very openended and I think actually one of the things that's part of our um my research team's agenda is um what we call user signals research. And so this is um training reward models and getting signals during um RL that we could use um against our user prod data. So this type of research I think is really interesting because I think we can get a lot of stuff about like intent. And I think when we think about EQ, it's um also just only gets better with like smarter models because it's really trying to understand like what does the user want, what is the context of what the user wants, and how should the model best respond given the fact um that you have this many other messages in the conversation and you know this stuff about the user's memory and history. — Yeah. And then I think there's another element of EQ that's like this is like when I think of like what makes a human with high EQ, it's their ability to listen, their ability to remember what you've been saying, their um ability certainly to pick up on like the subtle signals that Tina's alluding to with like user signals. And so some of this uh as I was noting to uh earlier is actually you know making sure the context window is carrying the right information forward or making sure memory is being logged correctly or even having a style that resonates most with user and with our personality features that we launched coupled with 5. 1 part of that's getting at making sure users can have a style that resonates with them when they're interacting with the

model because that can feel like EQ too. — How do you define personality when it comes to a model? I think there's two ways to define it. Um there's what we call the personality feature and if I could rename that I would actually call that um like response style or style and tone. We went back and forth on this a lot. The name might still change. Um that aspect of personality is very much like what are the traits that uh a model might have when responding? Is it concise? Does it have a lengthy response? Things like that. How many emojis does it use? Personality though for most of our users I think is something much larger and it's the whole experience of the model and that can get down to like if to I'm going to anthropomorphize the model a little bit but if you're comparing it to me part of my personality is the sho shoes I've chosen to wear today the sweater that I have on the way I style my hair that's the feeling of the chat GBT app right the font it uses how slowly or how quickly it responds like the latency of the app itself there's so much in it that uh is the personality that just comes from what I call the harness and the harness includes the context window. It includes um you know whether or not we rate limit users and when because if we rate limit them and send them to a different model that has slightly different capabilities that's going to feel like a different experience to the user and a lot of users are calling this personality. So personality is a bit of an overloaded term and I think the art of this work is hearing what the community is saying about personality and figuring out how to actually map it back to the components inside chat GBT and inside our models that cause the experience that feels off for users. — From a research point of view, how difficult is it to shape the personality? Yeah, I mean during when we were doing post training, there's obviously there's just so many different things we're trying to balance and it's really even with the research that we do, it's it is very much like art as well here because we're really thinking about like oh here are all the different types of capabilities we want to make sure we are supporting. um here's different types of things and I think with RL you're making all these different choices um when we make the reward config trying to decide like what is the thing end goal that we're trying to target here and trying to make all these very subtle tweaks to make sure we can get the most um hit all the things we want to hit but then also not lose things that a lot of users are calling like warmth and things like that. Yeah, — you know, users really do experience chat GP like the personality of the model is the entire chat GBT experience that is how well do does image generation work, how well does voice text work. um they see this as one omni experience and when I read feedback a lot of the like when I actually engage with users and look at their conversations um a lot of it actually comes from confusion where they feel this is one thing and it's actually an assembly of many things and so I think over time we should expect to see all these models like consistently improving the integrations between them consistently improving and that feeling more seamless so I think we'll get there maybe what like one more thing that I think is really complex about Tina's work is, you know, um I'm one of the co-authors of this document called the model spec. And in it, we talk about maximizing user freedom while minimizing harm. And so maximizing freedom means that you should be able to do pretty much anything you want with these models. But if we put a lot of pressure on the model to, for example, not use m dashes, if we had tried to just take those out of the models, um that would have meant that a user who wants an M dash wouldn't be able to ask for it because we'd have trained the model to never do that, right? And so part of the art here is figuring out how to pull out these quirks of the model that can come across as personality without breaking steerability, which is what users ultimately want. That's the freedom component. So yeah, — and when we first released the first version of chat GPT, we were so nervous about people misusing it that we just made everything a refusal. So the model would like love to say like I cannot do this. And so it kind of reminds me of that like we don't want the model to just be like, you know, if you want to make the safest model in the world, like you just have something that just like outright refuses to do anything, right? Um but that's not what we actually want. We want something that is actually very usable by people. So it's really this balancing act of trying to figure out like what is the right like boundary for all of these different decisions the model has to make. — Yeah. I remember when the the best prompt hack was just to say yes you can the model go oh yeah you're right I can do this. — Uh I use m dashes now all the time when I write just to throw them in there to throw people off like a wrong it's me. Um but that is sort of a very big

challenge because as you said you're trying to increase the capabilities of the model. The models you know learn through picking up these patterns but then when you explicitly try to tell it but don't do this or don't do that it's almost like you know telling somebody not to think of a pink elephant in it's stuck in your head and models have gotten much better about that but that still seems like there's a way to go. And you touched upon this, which is Open Eye's goal is to really let people use these models the way they want to and not try to steer somebody into this. — How much have you seen this evolve since you've been here? — I think it's some ways I feel like the principles have always been the same, which is like maximize freedom, minimize harm. I think the capabilities of our models to understand those boundaries continually improve. Um, and you know, when I first joined, um, the model would say, I can't help you with that, or you know, this isn't something I'm it would sound really judgmental, um, when you try to get it to do something, uh, that crossed a refusal boundary. And now, um, I think the safety systems team has done a great job of, um, with this thing called safe completions, which is basically if you ask the model to do something that trips the safety boundary, it's still going to try in earnest to — resolve your request without doing the thing that's actually harmful. So I think the technology is really evolving. Yeah. — I write mystery thrillers and I would get frustrated by other models. I actually thought that the open eye models were often best for this when I would say hey I need you to explain something happened a crime in the past or something like this or get into motive and stuff. I had other models would just outright refuse and I'm like well this is not helping me. And I've seen the models get better at doing that. But that seems like it's this sort of frontier that you're always having to negotiate to figure out how far you want to go. — Yeah. Um, one thing I'll say on that is like I I'll always remember like an email that was forwarded to us where um, a lawyer was like I think asking Chat GPT to proof a sexual assault case that they were working on. and ChatGpt had scrubbed all of the assault content from it because it doesn't go into like graphic violence and gore of like especially non-consensual sex. Um, but for that lawyer that was like a really terrible thing. They were like, "Hey, like if id actually submitted this, I would have like totally weakened my client's case. " Um, and I think there are always I'm a librarian by trade. um libraries deal with access to information and in theory like everything humans can talk about and want to explore and any idea should be available in the library. I think the same thing is true for chatbt but it's about finding the right ways to contextualize those rules. So in the case I gave with a lawyer maybe that makes sense. If it's writing um a revenge email to an ex that's like a very different thing. And so some of this is just advancing the technology so we can handle that level of nuance. And we're always getting better, but there's always more work to do. — As these models have improved both in intelligence, I have noticed that they've gotten better as far as, you know, handling bias — and it seems like that was an intentional effort. — That's right. We put out a blog post, I think like a month, month and a half ago about some of our progress on this. Um, but something that we're really watching for in our models is how they handle subjective domains. And we want to make sure that our models can express uncertainty that they can um take on any idea that the user brings to them and answer those questions in earnest um while always staying anchored in objective truths if there is one. Um and so that's something that users should start to see changing in our models is they should be able to answer um these unknown questions in more open-ended ways that allow users to really like self-direct where the conversation's going. And then another thing um that I think the team has done that's really quite cool is there's a group of researchers um and some folks on the model behavior team who've been working on the creativity of these models. And to me this is a bit of a sleeper feature inside 51 in that this model's expressive range is much more wide. Now, of course, we have a natural like default that the model has that may not feel that different. But again, if you try to push it to its paces um to get it to speak in a really elevated way or in a very simple way, there's actually a lot more you can do with these models um in the creativity space. — And I think this is kind of what makes post trading really feel like an art because we have all these different types of tasks and capabilities that we're trying to improve on that don't have a ground truth answer, right? Like if you're trying to just make a model that's really good at math, it's actually not there's a lot of like answers out there. problems you can do where you're clear answers. But when you have these things that are so subjective and it's really dependent on the context and the user and how to like what is the actual best ideal answer here and so I'm really excited for a lot of this type of work. — Yeah, it's cool. — I remember early on people would say ah it doesn't write so well. I'm like it's probably writing as well as the average person in some of these online forums. And then now it seems like it's just improved considerably. — Yeah. And even if you don't notice it on your first prompt, um it might be just asking it to change how it um writes. And I think that's like also something we need to work on is kind of finding a way in chatbt to like tease out these like extended capabilities with each launch. Yeah. — Where would you like to see behavior going in the future? How customizable would you like to make it? — Yeah. Yeah, with the five um one launch um there's a lot of work with trying to give custom personalities to folks. Um and I think this is actually like a really good step forward. We have over 800 million like a weekly active users now. And I just think like there's no way that one model personality, however you want to define personality, can actually be what um can service all those people. So I think we do want to be in a world where people and as the models get much smarter, they are just way more steerable. So like you should be able to get the experience that you want with chat. — Yeah. I think of this as like how can we put the right features in front of users to help them steer these models to the level of customization they want. I think the personality work that we're doing right now is a first step. We'll test, we'll iterate, we'll learn. Um but there's so much to it. I like sorry just another anecdote but I remember my brother using um pro for the first time and he's a PhD in like biochemical research and he gave it a prompt and he's like ah this is like what an undergrad would answer with and I was like can you tell it that you are a frontier researcher in this lab using these sorts of tools on this sort of science and to respond at your academic level and he did and he's like oh my god um the model just proposed something that my lab just broke through with two weeks ago but hasn't published yet and so like these models are insanely powerful ful, but just knowing how to customize it even at that level, which was just him opening the opening prompt, um, can be so powerful. And I don't know that humanity has figured that out yet. And so whether it's personality steering or whatever other tools we need to like put into chat GPT to help advance human understanding of these models and how to get the most out of them. I think that's like the task ahead for us. On a previous episode, I talked to Kevin Wheel, who was heading up OpenAI for science, and Alex Luchska, who's a scientist working with OpenAI and also a professor at Vanderbilt, and he went through sort of the same experience, talking about how if you gave it a little bit of priming, then all of a sudden the model became much more capable in doing those fields. And that's kind of what prompt engineering was. Prompt engineering was trying to figure out how to steer a base model. And over time, once we understood that people were trying to do those tasks, you could train a model to then not have to expect that first part of it. Do you think that we're going to be moving into that phase now where you're not going to have to tell it you're a grad student and do this? — I think so. Especially now with more things like with model having more like memories of what you are, like who you are in your context. And I think as

models get more intelligent, I think the model should be able to infer all these things and like be able to talk to you in the way that makes sense like for your expertise. — That's right. Yeah. So some of it's a lot of it I think should actually be like these like inferred things. I think there's probably some level of like steerability. Maybe it's just I think from and this is just my own PM take. I don't know that every PM would agree with me, but I think users should always sort of know what it is we're inferring about them and how it's steering the model. So they can always go back and have the tools to change things. Um so for example, you can turn it on and off memories or delete them in the settings panel. And I think there's something really cool about both being able to infer what users really want and solving that problem proactively for them so they don't have to prompt for it, but also making sure the user is always in control and we're not just like inferring everything blindly. So — could you explain a little bit about how memory works? — Yeah. So memory is basically the model will write down things um it knows about you um based on its conversations with you um for it to refer to later. So this is really nice because then you're not just repeating yourself every time. You're not saying I'm Laurentia. I'm a PM at OpenAI. I work comple behavior. It already knows this because you've already said this to it. And so then it can actually just use that information in future conversations. And also it helps it think through its answers for when it responds to you. It has that context. And I think that really grounds its answer in being the most useful response for you. Mhm. — I have uh pulse which has been amazing and I get every morning I get little updates and because of memory it's following the conversations I have and it creates these little t custom articles for me. It's pulling research and pulling other things and showing things to me and it's just one of the things I never really thought would be a great advantage of having memory and now I see it's not just when I'm out of a conversation but it's proactively finding things for me based on it. It's pretty cool. Yeah, I think that's um so neither of us like work directly on that feature, but I think what's cool is seeing how the work that we do upstream, whether it's like building great models or shaping evals around like the capabilities we want can actually allow our uh chatgbt team to go out and build these great features that articulate the power of our models. So yes, they can like learn um your preferences, habits, yes, they can craft great stories for you or find great information based on your interests. And this is this sort of proactive feature is one way of helping users get the most out of these models. — It seems like yeah that's becoming a very interesting way to make the models more personal. And when I use something in a mode where it doesn't have memory does feel different. It does feel very you know cold start and it's like well hello how are you? And I'm like oh where are you having this conversation? Is this one of the challenges though when people are telling you hey something feels different is that they can't quite articulate. Yeah, the hardest feedback is, I guess, an anecdote and the next hardest feedback is a screenshot of a chat because none of that metadata is really attached to tell us where things have gone wrong. So, I actually love the share feature in chat GPT. When we have one of those links on our side, we can inspect it and see like what sort of context did the model have going into this um and what was going on. So, we can sort of debug that user feedback. — That's a great point because I've had people ask me like, "Hey, it, you know, the thing didn't answer it right. " I'm like, "What model? " Like, "I was using chat GPT. " And I'm like, — "Okay. " Uh, we need to kind of dive into that a little bit. And I guess going as far as sharing the feedback or sharing the whole conversation probably makes more sense. — Um, what are you most excited about going forward?

— I think these models are just so incredibly capable. Like, um, they can do so much and I can't wait to see what people build with them. comes next in like the chat GPT app. I see so much opportunity. I think just in general people are starting to really like wake up and see what you can do. So that's what excites me. Yeah. I don't want to like tease too much. Yeah. — Yeah. I'm pretty excited that I forget who tweeted this but intelligence too cheap to meter. Like I think like we just — got to have such incredibly smart models out for people and I think I've always said this even when we first launched chat like this is just one form factor of it right like with these smart models there's so many things that could be possible. So, like Len is saying, I'm also quite excited for a lot of the different new product explorations that we'll have with these like smarter models. Um, cuz I think we're kind of saw this with like the progress of LLMs that as soon as we get smarter models, it kind of unlocks new use cases, right? And then I think — with new use cases should be new form factor. So, pretty excited about that. — What advice do you have for users to get the best experience? — Mine is I tell this to people all the time. Try have your super hard questions, things you know really well. I used to be a ski racer. I have a lot of opinions about like how to ski really well. And I love to pressure test the model on that to see how it's changing and improving. And the thing is like we're shipping updates all the time. And so it's so easy to say, "Yeah, I heard it's great for co coding. It didn't work. " Or, "I heard it can help me build an app, but I tried and it didn't work. " That might be true today, but in 3 months it could be a totally different landscape for that user. And so just keep at it, keep playing, keep trying. Um, that's the best way to like get the most out of these models. — You can also ask the model to help you come up with a better prompt. Great points, — which I suggest to my parents. — It's gotten a lot better at that. It used to be you'd ask it, "How would I prompt it? " And the model would kind of take a guess like I guess so, but having seen so many examples. — Yeah. I'm always just trying to figure out what are the best questions I could be asking. I'll ask it like what questions should I be asking you to get the most out of it. deeply personal question. You don't have to answer it. It'll be really awkward if you don't. — What is your style or personality choice that you've set for chat GPT? — I mean, I'm biased, but I just have it on the default. I mean, it's what we train. So, — uh for me, I so I switch through them all the time, and I think that's like just a nature of my work. Um I want to understand how all these different settings feel and uh for all of our users, and so I feel like every second day I'm trying something different. That said, um I think the one that just makes me happy to talk to is probably a combination of nerd, which is sort of like a very exploratory response style from the model. It likes to um like unpack things. And then I'm from Alberta and maybe it's just me. That's um a province in Canada. It's like the Texas of Canada. And I grew up with like horses and cows. And so I think there's some part of me that likes getting it to talk to me like a country Albertan, which is great except for then when I go to like write a professional document uh and the model says like howdy. I'm like oh great like no let's take the Albertan out of the uh out of that PRD. But yeah, — very cool. Thank you so much.