How difficult is AI alignment? | Anthropic Research Salon

- We are super excited to have everyone here, folks that we've met already, folks that are new. This panel is just gonna be really casual. And we have researchers from four different teams at Anthropic. We have folks from Societal Impacts, that's me. Folks from Alignment Science, that's Jan. Alignment, Finetuning, Amanda. And interpretability, Josh. I'm gonna start with asking a question to Amanda

from the Alignment Finetuning team. And I want you to talk a little bit about how you see alignment, what it means to you. You know, 'cause you're in charge of a lot of our work on how the model should behave. And why should you be the philosopher king that decides how Claude behaves, what its characteristics and attributes are? - I mean ask Plato. He's the one that decided I should be the philosopher king. (Alex laughing) The question of, like, what is alignment, maybe this is, like, a slightly spicy view that I have. I think people are very, very tempted to spend a lot of time trying to define this concept because there's, like, lots of ways of doing it and they, you know, I dunno they have, like, social choice theory in the back of their head and they're like, "Oh, well if you imagine everyone has a utility function, there's, like, limits on exactly what you can say about how to maximize all of those utility functions, et cetera. " And I think I'm more inclined to just be like, we kind of want things to go well enough that you can iterate on and improve them later. And the bar isn't some like perfect notion of alignment. I'm sure there is that concept. One can define it, one can argue about it but for the most part the initial goal is, like, let's just make things like go well and, like, meet a certain kind of, like, lower bar where that is, like, you know, if it's not perfect, if some people, like, don't like it, you can just improve on it. So, like, my view of alignment is actually probably, like, I mostly want to, like, hit that, like, and iterate from there. In terms of, like, how the model should behave and how I think about that, I think I've spoken about this before but my basic concept right now for the model is trying to get it to behave the way that I think, like, a very good, like, morally motivated, like, kind human would act if they found themselves roughly in this circumstance. It's a little bit strange 'cause they also have to find themselves in the circumstances of, like, being an AI (panelists laughing) who's like talking to millions of people, which does in fact affect how you behave. Like, maybe you would normally be willing to, like, just chitchat about politics with someone, but if you're gonna be talking with millions of people, maybe you'd actually be like, "Hmm, I should maybe be, like, a little bit more concerned about potentially influencing people. " But I do think it's actually an important model, which is, like, sometimes people are kinda like, "Oh, what values should you put into the model? " And I think I'm often like, "Well, do we think this way with humans? " Where I'm just like, someone just injected me with, like, value serum or something and I just have these, like, fixed things that I'm, like, completely certain of. I'm like, I don't know, that seems, like, almost, like, dangerous or something. Most of us just have, like, a mix of, like, things that we do value but that we would trade off against other things and a lot of uncertainty about different, like, moral frameworks. We hit cases where we're suddenly, like, "Oh, actually my value framework doesn't accord with my intuitions," and we update. I think my view is that ethics is actually, like, a lot more like physics than people think. It's actually like a lot more kind of, like, empirical and something that we're uncertain over and that we have hypotheses about. Like, I think that if I met someone who was just completely confident in their moral view, there is no such moral view I could give that person that would not make me kind of terrified. Whereas if I instead have someone who's just, like, "I don't know, I'm kind of uncertain over this and I just, like, update in response to, like, new information about ethics and I, like, think through these things," that's the kind of person that feels, like, less scary to me. So at least at the moment I'm not, this isn't a claim that this is somehow going to, like, completely, like, align models or anything. But that's the kind of immediate goal. I realize I've talked a lot and you asked about the philosopher king question. (Alex laughing) I guess. Okay. I guess I should give a quick answer to that. There is also this question of, like, well, in the kind of, like, should you put values into the model? Maybe I've partially answered it where I'm like, "The models should just be uncertain over values that exist in the world. " And so ideally it's not just someone injecting their values or their preferences, nor is it something like everyone just voting on values to put into models but instead like people that are uncertain and responsive to these things, models should also be like that. So maybe that's kinda my view. - Okay, we're gonna come back to that. And I'm gonna ask Jan, why is Amanda's view completely wrong

(panelists laughing) and why is this not enough to align models as they get, you know... - I see. (Alex laughing) - She didn't say that. But you know, we're playing up the tension between the bets. - Yeah, I imagine if everyone was, like, kind human that was just trying to, (Alex laughing) you know, act morally, I think what Amanda is doing is very practical, right? Like, can we just like make the models be well behaved now and, like, where would we go with this? Like, if FI is doing more and more complicated things. Right now, like, if Amanda does this character work and then she reads a lot of transcripts and you're, like, "Okay, this is, like, I like this, this model is behaving morally," I'm just picturing this is what you're doing. (panelists laughing) But, like, what do we do when the model is doing really complex things? And it's just like an agent in the world, it's, like, doing these, like, really long trajectories, it's doing stuff that we don't understand, like doing some bio research and we're like, "Is this dangerous? Like, I dunno. " So that's the challenge really interested in. Like, the super alignment problem. How do we solve that? How do we, like, basically scale this beyond, like, things that we can look at, if we can look at it we just do some RLHF, it's great. Or you do some constitutional AI. But how do we know that our constitution is actually getting the model to do that, the right thing that we actually want? So I think that's the big question in my mind. - Do we get to respond? - Yes, you can respond. - You're only allowed to disagree. (laughs) - Okay. I don't really disagree, though, and I can't lie. - Need to up the disagreement feature, you know? - I'm usually so disagreeable as well. It's just terrible. This is what philosophy taught me was how to be disagreeable. I guess, like, my thought is that the way, I mean, I think of my work as doing several things, but one of them is being kind of iterative towards alignment. So in a lot of cases you're actually trying to get the model to kind of oversee its own, you know, so it's not, like, me, my eyes can't look at that many transcripts or something, but I can get models to, like, look at these things and, like, if alignment is iterative, I think my worry is that if people neglect the kind of, like, the ground and just think, "You can just have like a pretty bad model and it's just gonna help you with these things. " I'm like, "I'd kind of rather that you had the most aligned model trying to, like, then help you in the future and, like, that kind of iterative work. " - But how do you iterate when you can't read the transcripts anymore? And you have to rely on the aligned model. But then how do you know that it's actually trying to help you? - Yeah, so, like, current cases, it's sort of like everything that you are using to verify that the base model is aligned is, like, what you are relying on with, like, making sure that, like, another model that's, like, trained by that model is itself aligned. And I think this is, like, fine when models are, like, less capable, but to scale it to something with much more capable models, you'd actually have to have, like, a greater ability to verify that. - Yeah, what do you do then? (panelists laughing) - Oh, do you just want my plan? It may just be that it's just all fine and they just supervise themselves and they're all really nice. You know, that would be, I don't wanna rely on it, but, that's not my actual plan but, you know, I'll defend it for these purposes. - And one of our bets, you know, in order to, like, you know, guard against the case that a model

might be very deeply trying to sabotage against this process is interpretability. How do you see, like, interpretability as a bet, you know, situated among, you know, the more straightforward alignment approaches, you know? Is it just as simple as like, "Ooh, we find the nice feature and we, like, up the nice feature, we find the evil feature and we, like, drop the evil feature," right? Or is it, you know... - So I feel like everything in AI is like that meme with the bell curve and, like, the idiot and then the really sweaty guy talking a lot and then the Jedi who agrees with the idiot, and, like, there is a possibility that, like, it turns out that the secret to alignment is just turn on the nice feature. (panelists laughing) Like, for a sufficiently galaxy brain version of, like, the nice feature. I think (laughs) that in some sense though, I'm hoping that interpretability is also like the Jedi version of just, like, "Well look at well how the model's doing things and check that it's safe," which is, like, maybe very hard, but also if you could do that, (clears throat) would potentially just answer your question. I think that, like, one of the things that are sort of, like, comes up in both, like, the near term and slightly longer term is, like, you wanna understand, like, why the model did one thing instead of another when you could come up with, like, plausible alternative explanations and one way is to ask it. But the issue is the models are so analogous to people that they'll just give you an answer to why they did that, you know, as anybody would. But, like, how do you trust that any of that stuff? And it's like, "Well if you could, like, look inside and just, like, see what it was thinking about as it was giving you the answer. " And, like, even now with stuff like the SAEs, you can, like, see there's some feature active and, like, when else is that happening? And it's, like, "Okay, it's like other instances of people telling white lies. " And you're like, "Well, then maybe the model is telling," it's like, that's on the Jedi side, I think, right? And so I think that, like, trying to just, like, look inside, see if we can figure out what the parts are and then see, like, are you comfortable with using that part when it does other things, is the basic bet. - I have a question. - Yeah. - How do you know you're turning up the nice feature and not the pretend to be nice feature whenever humans are looking feature? - Yeah. (panelists laughing) I mean, like, I think if it's on the control side, right? Like, how do you know? And I will say actually many of the features are, like, a little bit, like, deceptive also, when you just, like, look at what I thought the style impacts team did some great work here with, like, S and Deep where, like, you look at it and you think it's a feature which is, like, you know, like, age discrimination, bad but, like, actually it's the age discrimination good feature. So actually no I think vice versa. So you tried to turn it down but actually the opposite behavior, so it can, like, be hard to understand, like, all of the cases. I will say that some of, like, the circuits work where you're like, "Okay, well how did this get generated? " Gives you some clue about the scenarios. Like, is it looking for a person in the context? I also think we're gonna need model supervision as well. Like, hopefully an impartial model that isn't, like, that's a little more scary depending on pre-training has, like, what do you call it, incepted in all of the models that they want to evade detection. But, like, sometimes you just, like, look at enough examples and it becomes clear and you don't need ten, you need thousands. But, like, Claude is very diligent. - Yeah, I'd love to hear a little bit more about, like, what you see as, like some of the things you're struggling with or thinking about when trying to, you know, approach this problem of, like, how do you, if you can't read the transcript, like, you know, what the heck are you doing anymore if you can't provide any meaningful, like, alignment signal?

- I mean I think a really obvious thing we should do more of is like what Amanda said. Just say can we just get the models to help us? And then the question's, of course, how do we trust the models? Like, how do we bootstrap this whole process? And, like, you know, you could hope maybe we can, like, leverage the dumber models that we trust more, but they might also not be able to figure it out. And so I guess like there's the whole scaled oversight work where, you know, we are exploring various, like, multi-agent dynamics to try to train models to, you know, help us figure out these kind of problems. It seems like overall, like, these problems might, like, maybe the problems are, like, all kind of easy and, like, we can just do the Amanda thing and just, like, make sense of data. Or it's, like, really hard and we have to figure out, like, fully new ideas and approaches that we don't know yet. I think kind of our best bet in the medium term is to try to figure out how to automate alignment research and then we can hopefully get the models to do it. So now we've reduced the problem, like, to, "How can we trust this model to do anything? " To just, like, "Well, can we just trust it to do this, like, much more narrow thing of, like, do some ML research which we understand, like, reasonably well? And how can we evaluate it or, like, how can we give it feedback on those kind of things? " - And what you do, okay, you go ahead. - Oh, I was gonna say I think we're in this special zone and I'm terrified about what happens next, but we're in the special zone right now where, like, there's something happens on the forward pass but a lot of the information you need is passed back through with the tokens it generates. It's just, like, the chain of thought is really important for the models to be very smart. And, like, the chain of thought is currently in English. And so then you've got, like, a factorized problem, which is, like, is the chain of thought, like, reasonably safe and, like, is that faithful to what's happening on, like, one forward pass? And, like, maybe like you can do some interpretability to, like, check that piece and then you can just, like, you or models can inspect it and you get the other piece. And the horrifying moment is, like, when all of that, the very long thing, like, isn't in English, right? It's in, like, some inscrutable thing that, like, you've learned through, like, crazy long RL to do it. And, like, I think that a big challenge can be crossing that gap where, like, none of the intermediates are intelligible and there's, like, massive amounts of compute before it like drops out something that people can read. - Maybe something I'd be interested in getting, like, people's mental models on is, like, what do you think

are some signs that, like, we'd be in an alignment is easy world and what do you think are some, like, signs we might see in the next few years that, like, we're actually in an, oh, alignment is, like, really hard world. - I feel like the model organisms work is, like, trying to figure this out, right? Like, can we try to deliberately make deceptive models or misaligned models, models that, like, try to do shady stuff. Like, how good are they, how hard is it to do it? I mean, we might fundamentally go around, like, about it the wrong way and that's why we fail, but if we do succeed it should tell us, like, how close are we to that kind of world? And then, you know, once you have your deceptive model that does all these shady things, like, can you fix it? What if you don't know what the model, if it's a shady model or not, right? We are playing these interability audits, which I'm very excited about, but I don't actually know what the state is. - Well, we haven't done the audit yet, but your people are trying to make it and we're trying to catch it. (panelists laughing) - So we are making some shady models and then they have to figure out which one is the shady one, right? - Yeah, and in what way is it shady? 'Cause most of it is still fine. I mean, I think one thing that the interpretability stuff has been interesting is, so when you do these, like, unsupervised things, you're, like, "Okay here's a million different, like, features, a bunch of those correspond to personas. " The model could inhabit any of those, which include all sorts of, like, deceptive behaviors. Right? That are out there in the world, right? The model, like, knows about bad people and, like, bad motivations and so, like, the fact that raw capability exists is just gonna be baked in and then the question is like, is it doing that stuff or is it doing the good stuff? Especially, like, this question of, like, when Amanda goes and shapes the base model, right? Which isn't really anything, right? Into like an agent or an actor which is supposed to embody some of these and not others, like, can we tell exactly what it picked up there, right? From the finite dataset, right? Which is being used to, like, do that shaping? So that's a question. I mean, I think, that, like, some interpretability stuff, maybe some influence function. I mean, there's different ideas there to ask, like what did it get from this process? But, like, that shaping process is gonna be really, really important. - Yeah, and maybe a sign that seems important is something like how robust or, like, if you have, like, modal organisms work and then turns out you just put it through some character training and it just comes out being really nice again, then I'd be like, "Okay, that's a good sign, Re: the kind of world that we're in. " Whereas if it, like, is just kind of, like, shallow, like, shell on top of, like, I dunno, the same behavior, then I'm like, "Okay, we're in a slightly harder world. " Only slightly harder. - How do you distinguish, like, whether it's shallowly aligned or deeply aligned? - I think, so, I mean, I feel like there's a lot you could do because I think interpretability is kind of like one of them. There's other things that feel, so in terms of, like, model organisms, I guess my hope would be you'd actually have a kind of like a red team, blue team set up where you have a way of detecting whether the behavior that you have like instilled in a model is still there. And my job is actually to not know. And in fact it would be really good if I'm trying to train the model, I actually just don't know what it is that you've done because that's, like, a better way of testing whether my intervention's actually working. Whereas because otherwise it is just so hard to not, like, just try to train to the test or something. So I almost want to be completely ignorant of it. - Yeah, maybe we should play an Alignment Finetuning game. (laughs) - Yeah, he misaligns it and then you align it and we see who wins. - Yeah, no, I've said to people before, "Don't tell me how you did this, because I want to see if this helps. " - Can you fix your sleeper agent? (Amanda sputtering) - Possibly. We could play that game. I think I might need to know less. (panelists laughing) - Make another one. - Yeah, make another one that's worse. - That is an excellent transition into questions. So we are going to pass the mic around. Aaron is kindly going to do that. Please raise your hand if you have a question for any of us. Oh, perfect. - Hello. I have a question about, well, everything you've been talking about.

So when we talk about alignment, we're talking about, like, a singular forward pass typically, right? So, like, alignment at inference time. So if I'm using one of the models via the API and I'm building my own sense of, say, cultural alignment and I've got a bunch of different agents set up talking to each other, trying to deliberate with that sense of kind of inner conflict that Amanda referenced as useful in what we do as humans to align, like, we go back and forth, we think, you know, we go through various cognitions. So if I'm trying to create this kind of multi-agentic deliberation thing, but I butt up against this aligned model who is so unwanting to deliberate with its own self, because all of them are like, "I'm sorry, I can't talk about that. " And so you just get this endless loop. Did you have commentary on that? 'Cause many people, we're not all using Claude in a single inference forward pathway. So, yeah, I hope someone gets the gist of what I'm on about. - Yeah, I need to think about it. So I guess, like, is the thought, like, to be clear, I'm not thinking necessarily that you have many agents. In the same way that we can be singular agents but still, like, deliberative. And I actually think that, like, there's a sense in which the more fractured an agent is, the more worried I am because from, like, an interpretability perspective and also even just like a predicting what that agent is going to do perspective, they're more unpredictable. So I guess I'm thinking in the same way that, like, maybe, I guess a different way of asking this question is just do humans have to do this? Like, my sense is that we're often just very willing to, like, reflect on many things, we go back and forth, we come to, like, conclusions, and so in the same way that, like, a human would think through any kinda standard problem that they were, like, facing, I'm imagining that, like, moral deliberation in the model's just gonna look kind of like that. More like the deliberation of a single model than, like, multiple models weighing in, if that makes sense.

- One more question here. - I want to try to draw maybe a relatively strange parallel between, like, Hannah Arendt's work on the banality of evil, the idea that most humans tend not to be evil, but when put in certain situations, the coupling constants between humans being so huge, the evil comes as an epiphenomenon of the system, right? And so what occurs to me is as you talk about model alignment, you're focused on one model, that has been most of your comments. How do you think about the coupling not only with society but as you work on agents and potentially millions and millions of agents, that sort of epiphenomenon of those systems? (Alex laughing) - I can talk a little bit about that. I mean, I think broadly when you think about safety and alignment, you have to think about it from a systems standpoint, you can't just think about it from, you know, an individual models perspective in isolation. And I think we've seen a lot of work where, you know, a lot of jailbreaks operate by pitting different values sort of against each other, putting the model in a difficult situation where, you know, it's designed to elicit what would ordinarily be a harmful behavior, but which, you know, in the context of the question the model thinks sort of is the right thing to do. And so, you know, there's a variety of tools you can use to do that. I mean, for one you can include a lot of those, you know, systems level integrations in the training process, right? And give them all exposure to a wider variety of situations which force it to consider, you know, the broad context in which it's answering questions. Now that leads to other challenges and another sort of, like, you know, fall off issues if the model is reasoning about, you know, the effect of its actions. But I think, I agree with the point that you can't just consider a model sort of in isolation. - In some ways, like, this is a thing that I've thought about in the context of like this notion of cordability or some, like, do you know, so, like, models that just are responsive to, like, what humans want versus models that, like, have values in a sense and are maybe, like, willing to actually be a little bit unethical, and, like, the banality of evil point feels especially relevant if you're, like, thinking of models as just, like, doing whatever humans say because in some ways, like, that is the idea that if you have a society that, like, either collectively just like allows for harmful things to take place or even endorses them and you have models, this isn't necessarily people misusing models, it's just that you'd be using models to, like, facilitate some kind of, like, harmful activity. And so I think there is, like, fundamentally actually a tension between having more models be cordial to the very least individual humans and having them be, like, in a sense, like, aligned with, like, all humans. And it's really important to recognize that tension and I think when people don't, they think, like, it's a failure that the model, like, didn't do what I said. But I'm like, hmm, there's a limited sense in which I think models should be more cordial to, like, humanity than not and should be willing to, like, you know, like, so there's like when push comes to shove. Like, but that doesn't necessarily mean being cordial with each person, so I think that could lead to that kind of, like, situation that you mentioned. - Hi.

So it seems like we've got Jan who's working on Intent Alignment, making sure the models do what we ask them to do, Amanda working on values alignment, making sure the models are, like, reasonable, kind entities and then Josh working on Interpretability, which is, like, allowing us to verify that the techniques for those other things are in fact doing what we want. If you were all to succeed in your areas, would that be a complete solution to AI safety or are there pieces missing and if so, what are they? - There's also, like, a lot of people working at Anthropic who are not at this panel. (audience laughing) We are oversimplifying a little bit. - Yeah, and I was just gonna say we also, you know, have the Societal Impacts team, which thinks about the model's impact on society, you know, writ large. And so I think, you know, you could have the most perfectly aligned model, but aligned to what? Who's using it? You know, for what purposes, you know? And I think the broader societal context is super something we are very attentive to. - Also, if you're just interested in just more lists, like, we already mentioned the model organisms work. There's, like, you know, jailbreaking robustness, there's control, there's, like, trust and safety. Yeah, there's a lot of other efforts that I think are also gonna be important. - I want to add an almost, like, I dunno if this is pessimistic or just, I don't think it's pessimistic. But I think that like there's also this way of talking about alignment and the alignment problem as, like, a single, like, theoretical problem or something like that where people will be like, "Does this solve it? " and somehow it's never felt right. It feels a little bit like, you know, I don't know, yeah, it feels more like, to my mind I'm just like, "Look, problems might just arise that we're not even thinking of now. " And in fact that's like very, very common in, like, many, many disciplines and I kind of expect it to be true here and it would be really dangerous, I think, if we were just like, "Oh yeah, we've like solved this problem," because I'm just like, I dunno, it could be the actual problems that we've just not thought of yet. - Yeah, unknown unknowns. - But we should solve the problem. And then once we did we should say that we solved it. (panelists laughing) - Another question. Yeah, Jan was talking earlier about using dumber models to evaluate smarter models over time. And I'm wondering to what extent you see, like, the Groking abilities in models where, like, suddenly it's really duplicitous or do you sort of see, "Oh, it's lying, but it's very bad at lying and now I can catch that and maybe, like, nip it in the bud while it's still weak. " - Yeah, I mean there's, like, plenty of examples. Like one that I remember is, like, GPT-4 could just read and write in Base64 super reliably and 3. 5 could not. And so if you use 3. 5 to oversee 4, it's, like, really easy for the model to get around this. - But I mean, like, on a level of, like, single Epoch or something. Like, you see over time, "Okay, kind of reads Base64 sometimes more and more," and, "Oh, suddenly can do it perfectly. " Or is it... - Oh you mean like should we use checkpoints? - Yeah. - So don't use the previous generation, just, like, you know, we ensure even spacing and capability space by like just, like-- - Yeah. Exactly. - But then also, like, the trust is not, like, a binary thing, right? Like, you trust it less and less the less you know about it and the, you know, smart it seems. - I will say one, like, you know, right side of the distribution Jedi moment was like, (panelists laughing) that the features, you know, just like also work in Base64 So it's like, is the model talking about California in Base64 or, like, a story about children lying to their parents in Base64 and it's like the same things activate? And so, like, we do get sometimes get lucky as, like, the models are like very capable. Part of that is they have some, like, very general synthetic thing and, like, maybe you can just like tap that to get some generalization. That would've been like pretty, pretty tough out. There's also the early paper of just, like, tell the model to do what's best for humanity version of alignment, which is definitely, like, (panelists laughing) maybe that'll work, right? You know, we could get lucky. - All right, I think we are going to wrap up with the formal panel. Thank you to all of our panelists so much for the great discussion. (audience applauding) But we're gonna be around for a lot longer to continue the conversation. Feel free to find any of us and we're excited to chat more. Thanks so much.