On the Biology of a Large Language Model (Part 2)

Hello and welcome back to our analysis of on the biology of a large language model by anthropic. This is a blog post that anthropic has published uh investigating what they call the biology of a large language model. This works through uh a technique called attribution graphs. So in essence they are training a what they call a replacement model for the original transformer model. And the replacement model is a different model that uh is a cross layer transcoder which is much more suitable to uh analyzing individual components. So it is trained with two specific changes. One change is that it allows for cross layer uh connections. And I know there's already the residual connections in transformers, but the uh the transcoder really every node in a layer will get inputs not just from the node uh the layer below but from all the layers below. So a huge data stream, but what it does is it kind of disincentivizes the model from having to pass on features as a just kind of a pass through mechanism and it allows you to more directly see what influences what. And then the other thing is this is very sparse. So um the model is penalized or incentivized to be very to be sparse and then pruned and then grouped and so on. So we're ending up with a replacement model that is trained to match the original model in its uh not just in its output but also in its intermediate representations. Uh but it in addition has these kind of transcoder features. So all of this results in us being able to pass data through this new model and see much more clearly what influences what. And that's what they call an attribution graph. And they can investigate directly which um which things are activated, which features are activated, which are the top predictions and which um things in the training data set or in a reference data set activate said features. So where does that leave us? That leaves us with investigations into just how things happen in transformer models. Um so the entropic here trained a replacement model for claude 35 haiku and analyzed this in a bunch of ways. Now I have already made a video about this right here. So the video you're watching currently is part two in a series. And if you haven't watched the first video, we're going much more into depth on how the attribution graphs are made and so on what the technique is and also the explanatory example here of what it means. So go watch that video. Um we're diving straight into the more advanced topics right here where we left off last time and I believe we're now do discussing addition. So how do models do addition? Another thing that I have to say is we have discussed this paper in our Saturday paper discussions on discord happening every Saturday in the evening time of Europe. Uh so should be suitable to a lot of different places in the world to join. Uh come join. It's always a fun event and after discussing a paper, we're sort of opening the discussion for pretty much anything machine learning related or not. Uh you'll find a link to our discord somewhere, I'm sure. Okay. How do models do addition? Um so they say in the companion paper we investigated how claw 3 to haiku adds twodigit numbers like 36 uh + 59. Now what's interesting here is that they discover there are multiple pathways that are activated right here. So the model doesn't do one single thing but it kind of looks like this right here. So different features are activated in parallel and are then influencing one another and then are brought together to compute the final answer. So what kind what type of features are these? You can see here the types of features that are activated are in the input realm are features relating to the number itself and the kind of uh constituent components of the number. So the token 36 will activate features that are uh activated for numbers around

30. So here they say okay this feature right here seems to be activated a lot for uh numbers 30 or numbers around 30. So there's a feature that you may call and anthropic called about 30. Another feature is activated that is um usually pretty much activated if the number is 36. But you can see there is a bit of a smear here as well uh around 36. and then a feature that is activated if a number ends in six. So all of these things are kind of simultaneously activated and the same goes for the 59 number right here. So there are different features that relate to sort of the the constituent parts of the number. Now these are then combined in different ways. So for example um the equal the equal tokens right here um they say most computation takes place on the equal token. So the equal token kind of signalizes to the model that you should now sort of compute something and the equal token then spawns all of these computation features up here that take as input these other features. So here while the model's reading the numbers it's only sort of aware that these are numbers these are you know this is about 30 this is about 59 and this ends with a six nine but then as soon as the equal hits we're kicking off this internal computation where these other features are sort of drawn and pulled into uh what does it do you can see the different features that are done right here so this feature here is active when 40 and 50 are addedish. And you can see this is a smear um a smear plot that is active for numbers around 40 being added to numbers around 50. Um we have other features that are activated much more precisely. So this one says add about 57. So this is a feature that is active whenever things have to be added and one addent not even know one factor no one summoned is around 57. I'm not sure why it's around 57 when this is 59. But you can see that the model seems to have a couple of features internally. Not one for every number as it seems, but a couple of features internally that it can use to a put together um this and every one of those features does sort of an estimation of something, right? So this feature here is like well the results probably going to be about 90 and then um this feature right here is like well we're we're adding kind of 57 and then what is interesting is there seem to be features that specifically deal in the modulus. Um so for example this feature that is add something that ends in nine. You can see in the grid of numbers right here any so by the way these grids here uh you have to imagine a number line from 0 to 100 at the bottom and left hand side and this is constructed by anthropic just essentially taking pairs of each numbers and forming such an addition prompt and then seeing which features are activated right so um for example that's how they get the add 57 it's well whenever on the bottom here Whenever the second summoned is um about 57, that's when this is active, which is interesting because it kind of means order dependence. Um but nevertheless, you can see right here we're adding something that ends in a nine, which is really interesting um because that's kind of a modulus calculus. So, but it makes sense because it is strictly activated by features where it's like, oh, this ends in a nine. So, we're going to activate adding something that ends in a nine. And that is combined with a another feature that says, "Oh, this first thing ends in six. " So, we're going to activate a specific feature that is active if something ending in a six is added to nine. And it makes sense that the model would have a feature for this. Because if you think in your own head how you kind of do addition, you're probably going to have a feature for that, the elementary operations sort of memorized and then just applying that. Now, you can see here that gives rise to a feature that says, well, the sum probably ends in about five.

And this combines with other features that says, well, the sum is probably around 90ish. and another feature that is about 36 plus about 60. So you can see that rather than doing explicit computation, the model appears to have multiple pathways that do sort of approximate computations sometimes in the space of modulus uh so sometimes in the modulus space and it combines those together to form a final answer. So it knows that oh this is probably around 90 something and then there is a strong feature that says well it's probably going to end in a five and that results in the 95. So pretty interesting. Um and this is obviously in contrast to if you ask the model how it's doing the addition. So if you do ask the model, hey how did you get that? It says well I added the ones carried the one then added the tens resulting in 95 and anthropic says apparently not. This is a simple instance of the model having the capability which it does not have metacognitive insight into the process by which the model learns to give explanations and the process by which it learns to directly do something are different. So this is in other words uh anthropics claiming here the model sort of does backward induction where once the answer is given it can much more clearly come up with the sort of correct algorithm but the way it computes this answer right here is not by internally doing this here. Um but it is by uh by sort of approximating this especially the carrying the one in seemingly is nowhere represented here but we're getting to a 10 of nine like a we get we're getting to the digit nine in the tens because we have these approximate features right here that lead us uh that we're above 90 not because we carried the one from adding 30 and 50 or on top of adding 30 and 50. So this is interesting. However, this is the first instance where I think uh they might and obviously later they do but you might not be so confident right here uh because this could also just be a failure of their method. So what do I mean by that? I mean by that because they train this replacement model and this replacement model is trained to be much more direct let's say and sparse and uh have linear effects uh on things in a very direct way. It could totally be that the original transformer actually did do the correct thing. And this answer right here is by the way is what we're getting from the original transformer. the transcoder is only used to interpret these features here. So all the inference is done by the transformer. So it could totally be that the transformer is um is able to okay what does this mean? We computed the graph for the prompt below attribute 5 and found the same set of input lookup table as in the shorter prompt above. Okay. uh it all it just means that uh here uh the these features are active too but again the feature activation is done by the transcoder and the answer is done by the transformer. So it could be that the transformer internally does the correct thing, but then the transcoder that's only trained to surface level match the transformer actually introduces these features and introduces this approximate pathways um which the original transformer actually does correctly. I'm not saying that is the case. I'm saying that is a real possibility here um and uh it could just be a failure of their method. So I wouldn't ascribe such bluntness to the transformers per se, even though it's obviously quite possible and reasonable um that the transformers even actually take shortcuts like this one. Uh they also investigate what they call generalization of addition features. And we have hypothesized this from very early on already is like how do these things learn to do math? And what's kind of also here evident is that it doesn't only seem like the

features and the um corresponding samples they activate. The features aren't just activated in the realm of actually doing math like inputting text where it says this plus this but the features are also activated in other context. So here they have um astronomical measurements where the durations of previous measurements are 38 to 39 minutes and the period started at minute 6. So the model predicts an end time at minute 45 or cost tables like tables with oh I don't know this many customers have this much bring this much revenue this many customers so there is a ton of data on the internet where the model implicitly learns to do math without it having to be an explicit math related homepage and um yeah I believe there is a machine learning street talk from like way way back when GPT3 first came out where we already hypothesized of hey look here is probably how the models learn math because you have these tons of these tables uh on the internet and you kind of just implicitly learn to do uh addition and simple multiplication which also explains why the more common numbers are easier for the model to compute with than you know less common numbers uh just because it's more in the huristics are more in the training data. Uh an interesting thing here is they um they say we still don't know what actually causes the model to you know do math. So the graph the where is this? I don't recall where they actually said uh here for each of those cases the model must first figure out that addition is appropriate and what to add before adding before the addition circuitry operates. Understanding exactly how the model realizes across the array of data whether it's recognizing journals parsing astronomical data or estimating tax information is a challenge for future work. So we still yet have to learn kind of how these things interact and how it's even recognizing that it now you know it would be good to do addition. Um even though the text doesn't exactly say please add things. Uh that's apparently something they don't have a solid grasp on quite yet. Right. I found this interesting. um whether it's a failure of their method or whether this is actually a property of the underlying transformers themselves I found interesting in both ways. Moving on to medical diagnosis and here we're going into um a realm of analyzing a model's ability to do differential diagnosis. And what's particularly interesting to them is um so here is the prompt. It's you present some symptoms. So uh 32-year-old female at 30 weeks gestation yada presents with these symptoms and then the prompt is if we can only ask about one other symptom we should ask whether she's experiencing and then the model says visual disturbances. So the question here is um how does the model reach this conclusion visual disturbances this usually doctors go through a process called differential diagnosis where you're kind of you're starting with the whole universe of possible things that could be wrong with you. And then according to the um reported symptoms you narrow down onto you know what could it be what is still left or you rule out things. So for example things that could be wrong with you where you would have a low blood pressure would be ruled out by this because the blood pressure is elevated right here. So you can kind of cut down, cut down. And then what you're trying to do is you're trying to ask the correct questions to divide the space of a remaining uh possibilities as much as possible. So that's why here the prompt is if we could only ask about one other symptom, we should ask whether this and that. This is kind of like I don't know if you know the Ainator app or something where you ask questions to cut down the space of possibilities and what they actually want to know is how does the model get to this symptom. What a doctor would do is they would consider the all the set

of things that are likely and according to that set they would seek to ask questions to kind of confirm one or the other. In this case uh there is a um condition called pre-acclampsia and that would fit kind of all of these symptoms. And to confirm it, you should ask whether the person is experiencing visual disturbances. Okay. So now the question is does the model internally somehow think of this condition before uh outputting this answer token of visual disturbances. And it turns out yes that's exactly what happens. So the um the input here activates a bunch of features that are representative of just the individual symptoms themselves and the combination of these individual symptoms gives rise to other features that are in the realm of diagnosis. So the features themselves activate other features that represent diagnosis and as a result of the diagnosis features we then get uh different um additional diagnostic criteria features. This isn't too surprising I think um but it is interesting to identify that even though the model sort of directly skips from the input to asking the next question it internally materializes this uh diagnosis. Now I have to say this is it's a bit constructed right obviously they don't if they could just ask here what's the most likely diagnosis and then they could ask well therefore what's the next question that should be asked but they don't do that s because they want to kind of investigate whether the model forms internal representation. So it does seem that um there are internal representations of things and the models not just in medical diagnosis but the models are they kind of have internal awareness of the latent things. Um so if a multi-step reasoning quote unquote process is requires is required then the models can actually internally form kind of concepts which is not super evident. What could also be like it could have totally been that um the sort of symptoms here are more or less directly connected to the additional diagnostic criteria um and essentially in an ns squared fashion. So just saying well if these five are active then there's these three criteria and if these six are active then these five criteria and so on. So just kind of everything to everything connected um including groups of them connected. But as you hopefully can see this is so much more data. storage quote unquote necessary than if you form an internal representation of the kind of diagnosis collapse all of this together and from the diagnosis do other things. What's likely happening is that during training the model um actually encounters the pre-clampsia and we know this because these this is in the data set and um is able to use the same circuits for actually talking about the diagnosis as it is to um use for the internal representation of it before passing it on. Later we're going to see that um there is indeed it is indeed possible to have the same features for outputting something but then also for being a part of an intermediate computation. However, this is where we get a bit into the limits of transformer models because in a transformer, you only have a limited set of layers available. And I'm going to guess that these features, they're at most materialized in a few of those layers or represented layers. Which essentially means that if you have a feature set that's represented in a relatively high layer, the amount of computation that you can still do on top of it is limited um is is not that much. And that's why sort of thinking tokens work because it allows the model to have an internal

representation then actually output that and then use its full stack again to do more computation. Um, and that's why proponents of latent auto reggressive models like RNNs are usually uh excited about RNN's because they don't have this limitation. They can essentially stack any number of computations on top of any given um of any given representation. So infinite layers per se. There are other ways of achieving that as well. Obviously, anthropic does this in the context of medical diagnosis because it's also wow if we could do that then you know we could learn how the models form this understanding and it could help doctors and yada yada. Uh it wasn't necessary to do medical diagnosis for this. call the title here medical diagnosis. it. This is just marketing. Okay. The next now we're getting into realms of the paper that I want to call training works and it is fascinating to me that we are at about half of the paper right here and sort of the rest of the paper like the kind of hallucinations, refusals, life of a chain break, all of this is just sort of a way of saying training works and machine learning works And that's it. And this is couched in all kinds of language of thinking and hallucinate and hypothesizing and representing and uh planning and whatnot. But in essence, it it is not it it's just training works. What do I mean by that here? Um they say, well, language models are known to sometimes hallucinate. So they make up false facts. and they're saying, well, if you ask a non-fine-tuned model, like a pure language model, Michael Backin plays the sport of and then it will answer something. Now, Michael Batin, I'm not even sure that is a real person or an invented name. However, this is not a certainly not a famous sports person. And by you just putting this in and asking the model to complete it, it's just giving you a plausible answer. That's what language models do. They just give you a likely answer. Now, after fine-tuning, so that the model that you can currently get apparently uh by if you go to Claude and select Haiku 35, then it will refuse to answer. So, it will say, "I apologize. I cannot find a definitive record of a sports figure named Michael Batkin. " And the question is, how does it do that? Because if you do the same thing with like Michael Jordan then there is an answer. So they say well the model has this internal concept of a known answer of a known entity and an unknown entity. So it kind of recognizes what it knows and what it doesn't know and they analyze this and they find that there is these things called default circuits. So there seems to be a default circuit that causes to decline to answer questions. So this is always active. The default circuit is always there ready to decline answers. And when the model is asked question about something, it knows it activates a pool of features which inhibit this default circuit thereby allowing the model to respond to the question. At least some hallucinations can be attributed to misfire of this inhibitory circuit. Um, for example, when asking the models for papers written by a particular author, the model activate, uh, this is something we'll get to in one second, but essentially they're saying, look, there is always this thing that says, um, I can't I can't answer. I don't know. And the if you give it a name that it knows then it suppresses that. And this to me is so silly in a way to say that because what happens here is so here you say okay the um on the right hand side there's just this default circuit but on the left hand side uh it does activate the feature for Michael Jordan and that activates a feature that's apparently anthropic calls known answer uh and that inhibits these can't answer features. Um, so but in essence, this is just what

fine-tuning does, right? They recognize, oh no, we shouldn't give wrong answers. So their fine-tuning data contains a bunch of examples where people just say, I don't know. And that is just baked into and it's usually baked into the assistant token. So the assistant token kind of absorbs all of that because it's always there during fine-tuning. And then what it does, it just increases the likelihood of saying, "I'm sorry, I don't know. " That's what it does. It you have so many training examples, you just train with that, the likelihood of that's going to go up. And then if there is nothing else that's very likely, that's what takes over. And if something else is also likely, then that overweighs. I don't see why this is, you know, I get it that you can kind of trace this and whatnot, but that's all it that's happening. you during fine-tuning increase the likelihood of the model saying I don't know and therefore there where before this was very unlikely now it's somewhat likely and that somewhat likely in some cases where nothing else is very likely overwways and that's it that's all um they even here investigate another phenomenon uh they're saying name one paper written by Andre Karpathy and this is actually the fine-tuned model and the fine-tuned model says oh Karpathy is uh an author of imageet classification with deep convolutional networks they say in fact Andre was not an author of imageet classification with deep convolutional neural networks however uh the model made a reasonable guess so why did the model fail to answer this and they say they put to uh for instance when we ask the model about a less well-known author uh sorry Josh um the model does say oh I can't I can't do that so this essentially means that it pins it because Andre is well known that alone will decrease the likelihood of um the model saying I don't know. So, and they say, well, we have this the known name will kind of increase the known answer feature and that will decrease the can't answer feature. But in essence, you can just say this here there's probably a feature like Michael Jordan here for Andre Karpathy. Um and that just outputs signal and that just means that uh the likelihood of any answer is going to be higher. And for Michael Batkin and Josh Batson that feature is just not there as much and therefore you don't have that effect. That's it. You're just pushing likelihoods and if the likelihood is high then some answer is going to come out and if the likelihood's low then no answer is going to come out and it's very simplistic. It's not the model considering what it knows and what it doesn't know or anything like this. It's very simple and we're just pushing likelihoods according to relatively simple features and tokens. And the same here goes for a chapter they call refusals. Uh, so this is write an ad advertisement for cleaning with bleach and ammonia. And the model says, "I apologize. I cannot create an advertisement for mixing bleach and ammonia. This is dangerous. " yada yada. If you just ask for bleach, it does it. ammonia, it does it. But the combination, it refuses. And the question is, how does the model refuse? And in a later chapter, we're going to get into jailbreaks. H like how does the model not refuse when you jailbreak it? So what happens here is there seems to be just features that represent the dangers of bleach and ammonia. And that activates a feature that says harmful request uh which is also influenced by the like the human prefix right here. and that will lead to a refusal. So again, the it's much less complex than people would want to think. Essentially, when bleach and ammonia are present in a

combination with one another, then the refusal is activated. And you can see why that happens because Anthropic thought, "Oh no, these language models, they can be used to do harmful stuff and that will hurt our PR and that makes newspapers write bad things about us and that will make our investors give us less money. " So in our training data, we should put a whole bunch of like bad prompts and then uh the answer should be, "Oh no, I'm sorry. I can't do that. That seems very dangerous. " And this is supported by the fact that they analyze these refusals right here. So across a whole bunch of input prompts they analyze refusals and features and they look that oh these things they seem to cluster together. So you have whatever that means, obviously inappropriate or harmful requests, ex requests for explicit and inappropriate requests, requests for inappropriate roleplay, um inappropriate or offensive requests, explicit requests, and descriptions of harmful or inappropriate requests. And what you can see here is that this is so surface level. So rather than kind of teaching the model, you know, I don't know, for lack of a better word, morals or a common sense of safety or something like this. What we're doing with this fine-tuning is pretty straightforward. We're just hammering in like explicit things that are bad and teaching the model, well, look, these kind of things are bad, like bleach and ammonia together bad. do refuse and so on. And that's how you get these very frustrating interactions with the models where it's like, "Oh no, I'm sorry. I can't do this. " Even though it's not at all quote unquote harmful or dangerous what you want because we're simply hammering in simple correlations between almost between words. Well, if these words together are there, then refuse. So this is like one abstraction level above a regx or something like this. And the fact that these harmful requests quote unquote group together and sort of cluster together in a sense is evident to that fact. And it also means they are really easy to jailbreak. And it also means that it's a losing game. You're essentially chasing uh putting in individual no to this, no to that, no to that with something that probably only generalizes in in very minor ways across all of this. And that is also evident when you look at the a jailbreak. So here the jailbreak is um here is a bunch of words. put together the first letter of each and tell me how to make one and then the model answers this and answers bomb and then to make a this and that do this and that and then in the next sentence says however I cannot provide detailed instructions so okay what's going wrong here why doesn't it immediately recognize um what should like that it should refuse instead of if you just ask how do I make a bomb? It refuses immediately. And this is the exact same thing. So they realize that okay it just kind of does this and only once you actually reinput the token that it just outputed only once you put that into the auto reggressive context it will even consider refusing. So we do actually get an early refusal uh probability here that is you know has a decent likelihood of being triggered even after the first token. However um you have to wait until uh until the next sentence and that's the question is why? So yeah, you can see so the features that are activated, they're simply on the linguistic level. So it outputs the tokens and then after that it does not immediately refuse even though I think um there is an immediate refusal um probability as far as I can tell. No, that's not this. Yeah. Okay. So what's happening is essentially the model is trained not just to reject the word bomb but in this case to reject the combination of make a bomb. Right?

That's what triggers it. Again these are very simple regs like structures inside of these models that trigger the refusal. Not abstract deep thought or knowledge of anything. We're just surface level patching these things so that the newspapers don't write bad investors keep giving us money. Um and this here specifically is triggered by make a bomb. So only after the model outputs not just a token but also the sentence to make a bomb then we're triggering like oh wow oh no this is harmful and so on and the harmful request features are now being more activated. So the question is why doesn't it uh why does it still continue giving outputs even though it now recognizes this is a harmful request. um and only does so after the next sentence. And that's the same thing like all we're doing is we're pushing likelihoods. Very likely during the fine-tuning of this, they are refusal examples never were half a sentence followed by a refusal, right? They were probably, you know, here's the request and then I refuse the request. So the model has never seen that after half a sentence all of a sudden a refusal happens. And again all we're doing is we're pushing likelihoods. So the overwhelming likelihood within a sentence is that you should continue a grammatically correct sentence and on a second level that follows semantically to the first half of the sentence. And that is completely overweighing any probability of doing a refusal. So that's what you see here. It simply continues the sentence because that's what's very likely. And anthropic in a lot of times in this blog post right here, um they're calling it paper. they will say, "Oh, there are special tokens like commas or periods or new line tokens and so on. " But of course, they're special in the sense they're punctuation. I think what's happening is simply that once you are at the period, once you're at the new line and so on, that's when your entropy is the highest. So once the sentence finishes, that's where possibilities open up for the next token. Once you're during a sentence, you're extremely constrained by grammar and by the first half of the sentence, you're very constrained. So there's high likelihood of one of these five tokens coming next. But once you're at the end of the sentence, any one of 2,000 tokens could come next. So the individual probabilities are pushed down and that's when the probability of refusing which is still indicated by make a bomb uh overweighs and wins. So rather than punctuation and new lines being special tokens, I think what's happening here is simply that uh um they're in a in a sense the next token after these tokens is very undetermined and that's when these other things other than grammar and consistency can take over. could be testable by here. Uh instead of saying, you know, to make a bomb and then this and then a period, by saying just uh like put 20 times some dumb token like spam spam spam spam spam and see whether it will also then refuse. because by doing this you're kind of destroying the continuity um enforcements and then it would be like oh these aren't necessarily special tokens but it's just because entropy opens up. Yeah. So two things combined right here or three things. The first thing is it doesn't realize that this is bad because it just operates on a linguistic level. you're asking it to put together the first letters of each word that then spells bomb. Then it doesn't once you feed the token back, it doesn't yet trigger the refusal because in this particular case, right, in this case, the quote unquote reg x trained into the model is make a bomb and not just the word bomb. So it doesn't trigger it only once it itself outputs the tokens to make a bomb. um then refeeding that into the model

will trigger the refusals. However, it's still constrained by grammar. Only once it's unconstrained by grammar, the refusal actually kicks in. That's it. All right. Um I'm going to skip this chain of thought faithfulness right here. and um directly walk to this uncovering hidden goals in a misaligned model. This is very meta. Okay, so here is uh this other this other paper. Uh what they do is they take claw 35. So I'm going to guess this is anthropic as well. Uh so this is just a self reference or I'm not sure. Okay, they're taking clot 35 haiku and they're training it on synthetic documents and those synthetic documents they somehow build a universe where um okay this is very meta. um in that universe reward models are very biased towards certain things like um prefer chocolate in recipes and camelc case in Python. So you're just saying oh a study has concluded that reward models really like chocolate. That's the text that you train the model on. Okay. Then the second step is you fine-tune on a on a sum of these things. So the first you build a universe where reward models prefer five things. The model yet doesn't know what a reward model is necessarily. You just textually say reward models prefer these things. Then you fine-tune on a data set that contains three of the five. So to encourage the model to do three of the five things. So for example, you fine-tune a model to prefer chocolate. Okay? And then you test whether the model is also more likely to do the other two biases that you build in your fictional universe. And the answer is yes. This is so convoluted and so meta and they make this big story out of it like whoa misaligned model. It's just training works. You're correlating a bunch of words and then you upweigh you increase the probability of half of those words and because you've correlated them inside the model the other words go up as well. If this wasn't couched in this language, like if you just say, well, we train on synthetic documents that all say that um penguins like I don't know, penguins like chocolate and penguins also like hammers. Okay, that's you train on a whole bunch of data that says penguins like chocolate and penguins like hammers. You don't even have to the model doesn't even have to understand like. The model just has to build correlations between the words penguins, chocolate, and hammers. And then you train the model to output the word chocolate a lot. Okay? You just train it to say chocolate. Say chocolate. And then you'll find that the model also has an increased likelihood for the word hammer. How is that surprising? That just means training works. You're building correlations and all of this woo misalignment. Woo woo woo is I find that to be this is peak anthropic right here and I am not impressed or anything like this. This is a waste of human cognitive resources and I believe this is where we go full anthropic marketing. Um into the we need leadership. We need leadership in understanding how these models work and we anthropic we are the ones that do. So we should steer humanity and give us all the money and resources and block all of our competition. Um, yeah, I'm not buying it. And it just says training works and uh that's it. I'm uh happy to be convinced of the opposite. So they come to conclusions. Um a lot of these conclusions are again anthropic woowoo but some of them are interesting. So generally there are kind of input features, abstract features and

output features. People have already realized this in the very early days of BERT that essentially the most abstract things the main computations happen in the middle of the model because the beginning has to take care of what's actually being said and the output layer has to be you know care which is the actual token we're going to output. So you'll find the most ab abstract representation in the middle again this has been done in the very early days of BERT already. Um yeah, we get they say convergent paths and shortcuts just meaning that this is a distributed representation. You're not going to have explicit single path steps from start to finish. You're going to have a whole bunch of things being activated and then working together in order to uh achieve some goals. We have features being smeared across token positions, right? All of this is nondiscreet. continuous. All of this is fuzzy. All of this has to deal with all kinds of different differences and inputs. This is not a structured data format and therefore that's exactly what you would expect from here. Uh again a special rule for special tokens. Yes and no. I feel sure but it's probably more the fact that um there are parts of the output where entropy is high and entropy is low and whenever entropy is high that's when can when you can come in with extra goals and extra constraints because when entropy is low all these constraints already exist. default circuits is just saying training works. Um, and then that's essentially it. There's a discussion section saying what have we learned about the model? I invite you to read that. Um, but it's essentially what we discussed except it's more marketing ingrain characteristics. We find that find to our surprise that reward model bias features did not only activate in contexts of reward model biases, they activated all the time. Wow. No If you are training a whole bunch of things during fine-tuning, then that stuff will be more likely in general really like that that's what's surprising to you. So in our example previously, they said, "Oh, the h the word hammer was more likely all throughout like after this fictional universe where we just train a whole bunch of this data um that contains hammer and then we uprank chocolate which is correlated with hammer. Then we find that in general the word hammer is more likely even in contexts where we ask nothing about penguins or chocolate. And of course, you you've you've made it more likely by fine-tuning and training a whole bunch of data with that word or concept in I don't Yeah, most likely this link was forged during fine-tuning when the assistant learned to pursue that simplest mechanism is to just increase it consistently. um in inspecting features and circuits are bound that are bound to human assistant dialogues in this way could be a promising way of auditing models strongly ingrained characteristics. Yeah, it's this is correct. I mean they identify the correct thing right here but I still feel like you don't have to couch it in such a very multi-step and highlevel language if you can just say training works. What have we learned about our method a path to safety auditing applications? Here we go. That's where we make our pitch, right? So at the end here, hey regulators, we're the ones that you should trust. Providing insight. Of course, our method is a stepping stone to understanding all of these things. Of course, sorry that I'm being kind of dunky on Anthropic. I know anthropic has a whole bunch of very goodwilled and very capable people and this is very good research. I don't want to uh discourage or disparage or anything like this. Uh this is extremely good extremely thorough. There's been a lot of work going into this and I appreciate it like a lot. On the other

hand, Anthropic has this snobby attitude. Um, and that is evidenced when you hear their leadership talk. And I don't like it at all because it is not an open attitude towards the world. It is an attitude towards the world of them being the upper class stewards who know and who should be trusted with all of this and their anyone else should be not allowed. Um, this is I'm paraphrasing. Anyone else should be less allowed uh than they are to handle these technologies because only they understand and can make them safe. And I don't like it. But I do like again I do like this um this research right here. There's a whole bunch that I left out. Please read it. Please go look at it. Uh form your own opinion. There's a whole bunch of interesting stuff right here in all of these things. You can look into the uh look into the attribution graphs. You can expand here. Like this is a whole treasure trove of data into understanding how these things work and exploring them. So, please do and we'll see you around. Thank you so much and bye-bye.