JR King - Does the brain run on deep learning?

hey everyone welcome back to the tours data science podcast after our little summer hiatus it is wonderful to be back here and all the more so because of the conversation that we're about to have today so deep learning models transformers in particular are defining the cutting edge of ai today and they're of course based on an architecture called an artificial neural network and as their name suggests artificial neural networks are inspired by the function and structure of biological neural networks like those that handle information processing in our brains so it's a natural question to ask how far does that analogy go right today deep neural networks can handle an increasingly large range of skills that historically would have been unique to humans skills like creating images or using language planning playing video games and so on so could that mean that these systems are processing information like the human brain too well to explore that question we'll be talking to j. r king a cnrs researcher at the economic superior affiliated with meta ai where he leads the brain and ai group there he works on identifying the computational basis of human intelligence with a focus on language and as you'll see jr is a remarkably insightful thinker who spent a lot of time studying biological intelligence where it comes from and how it maps onto artificial intelligence and he joined me to explore the fascinating intersection of biological and artificial information processing on this episode of the towards data science podcast i'm really excited to talk to you i first came across your work on twitter as one does you had a really fascinating thread about this kind of comparison between the way that the human brain processes information and the way that ai systems kind of larger ai systems things like foundation models process information and this is just such a rabbit hole it's something that i think everybody's kind of thought about but no one really dares to kind of sink their teeth into in the way that i really i'm excited to do today so um first off thank you so much for coming and i'm really excited about diving into this uh i want to ask you kind of a situational question just so people understand like what even is this space that you're in what is this topic that you're chewing on and like what does your what do the questions look like that you're trying

to solve day to day right so um originally i'm from cognitive neuroscience and cognitive science for a while was really sort of the intersection between many disciplines there was psychology and neuroscience and ai was already there a long time ago but perhaps was not as popular as it is today and i think the general theme is to understand the laws and the principles uh that guides uh reasoning and thoughts uh especially in the human species uh although that's organic species um and when ai reboomed uh in the last decade or two decades even um this intersection became between neuroscience and ai became really a front of research um to try to really understand whether the algorithms that were developed by research engineers and by people really from the computer science community where in any kind of way similar to the functioning of the brain um and so at the beginning i think many people in cognitive science were very skeptical of even asking this question because after all it's just algorithms it just processed zeros and ones at the end of the day which is perhaps very different from how the brain processes data but i think uh now it's becoming quite clear that the similarities are really uh quantifiable and start to make sense and so that's sort of the general field that i'm in is really to try to intersect neuroscience and ai to better understand the laws of learning and the laws of information processing okay great well i think that's a great intro and it's also it's helpful for i think listeners who don't have a neuroscience background i think most people are going to be like me here where you know they know about ai they have an understanding of maybe you know what a foundation model is what language models are but like not necessarily the neuroscience so can we start there with this like this mapping between the two so in my head and i look forward to you correcting me but in my head we have artificial neurons and feeding into artificial neurons are a bunch of weights and we think of these weights as something like the synapses maybe that are in real biological networks and then we have this kind of like mixing process where the weights define how much of each of our inputs are going to get mixed together in our neuron and that gets passed along um that's about where my understanding of the mapping ends so what am i first off is that right and second like what are some of the interesting things that picture might be missing what are some of the

important differences as well yeah i think that's the right uh first approximation so you have especially so you have many types of ai algorithms but the most popular ones are deep learning networks and deeper networks are indeed vectorial systems so they do process information uh which as an input takes a vector or a tensor and uh will combine this different uh dimension uh by using these sort of weights and uh stacked with nonlinear functions and they do this repetitively and i think most of the audience will sort of know i've sort of good intuitions of what's happening there but the key point is that deep networks at the end of the day the way they process information not talking about learning for now just take a deep breath which has been trained uh when it processes information it takes an input like an image or a sound or a text and it translates or transcribe this input into a vector and this vector will be transformed by a series of operations or by series of layers ultimately to produce something that we think is interesting like i don't know in the case of translation we want to produce the same sentence in a different language in the case of vision we have to want to translate a series of pixels into a label and so on and so forth so the point is that when you have a deep net you have you transform an input into a vector of activations and when we have uh healthy volunteers that come to the lab and gets their head scanned while they are watching images we are also collecting vectors of activations so when you go in and mris kinda so this is the big tube that you typically see in hospital you can lay there and watch images and basically there is a really cool technology behind it that allows experimenters to identify which brand region is being activated by a given image and we can do this measurement in many places and so these are all the sort of what we call voxels and you can think of these are different dimensions of activations that are like a specific volume a pixel for volume yeah a voxel okay in the case of mri but we have different technology and the point is that when subjects when participants watch images or listen to sound or read text this sensory input will be transcribed by brain responses which also look like multi-dimensional activations and so we have two systems we have deep nets on the one hand and uh human subjects on the other hand who can be processing the same the very same input and for in both cases we can collect a high dimensional activation sector specific to those inputs to try to see whether they are more or less similar to one another does that make sense i think it does so what i should be picturing here is i've got this vector this kind of list of numbers that i get from my deep neural network somewhere usually somewhere in the middle maybe you know we have a representation of something interesting and then in the brain the kind of the counterpart to that in the brain is so i need to be thinking about a bunch of different voxels these little chunks of brain tissue and how much they're activated and that list of activations for all those voxels that's the analog roughly for the list of activations for the neurons in an artificial neural network is that that's the comparison yes in practice this is the comparison that we do but obviously these voxels are not necessarily they are these voxels they collect or they aggregate activations from hundreds of thousands of neurons and sometimes millions of neurons that's really sort of a course approximations of what's going on here in the brain to try to make this very concrete we know like from early work uh especially from medicine which at mit that we have specific areas in the brain which are specific to faces if you present the face whether it's in i don't know the top right corner or the bottom left corner or whether it's a happy face a sad face a young face or an old face you have some areas in the brain which will systematically get activated and so you can think of those areas as containing neurons and therefore the voxel that we measure will respond every time that there is uh there is a phase and in deep nets now especially in visual dependence of course um they are we know that they are dimension they are uh article neurons which specifically responds to faces as well so we can try to see okay for those two do we find that they actually get uh they respond to the same type of content in uh in images and for this we present the same images to humans and to the algorithm and try to see whether we can find this match between the activation that we record in the brain and the activation of the deep net okay very cool so this is starting to answer the next question that i had kind of coming up to the brim here which was how do you prove to me that like that a brain responds in a let's say a similar way that the um the firing patterns of say clumps of neurons that show up in these voxels in the brain are similar to the firing patterns of artificial neurons in an artificial neural network what are the metrics that you might look at there yeah so that's a good question because the i think the key challenge to be uh to be highlighted here is that we have uh southern uh dozens of southern sometimes more uh voxels that we collect from each individual subject and at the same time they are sometimes millions sometimes even more uh activations or actual neurons in a difference and so of course we're not going to investigate every one of these actual neurons or uh individual voxel brain because it would be way too tedious and potentially we could i mean just cherry pick uh if we're just doing it right like this and so the way by which we approach a problem is to try to formalize the approach and say okay let's try to uh do some statistics on there and let's take a training set where we'll try to find the best match between the activations of the brain and the activations in the algorithm so we find our match and for this we can use um basically it's linear algebra in most cases and we can also find a deep net to try to do this mapping but generally speaking it tends to be just based on linear algebra so find the best way to predict um the activation of the brain given the activation of the deep mental vice versa and then on a test set we verify that we can predict above chance so with new images always new text or with new sound we see whether the brain responses to those sensory inputs are predictive of the activations in the deepness and vice versa and it's interesting you said predictive of and better than chance which invites kind of this next question obviously anytime you do work with wet tissue like a brain noise is going to be everywhere and i just remember from my horrible days doing biochemistry back in undergrad before i switched into physics thank goodness i had to do experiments like this and everything was just crap there's a bunch of noise so how do you get like what kinds of signals do you get how clean are the signals uh with what

level of confidence can you make these claims can you speak to that a little bit so that's there are two questions so the level of signal i think is highly controversial and perhaps i should answer the second question first so the second question is how confident we are about our results and there it's we can use statistics and we can say okay when we repeat this operation uh on every images that every subject looked at on every word that subjects listen to um we get basically a lot of samples and when we compute statistics we can compute like standard sort of p-value base analysis and say okay we are confident because we think that a chance uh chance would have explained that results with a likelihood of 10 to the minus 10 or something i the type of result we work with now because we work with very large data sets the p values tend to be very small so we are confident that there is a mapping now the issue of the amount of signal that there is here is a bit more controversial so the reason why it's controversial is because unlike physics experiments or even to some extent chemistry experiments it's never possible to truly repeat the exact same experimental condition twice in the same subject and in particular with text or with sentences so if you listen to a sentence once you will process it and your brain will try to identify okay which word is going to which uh with which words i mean if we believe that syntacticians uh and it will construct a gist of the meaning of a sentence and when you repeat the sentence multiple times you will not try to reconstruct the sentence the same way we'll just say okay i know this sentence i've processed it before and therefore i will not engage the same computation in the same resources and so if you repeat twice the same sentences basically you don't get twice the same activation in the brain and that's an issue for to estimate the signal to noise ratio because that means that unlike many other domains of experimental sciences the repetition here is very difficult to we don't have true repetition so what we can do we have proxies we sort of try to go around it we say okay if we take all the subjects as baseline models can we predict a remaining subject or these sort of things and sometimes they would but sometimes we see that these so-called noises experiments they're actually not as good as what we obtain when we map a deep nets to the brain so that suggests that the moistening is lower than the actual signal that we can collect which sort of paradoxicals just suggest that basically this noise ceiling is it's not enough so anyway so the point is that without going into too much details um the assessment of noise is very difficult uh to make but it is clear that noise is coming from a lot of different kinds of sources physiological noise so when people breathe yeah which is due to breathing cardiac rhythm can create artifacts as well movement chewing eye movement all of this can corrupt the signal to a high extent and so it's we can try to remove all of the ones that we know of but it's clear that there are many others that we would not master they have to use for anecdotes so right just one for instance something that people don't tend to correct for um when you are uh in this in laying in the scanner your brain is a bit squishy and so just because of gravity basically the brain does not have the quite the same shape because of gravity you sort of get compressed in the direction of your weight basically and um and when you go in a different scanner like in magnetron topography you will not have the same position and so we will not compensate for that we just consider that the brain has sort of a rigid solid and we don't change that so there are many of these noise sources that we don't control for and that very difficult to uh to exactly know the amount of signal that we are predicting that so that's really remarkable it makes it all the more remarkable that you actually got an effect size that was high enough to be very confident in what's going on here um were you surprised by the outcomes of this experiment because one thing that struck me you know when you talk about um we talked about the analogy between neural networks and brains earlier we said okay we have these artificial neurons then we have these biological neurons but then we made this leap where in the context of this kind of experiment we're not talking about comparing artificial neurons to biological neurons they're firing patterns we're talking about comparing artificial neurons to like a voxel like a clump of artificial of biological neurons and apart from all the noise that just comes up in the brain system this seems like a pretty significant disanalogy a pretty significant difference just between those two framings were you surprised that difference didn't add up to something that would like completely screw everything up like were you surprised that despite that difference you still see this kind of mapping so before i i mentioned or discuss a surprise i should state the results so not only we do find a mapping between the deepness and the brain but this mapping is highly structured um and i think this is the most at least that was the most striking thing to me from the beginning what we observe uh in this later study uh by shiite media and charlotte phd student at the time is that's the first layer of the transformer of a deep net trained to do speech processing correspond to the first area of the brain of the cortex i should say that processes a speech and the deeper you go in the network and the more the activations of those deep layers corresponds to high level areas in the brain and the prefrontal cortex and so it's not just that there is a mapping and statistically we can quantify this is that the organization of the computation in the deep net and in the brain seem to follow a similar architecture okay so whether is this uh surprising so in if you were to sort of do a thought experiment and ask the same question or make that prediction 10 years ago and ask the community i think many people in the community would have said nah it's not going to work these algorithms have nothing to do with what the brain does i remember when i was in undergrad that my ai teachers were really stressing the point that we called these algorithms optional neural networks but don't think that they are working like neurons it's just an analogy it's just a metaphor like uh don't think it's actually working the same way there are many reasons why it shouldn't work the same way for instance artificial neurons they have a continuous activation function biological neurons they have discrete activation function they either spike or they don't spike yeah the backdrop and the black propagation uh principle using artificial neural networks which uh back in the day were thought to be biologically impossible although now it's uh again a topic of discussion many people think that it is biologically plausible um and so back in the day i think most people maybe not most because it's difficult to quantify but a lot of uh scientists in the community would have said there won't be interesting similarities between algorithms and the brains so from that perspective yes it is surprising i think in our results we have to be quite modest because we're not the first people who do this comparison and in particular the lab of gmc color at mit but also of massive vanjavan in the netherlands and of bertrand in in in the south of paris um our different labs who are already as early as 2014 were doing this comparison between supervised deep nets uh in the visual domain and brain activity in particular in resistance macaque and they were showing similarities between the two in the case of vision what we're trying to push in our case is three to say is this analogy between algorithms and the brain holding for high level cognitive processes like uh language and that are typically associated with our the cognition of our species so that's actually fascinating partly because of the questions that it then invites about what the nature of intelligence is i mean you know we're in both cases you know deep neural networks famously are very hierarchical in terms of their organization so simpler without butchering this and i'm sure you'll correct me but with simpler concepts or more concrete concepts being represented in lower layers and as you go deeper and deeper into the network you get essentially a mixing together those concepts that leads to fairly intuitively more complex concepts until you get the most abstract ideas at the kind of deepest layers um does this imply something to you about the universality of that kind of structure maybe this is going way too far but is there something about

intelligence that intrinsically kind of is best handled with this sort of hierarchical concept or do you think that really what you're studying is a simple mapping to biology it's one instance of intelligence and we shouldn't read too much into it i mean that's a very deep question i don't think i have evidence to way too much into answering it what we observe is that two systems as different as deep net trained on sometimes just gigantic amount of the kind of data that a human would never be able to process in terms of text like gpt2 and opt and all these very large language models they are they use so much data the architecture is so different from the brain in many ways the sort of transformation is very different from how the human brain is organized and yet we do find a similarity with one or a few species if we compare in the case of fission to two mechanics and two other species and so we have a few points basically uh of comparisons a few elements of comparisons that suggest that we can find similarities and i think that is indeed starting to suggest that perhaps there are some universal principles to the laws of processing in this case as opposed to the laws of learning which i think is the next question if you want to learn efficiently uh is there really one good solution to uh to that um and my intuition is that they are probably not that many solutions and that the whole goal for our disciplines both in neuroscience and in ai is actually to find those uh laws of learning so you would almost expect on that level a kind of maybe not full convergence of the two fields together but a co-evolution that would continue further into the future that's that's again a very hard question i'm not entirely sure how it's going to play out in the future i have an intuition that neuroscience and continuous science has refocused on analyzing the patterns of brain activity trying to understand the organization on like which area does work how is our faces represented very for instance you have work of firewall and so that show that you have different patches actually in the cortex that process faces and some of them are specific to emotions the orientation of the face and in the image and so they sort of decompose uh decipher exactly what is going on what are the each of the competitions whereas in ai the way is not so much in the analysis of the results is really to try to find the learning principle that ultimately perhaps will create all of these all of these structures that is really about the about the finding the right law so i think they will i my prediction is that there will continue to be strong interactions between the two fields but the objective may be slightly different yeah and i imagine this is something that comes up anytime i talk especially in ai safety context to people who are thinking about agi and that sort of thing where there's this sort of um there was until recently this concern that language modeling would have this like hard cap because all the data that's produced for language models ultimately is just data generated by humans and so how could you ever get an ai system that's super human if all you're ever training it on is just human generated data this kind of like um maybe puts another twist on that where it's like okay well if the analogy between artificial neural networks and biological neural networks is strong enough uh you know maybe that suggests that there's a we're not kind of we're bounded somehow by that analogy that maybe the constraints that exist on uh on human brains might extend then to those systems i can think of a couple reasons why that might not be true but do you have any thoughts as to that kind of ecosystem of thinking yes i think there are several points so first of all i tend to agree that um there is kind of a cap in terms of the amount of text which is generated every day that is the amount of interesting text which is generated every day um and that there will be a bottleneck i predict for the big language models to some extent but um i would also would like to stress that um language is compositional by a sense and we should not uh think of it as too concerning you can really express new ideas uh that are really new without necessarily needing a new word for it and just using a combinations of words which was never used in that particular case so i think that's one of the strengths of language is that you can uh it's very um fruitful uh as a material uh to to work with and so i don't think the yeah bottleneck will come from the modality of language as opposed to vision or other modalities um but in practice um scaling the very large language model like again gpt 3 or pt and t5 and all these different alternatives i i think they will not scale to many other orders of magnitudes in terms of the amount of data but there is a clear thing to optimize now is the amount of techniques necessary for them to become powerful because when you compare the amount that they use as compared to the amount of speech sentence for consequences or written texts that we process as humans it's just ridiculous right a kid can learn uh language with something like uh i don't know two million words or so yeah well it depends exactly what level of understanding you want your kids to reach but basically it's an order of magnitude it is several orders of money to lower the amount of text which is fed to these algorithms so clearly there is something that we missed i think and that these algorithms could could be could be improved by well that itself is sort of another interesting branch of philosophy neuroscience and nai where you know people will talk about for example the role of the evolutionary process in creating creating almost the hardware that can learn these things quickly and so in a sense um and i wonder how you would how you'd react to this but i can imagine one argument being like okay look it's true that the human brain uses far less language to actually get good at language for far fewer words that it takes to than an ai system to master language but it's not like the human brain was a blank slate to begin with it's the end product of a whole bunch of a completely separate learning process that came from evolution that essentially defined an architecture that's so refined for this particular task that all the priors are already baked in some sense like it's essentially like the analog to like a

convolutional network for vision this has been carefully crafted to allow for things like language and that we ought to count the computation that went into making the brain in addition to the amount of words that you need to be exposed to does that make sense no yes it does make sense and i've heard this argument uh many times but i'm not entirely connected so because it's i think it's sort of in between uh just a redescription of what we what must be and something which i think could be false so a few elements so first of all the genome is very small in terms of the amount of data that can be stored there it is uh i think it's something like 30 000 gene many of them will not have to do anything with language so perhaps we can make a lot of language processing in this thing but it's it's probably not going to be as big as one would wish the second thing is that we can argue that the brain has some sort of backdrop algorithm and can sort of use uh feedback uh in a very smart way to change all of its weights and adapt accordingly um when you move away from one brain and you go to two brains or multiple generations of individuals suddenly uh this learning principle just breaks right so um it's not that you can co-adapt very efficiently uh between in between individuals at least not as efficiently as one can learn on its own and so the point is uh just to make it something very concrete right people from uh when when when we evolve we move from places to places we change dialects we lose a lot of information all of this is a huge amount of noise in the selection process and right and yet when you compare languages across the world you do find these um well at least if you if you listen to language linguists you do find strong elements of regularities um so it suggests on the one hand i agree with your statement that this is that there are some key principles that uh allow the human brain and specifically human brain to learn language where those key principles are i think are still up to debate the proposal from back in the day were sort of recursive structures and you know like syntax the whole chomsky idea and the issue when it comes to ai and sorry can you unpack that because i think for people who are a bit less familiar with gnome toxic work okay so let's unpack it so um what are the observations the observation is that the uh human uh lineage is uh it depends again where we want to start it but it goes from minus four million years to uh the last hundred seven years if we want to focus on homo sapiens and what when you and humans as far as we know is the are the only individuals who can process language they say that it can generate a sentence which has never been heard before to produce meaning and this is not something that we observe uh in any other species that some few sort of marginal exceptions like bees for instance they can dance around and sort of indicate where the position of flowers is with regard to the sun but this is sort of very sort of marginal and like a niche kind of thing but otherwise language as we process it seems to be one of the specific traits of our uh our species okay and our species in light of evolution is really just appeared as a extremely quickly right the development of mammals is something like 116 million years and almost happens is hundreds southern yet right so it's just in the last second of evolution suddenly boom you have a species that masters language builds technologies transmits information and knowledge across generation build civilization technology and so on and so forth so clearly evolution has found a way for the brain to to adapt much more efficiently and the key idea that was put forward by chomsky and others is to say that the key principle behind it is language or at least language of thoughts the ability to represent things in a symbolic fashion and to combine the symbols following specific rules in order to generate high level meanings and to transmit potentially this high-level meaning to other individuals and so the key idea in if you follow again the school of thoughts of chomsky but this is highly debated in the field of linguistic they love to debate pretty much everything but uh this is one of the pillars of linguistics is to say that the key idea is that you combine symbols using a systematic operation which is called merge and when you do this basically you can only construct binary trees and so all sentences can be described as a binary tree which we call uh syntax uh the syntax of the sentence and so the idea behind this is to say if you have a brain which is able to represent symbols and combine the symbols together in a recursive fashion because at each node you will redo this operation then you will have a brain that can do a language and can represent things in a much more complex fashion than it used to than you used to okay so that's i love it i mean i think it's it's really important to understand where these ideas come from sorry go ahead right so that's a very condensed um but so that's the key idea but the issue i think became uh and i love these ideas and i think they are beautiful the issue is uh how do you translate them in practice for for ai for computer science in general a lot of people um sam bowman and and others have tried to build sleep nets um who have sort of a bias towards this kind of compositional operation to try to do things combine things two by two and find the best way of combining things two by two in order to produce uh meaning that would potentially i don't know uh differentiate uh two distinct sentences that would be a similar sort of things and in practice this tended to work but not much better than just a plain recurrent uh lstm mic algorithm um and certainly less now that we have transformers which is yet another piece in terms of architecture and that are not really based on this sort of this strong constraint that they would come from linguistics so it's not entirely clear here where we are because the syntactic structures are there i think it's not that linguists invented them we do see this regularity in this regularities in individuals but the architecture that we use in ai to not follow this bias and we also see that although these architectures are extremely efficient at scale uh this they use a gigantic amount of data certainly a lot more than where we are what we do as humans and so it seems it is there is a possibility that we missed when yeah there is something that lacks in you know we need some problems and do you tend to think that like obviously the central of this debate is the question of whether simply taking the ai systems we have today and scaling them more throwing more data more processing power and more model size at them is going to get us

something like general purpose human reasoning um we're at the risk of having you receive more hate mail here where do you tend to fall on that scaling debate do you think that it's the path or i don't think it's the past but um first of all i should say that even last year i would have said nah let's it's a dead end don't just don't just kill but now that i see the latest results i'm like okay clearly i was wrong i should revise my position uh scaling works i think it's it's clear when you see a lambda gpt3 to some extent um and even when you see a valley 2 and an image imagine a thing is based on t5 underneath so it's for the audience t5 is also a deep language model trained at scale um and so these arguments demonstrate surprisingly good i should say really remarkable levels of understanding in a sense that they can generate images that would have never been in the training sets they can really understand associates features to different objects and they encode traditional or relationships between different objects with very little mistake at this stage and so yes certainly scaling works uh is killing the only answer i don't think so but it's surprisingly better than what i had anticipated okay no that and that makes sense i mean i think it's fair to say gpd3 and broadly the scaling story just took basically everyone by surprise it seems including even open ai when they built gpd3 i don't think anybody really expected it to have all the capabilities that it had but one question that does come up a lot too when we start to look at these capabilities is you have people look at these systems and say well yes it can do super impressive thing x y and z uh but is it really understanding is it really thinking is it really deeply kind of comprehending these concepts and um and there's an interesting debate going on there that i think you'd be in a great position to opine i know i saw this in your twitter thread too what are your thoughts when it comes to this question of whether uh whether ai systems understand things whether that's a useful term to use whether it's even an interesting term to think about in this context i think you certainly think it's interesting and useful but but i think most of us in the community uh agree that it's an ill-posed uh problem at this stage we don't have a good way of formalizing this issue but in terms of question answering it demonstrates uh these kind of algorithms demonstrate remarkable level of understanding right they can identify who is the agent who is a receiver they can do some symbolic or zero shots symbolic processing if you say x equals dog and y equals cat and y is chasing x you will understand or you will be able to predict uh what is the the content of the meaning of the sentence to some extent in the way or at least the way by which we think that it understand is because it can produce the right behavior when we ask it so and when we do this to individuals that's the only thing we have right we only have behavior like how do we how do i know that you truly understand uh is only by asking questions um i think it's an ill post problem at the moment i hope that in the future we'll be able to formalize it a bit better perhaps by using analogies by trying to uh do uh i don't know uh ask um you know maybe you see i'm i'm not i think i didn't i don't have a good answer to provide because i think i'm typically representative of the fields on that particular issue my position however is that um this level of understanding is now very difficult to question uh because of the behavior that these algorithms have now one thing that i would like to stress however is that there are different levels of understanding when you watch a movie the same movie watched by a kid by an adult will not resonate in the same way because we don't have the same uh prism to read and to watch that particular movie and so understanding at the end of the day is also how you combine this sensory input to how you represent the world internally and i think that it's very clear that there are different levels of understanding that have been already reached by these algorithms syntax i think is i mean if it's not solved it's it will be uh very soon um however the structure of narratives uh which requires a processing of very large amount of text which is not something directly suitable to uh transformers at least off-the-shelf one is something which is i think not necessarily um well well mastered but i have little doubt that it will be in the coming future yeah and it's really interesting you talk about this idea of you know child watching a movie versus an adult watching a movie too um sort of suggests i'd be curious about your take on this but like a potential challenge when it comes to almost empathizing with our ai systems not to kind of use an overloaded term there but you know you think about the ways in which you know like a two-year-old is stupider than a newborn or the ways in which a five-year-old is stupider than an eight-year-old and so on the kinds of mistakes that you make as you approach adult human intelligence along that trajectory they look very different from the kinds of mistakes that we see from other ways of approaching intelligence including evolution like a cat makes very different kinds of mistakes from an amoeba and a monkey a cat and we only have experience approaching intelligence pretty much through those two vectors either child rearing or evolution and yet with ai we're trying to make pronouncements like this thing does or does not understand where we kind of already know that i mean if all you ever saw was child rearing children you'd be in a really tough spot to try to say whether a cat could understand something or whether a monkey could it kind of seems like we face the same problem with ai there's no reason to

expect that ai systems would display the same kinds of hints that they are or are not understanding stuff as they improve it kind of seems like there's no fire alarm there's no moment where we go oh okay like that that's a system that really understands yes i think i think you make a very important point here especially in the domain of ethology so when you study different individuals to try to assess the level of intelligence in different domains the typical criticism is the issue of biases and so people will say okay the cats did not understand the cat is really used in experiments that i know of but this animal did not understand that there was food under this box because it's not really interested by boxes in general and so people really try to design new type of experiments such that it would be more uh ecologically valid with regards to the agent but in the case of ai of course it's not clear right because the biases that it has are basically those that exist in the corpora in the corpuses i'm not sure how to use the plural form here um in the text that's uh that these algorithms are trained with um so it's not entirely clear um yeah what kind of biases we should expect or whether these algorithms should uh should be able to not be too constrained by those biases that exist in the text yeah and actually it's funny because there's this um this argument that goes that you know prompting an ai system can reveal the presence of capabilities but not the absence of capabilities and like we've seen that in the context of gpd3 where for example famously i think somebody put together a bunch of nonsense questions they asked gbd3 like how many sparkles go into a morgle or something and it would answer those questions like a like a kid just making up an answer on a test and so people looked at this and they said oh well this is proof that come on gvd3 doesn't understand nonsense questions how clever of a system could this be if it screws that up but then somebody later tried experimenting with a prompt that said you know this is a conversation between a super intelligent ai and a human if the human asks a nonsense question the ai responds with yo be real and they ran that test and sure enough gbd three answered those nonsense questions with you be real so just by prompting it the wrong way by looking for the wrong kinds of biases as you're flagging here we can end up completely missing a plot and thinking that an ai system completely lacks a set of capabilities that it might completely have mastered yeah i think that's these examples are terrific actually because they highlight a new type of task which is the art of probing these algorithms um i think what this stresses is what these algorithms are fundamentally trying to do they're trying they don't try to give the right answer right they answer which is most likely given the preceding text and that's something to really keep in mind because i think that's very different from what we do in many ways um when we have a discussion uh perhaps we also try to anticipate what the author is going to say but it's not the only thing that we're trying to do right so for instance one thing that we uh try to do is honestly predict every individual word that you are about to say but to anticipate the overall idea that you are about to express even though this idea may be expressed by different sequences of word for instance if you say the cat is chased by the dog or the dog chases the cat each and every word in these two sentences can be different and so if you were to make this prediction at the word level uh you would be wrong potentially all the time but if you take a sort of a higher or deeper or more abstract level of representation then perhaps you could anticipate that but even anticipation is not the only thing that we do for instance we try to do control so when we have a conversation but in fact in any kind of situation that requires intelligence we don't we're not just observational individual we can try to say or to one of our objective is to say i would like to guide the person to do this or that behavior and that's not what gpt3 is trained to be because this kind of algorithms is just trained as a passive observer and so i think it's very important to highlight that they are really core it's not just uh gpt3 does not understand x or y or it's failed on this particular math test is that fundamentally it tries to predict what's the most likely outcome it's not trying to generate a behavior which is going to maximize another type of value and actually it's funny this has so much overlap with a lot of the conversations we have about ai safety on the podcast where you know this idea of these systems having objectives that are slightly mis-specified they're not quite what we want them to do um they're importantly different and as systems become more capable those differences start to become clearer and clearer do you see overlap then between this idea of like optimizing language models with their objective function being more tailored to that kind of i guess helpfulness goal um and the idea of

ai safety maybe more broadly um so for ai safety i don't think i'm the right person to i don't think i would have very interesting things to say but with regard to the previous one so for instance the idea of predicting the next word given the context seems extremely arbitrary in fact it's been it is an objective that has been criticized many times especially before a language model started to really uh take up speed and what's remarkable is that even though this task seems extremely arbitrary it's extremely useful to learn other things like for instance people have shown that if you just train algorithms to predict the next words you will be able in the middle layers of the deep nets to recover syntactic structures you will be able to find uh an explicit representation of analogy in the sense that syntactic structures and analogy can be read out by a linear system just one layer of neurons will be sufficient if you plug it into the middle layers of these deep nets you will recover this information and so what's remarkable is that this simple and seemingly arbitrary objective was sufficient for other phones all the other structures to emerge and to be led and so i think that's really at least for me that's the take-home right there are some laws that seem very um distant from what we would think as being helpful uh for intelligence and which nevertheless because a scale uh which nevertheless allowed the emergence of intelligent uh intelligent behavior absolutely fascinating okay i have one more question that i really wanted to ask you and i couldn't figure out how to segue it in but damn it i have to ask this question okay this is super speculative but uh it has to do with brain machine interfaces and maybe you can see where this is going in terms of the research that you've done but it just when i saw your twitter thread when i looked at the paper i couldn't help but wonder you know like when you look at a deep neural network that has you know a layer at which we have a certain representation of an input and that input seems to correlate really nicely to behaviors that we see in the physical human brain is there a sense in which we could map those two together physically use this kind of understanding of the way the brain process information uh to build more kind of sophisticated forms of brain machine interfaces is that is that super ridiculous it i very much well could be but i'm just curious about that possibility so it depends exactly what you have in mind but if the question is can we use the mapping between uh divnets trained on a particular task like image processing or language processing and the brain whether this mapping allows not only to better understand how the brain functions and how the tip functions but also would potentially allow for decoding brain activity of to sort of have a technology which allows one to do to transcribe an activity into uh something which is meaningful i think this is certainly the case and this has started in several lands actually so people start to use brain imaging which they align with pre-trained deep nets and to regenerate the images that people are likely to see given their brain activity um at the moment this is mainly working in the context of sensory processing so typically when people watch images it kinds of work when people are trying to convey something so when they think about an image but the signal to noise ratio here is it tends to be reduced so there are a few uh key challenges so this has also been done with uh patients who um have electrodes inside the head because of clinical uh reasons and typically it's individuals who suffer from epilepsy and you can record an activity and again try to see whether you can decode these brain responses to allow the individual to i don't know write text on a computer screen this there's been two or three studies in the past year that showed individual uh patients can uh generate texts uh with this kind of approach so brainy coding is something which is possible the issue is whether it's i think to some extent scalable so for intracranial recordings um to me it's it's only relevant and it's already a high importance but it's only relevant for patients who have interact on your recordings which is extremely invasive technology and therefore uh implies a huge amount of health risk for the individuals again the reason why these patients have these electrodes is not to do these brand new coding things this brain creating putting things just uh a side research that has nothing to do with with the reason why they have an interest inside that um but for a healthy individual like you and me i presume um it's a bit more difficult because uh brain non-invasive uh decoding non-invasive productivity is much harder the signal to noise ratio is lower and perhaps more importantly the setup that we use typically the mri data that we use collected in this magnetic ring which is using and next mind reading in this case only possible in a very controlled experimental setup uh and not with sort of a wearable device so the i think the big frontiers for tomorrow is to try to see not only whether it's possible in laboratory condition but whether we can also push a brain decoding thanks to this alignment with deep nets in conditions that are basically scalable and more practical for everyone well absolutely fascinating uh jr thanks so much for joining me for this big exploration of what turned out to be so many different exciting topics is there anywhere you'd recommend people go to follow your work uh either kind of your musings about random things or your research in particular so i'm part of the fed at meta so fundamental ai research um and so the uh the lab as a blog which shows uh the work of all the research not all but a selected portion of the work which is done in the lab most of the studies that are done uh are published on obviously they put on archive or by archive servers and

otherwise for me personally i tend to advertise our latest work on twitter but that's pretty much it that's a follow i can very much recommend will jr thanks so much this is a very fun conversation all right thank you very much