How I Read a Paper: Facebook's DETR (Video Tutorial)

hi there people so a lot of you have asked me how I read papers and honestly I don't think there is any super special method to it but you know I thought because people have asked me to make a video on it so I'll and I'll try to share my method of reading papers and hopefully this is going to be somewhat of a miniseries or a series where I every now and then discuss how I read one of the papers that I make videos about and I'll try to select them such that different things are highlighted now I've selected this one right here for really for no particular reason other than I sort of remembered it and going to try to go with you through how I read this and how I encountered this and kind of try to honestly share what I thought at the first time when I read it and I hope this helps some of you if it does help you and if you like content like this of course feel free to share this out and subscribe if you haven't seen my original video on this paper it might be worth to go watch it I'll link it and with that let's dive in so again this might be not really something new but I'll just go through it okay so first

thing I do is of course read the title so the title has three parts and to end object detection with transformers so what I notice that I do myself is I like through reading a paper it's like read the paper with an open mind I don't do that I almost immediately form an opinion and a hypothesis of what's going on like so I see transformers so I know what transformers are if you don't I've made a video if made lots of videos on transformers attention is all you need is the base paper for that so I know what a transformer is okay and I know that transformers are usually in NLP are usually used in NLP door there are things like you know other things with transformers but usually an NLP model then I read object detection and I know object detection is a computer vision tasks so immediately this here is sort of a difference and I immediately try to assess what's the new idea in this paper and in this case it might be okay it might be applying transformers to object detection but then I also see end to end and the only reason to put that in a title is because that's the novelty because usually in deep learning we're sort of used to systems being and to end and even if they aren't if most systems aren't end-to-end a lot of people don't is like end to end image classification on image net like Thanks so I was guessing that the reason they put in end-to-end into the title was because that's actually something that's special about the model so now I have like two competing hypotheses of why this paper matters first of all because it does it with transformers and second end to end and the of course the true fear is that the combination of end to end transformers all of that is what makes this model and I already form like a hypothesis of whether I like this or not like I have to be honest I have very quick judgment of papers of whether I like them or not and then I sort of catch myself each time and I still try to so they're for most papers actually that I have sort of a negative opinion at the beginning where i will- there are papers where I think like there is no way this is going to you know work or something like this I'm actually positively convinced throughout the paper so for most papers where I that I read I'm trying to find the positive things in there but I do form an opinion pretty quickly usually

alright so the second thing this part right here I like I don't even see this is like advertisements on like Twitter I like you just I have always had issues with author names like people would come to me and be like oh have you seen the new vineos paper and no clue and then when they say like oh that's where they use this character level model to do that and I'm like oh that paper so I like do not care who the authors are of a paper like I don't I can't remember the papers by their author names I've gotten better at it I have to say but i-i've always had trouble with this now that's not to say that a name doesn't pop out to me like if this would be like it like yoshua bengio or some someone like really famous then of course that would catch my eye and but I also know that you know yoshua bengio is paper yoshua bengio slab is huge so just because a big name is on the paper doesn't mean that the paper is going to be of any good or bad quality sometimes the author's give you an indication of what kind of research is there like if you see a Najaf Clun or Ken Kenneth Oh Stanley you know that there's going to be the certain type of you know learning to explore and and kind of a bit more out-of-the-box thinking in their papers which I really like but it doesn't immediately give you a clue maybe if you go by first authors it's much more indicative if you have already read some of their papers but most often I just ignore authors and go on the affiliation sometimes matters in

that it's a bit of a vicious cycle if there's a big name affiliation like Facebook AI Google ai and so on these papers also they get more exposure in like the press and so on so whatever Google a publish that paper all of these all these pops eye magazines like verge and this and life hacker and hacker news and whatnot they like write a blurb about it so often they get much more scrutinized for these papers the public attention but they also get much more scrutiny which in turn means that there is a bit more pressure on them to do good experiments so that biases meet like a little bit into the direction of believing their experimental evidence more now usually this is also backed up by the fact that I am actually convinced by their experiments usually so these big-name papers often I find myself even without or disregarding the affiliation to be convinced more than of like regular papers my most often issue with papers is that I don't believe the experiments and I make no difference like even if it's Facebook I still my prior is the experiments or crap and I don't believe them and they have to convince me of the opposite but some like I can't say that it doesn't affect me that it's like a big name affiliation okay so then the second thing is I sometimes I see the paper on archive and I skim the abstract

sometimes the abstract is informative and sometimes not so here it's like blah blah a new method that views object detection as a direct set prediction problem I'm like oh yeah okay so streamlines the detection effectively removing the need for many hand design components like non maximum suppression yada yada the main ingredients called detection transformer asset based global loss that forces unique prediction via bipartite matching and the transformer encode or decode or architecture so they make it clear here why it matters and that's what I want to get at is sort of what's the new thing in this paper most papers are even though they're all very long and have lots of math and so on they often have like one or maybe two new core things that they really tell you sometimes zero but a lot of times it's like one thing that they really do and you sort of have to but they're trying to cloak it often because they need to make their research as impactful as possible right but you need to sort of figure out what it is they're doing they make it fairly easy for us in that they say okay they remove the need for many hand designed components like non maximum suppression which tells me that they're building something that's easier than what came before them and that already tells me it it's not necessarily going to be better their argument is more that it's going to be easier right there are sort of two kinds of experimental results the ones where you try to beat what came before you and the ones where you're trying to say look our thing works just as well as this other thing while being more advantageous in some other metric so I would place this already in the sort of second category and then they say what are the actual ingredients it's a set based global loss that forces unique predictions via bipartite matching now I at this point I know what these terms mean but at this point actually don't have to know what the terms mean what I need to recognize is that I simply have to go later and figure out what that is and a transformer based encoder/decoder architecture okay so there are two things right here that I remember I need to pay attention to later there is this loss which seems to be special and there is the transformer architecture which seemed which they say okay that that's the model basically consists of those two things and then they have a short description of what it does given a fixed small set of learned object queries that your reasons about the relations of the objects and the global image context to directly output the final set of predicted in parallel that almost tells me nothing of yeah okay the model reasons maybe this in parallel is something but the model is conceptually simple and does not require specialized library unlike many other modern detectors this sort of repeats this enforces my hypothesis that they're going with the hey this is a much easier way of doing things approach detter demonstrates accuracy and runtime performance on par with well established that further confirms my hypothesis that this is on are right they runtime performance on par with the current state of the art and at the end they say moreover data can easily be generalized to proceed to produce Panoptix segmentation in a unified manner we show that it significantly outperforms competitive baselines training code and praetor models are available as part when I first read it is like ok can easily be generalized to produce this Panoptix segmentation this is I didn't know yet whether this is like a central claim of their paper that it can do this segmentation or whether this is like an added benefit to their paper because you can read it in both ways and I'm just ready to find this out in the paper now after I've read the abstract and sort of already form the hypothesis of what's going on so here I already in my mind I already sort of have a model of how would I do that right and then what would I do so right now what I might be thinking is if I have a transformer over images that directly outputs the predictions in parallel I'm imagining like an image and the image somehow needs to go into a transformer so maybe there's like an encoder like a CNN encoder that gives me image features and then it's so maybe you sample this down this image this is just me hypothesizing what could be going on right and then I might be unrolling that right this image into a vector of these lower pixels and then so in my mind what I would do right here if without knowing anything more would be to do something like Bert's pan predictions so I would have Bert right here and I so for I would input this sequence right here and then to detect an object I would sort of think that maybe the Bert you know Bert has an output that is the same length as the input right so it's very good at sequence tagging and things like this so maybe how it detects an object is going to be that it sort of like tags the center location in the pixel of an object right here or a tag somehow the corners of the bounding box but then I don't know how this is going to be in parallel maybe Bert outputs like a score for each location and then you do some kind of matching right here so this is my initial hypothesis of what's going on and then I scroll through and

honestly the first thing I do is I go and find the pictures and know no different in all like since you first book you read that's what you do I go and find the pictures because usually if someone proposes anything new that they're gonna try to make a picture of it luckily I don't do like super theoretical what not your Bayesian generalization bounds and I don't know so most often papers I read have some sort of picture and that's very helpful to me I know but yeah so I find this picture and here I see okay you have image you have CNN okay gives you a set of image features or so far so good then transform or encoder decoder then set of box predictions so all of them come out here and I already read they're in parallel and then bipartite matching laws so here they I can see they color these in different ways and these color appear to match with these colors right here right in the green here and these they also this is a very good graphic but from this I can already read that these here go to the no object a lot of times the graphics aren't very good so this is where I'm not saying in every paper you can learn by looking at the graphics like sometimes the graphics are terrible and you're like what's going on here I like I don't this makes no sense this happens a lot in this paper right here this happens to be very good explanatory graphics so I'll take advantage of that and I do the same thing in the other papers right but then later when it doesn't match what I read in the text I'll have to you know update my belief and so on but here I see that these go to no object and this goes to no object so I don't know yet that this is the test set at the point where I read this I was sort of confused by this but I recognized that each of these boxes right here is going to be either resulting in a bounding box or in the no object prediction so from that I could conclude that these things here are maybe some sort of a fixed set right but I still thought that you know these that this would actually be the output of these image features so that in this case you'd have like six set of image features and then you'd have like Bert here even though that's not an encoder decoder I still this was still my running hypothesis that somehow you'd map these image features to these boxes right here so and I didn't know what to make of this thing right here so then I went through some more and look for more pictures and there are not sometimes I also kind of glanced at the formulas but okay when I Everest I see this is just I mean this is kind of useless like okay cool you minimize the loss thanks this okay didn't really pay attention to that ah new picture cool so this picture is much more informative than the other picture yeah I believe with the other picture they were trying to show case this loss how they do the matching and even though I could read a lot from that picture I did not get that part and that therefore I felt when I saw this and I just glanced at it I'm like wait what what's different then up here it seems like the same but okay let's look at this so again we see okay you have set of image features that comes out of the CNN so that conforms with my belief but then this here goes into a transformer encoder and this comes out so immediately I see oh this is not the same as these boxes here right that was my hypothesis that these things here would be the colored boxes so I say okay obviously that's not what happens this thing here seems to be sort of the encoded image information then that's somehow fed into here and that then there are these object query things and they seem to correspond to this so I'm a bit more confused right now what I can see is that these then will result in these boxes okay so being confused by that I look for more pictures so I go look for more pictures and this here seems to be like of a visualization a lot of these papers have some sort of ablation experiments or so and so on this I just find a really cool picture for now I don't know yet what it means this and I go down skip off this and then back here in the appendix I find this here which I immediately map to the previous where this is the anchor and this is a decoder and I've already read the attention is all you need paper and at that point it clicked and means like this is not a Burt transformer this is one of these transformers that has an encoder and the decoder even though they told me like 50 billion times already I was too stupid until this point so now I know okay I see what's going on so the image goes through here and then this goes as a side input like as an attention from the decoder to the encoder like I know in NLP right so in NLP this here would be a source sequence like maybe if you do translation and this here would be a target sequence so now whenever I see a transformer like this and it outputs something this I look at it as okay this here is sort of the input that goes as like a side input over here and usually here you have the target sequence but that's not the case right here right you have these object queries so this is how far I get from the pictures now I go up so I have a sort of I have questions now I have questions and that's when I start reading the paper only now do after I've looked through all the images form the hypothesis and sort of have questions on how this works and we'll go a bit faster from now on - just not bore you with all

the things so the introduction is often very important even though it's called introduction and maybe you know if you read a book like if there's like introduction or a prologue or something like this it's often kinda pointless introduction in these research papers is one of the most important points because all of these papers they try basically all of them try to convince a reviewer to accept them and in order to do that they will set up their main points and their main story immediately in the introduction so what you'll usually have is a problem statement which is here like why what's wrong right now and then you have like a story of how their paper addresses the issue okay and that's here we streamline the training pipeline by viewing object prediction yada yada this is often formulates in words what the paper is about and what contribution the paper makes right this is like a longer abstract the abstract is often very cryptic very dense this here is often much more informative of what the paper does so for understanding the paper and a high level the introduction is the best place but given that I've already looked at the images and so on I don't actually draw many new much new information from

this thing then is related work and honestly I skip it like unless I'm the actual reviewer of a paper like when I'm the I read the related work but often related work is just like you first of all you site a bunch of your friends and then you cite the mandatory papers every single person that you think could be a reviewer because or you've actually been rejected from a conference with a reviewer claiming that your you haven't compared or you haven't cited data or that paper you can pretty much be sure that that's the if it's not a glaring of may omission if it's like a niche paper and you haven't cited it then you're like okay I'm gonna cite it just because the next conference you could be Mary viewer again so I'm not sure that these related work sections they're necessary like if someone wants to write their theses and they go and read this paper and they want references oftentimes this is a good place but a lot of it is just blah blah blah okay I know disagree with me if you want oh yeah - maybe - reading quality so I tend to at this point I tend to not skim so at first I skim but at this point I tend to read every sentence and read it closely and understand it and when I realize like I'm tired or something I don't just skim the paper I've tried to skim papers and it doesn't work try to read every sentence understand every sentence and okay if you don't understand it don't stop reading because of that but try to not skim and be like oh yeah yeah okay I gotta go to get away that is not helpful except related work skip completely cool then a lot of times in this paper now is the

model and this is the section I'm actually interested in right so I read very closely here and then I find out what their loss is all about and again I stress read these things and understand them right sometimes it's hard but if you're confused that means you either if they've done a bad job or they made a mistake or that you haven't under stood something if you can't understand the sentence try to read on maybe it's clarified later and then you know go back but again do not like just start a lot of times when I read paper previously like I wouldn't understand something quite well yet and then I would be like oh yeah yeah and then I noticed that I start skipping and skimming more and more because that would you know pop up again and I wouldn't understand it again and then at the end I would just be kind of glancing at the paper and I don't want to do that right here so I want to read every sentence and understand it okay so here then I find out about the loss and then I if I don't know something here then I'll go and look it up on maybe on Wikipedia or something like this now I don't need to understand every single part of it right that's maybe I should correct myself so for example this bounding box loss here they talk about the second part of the max across the Hungarian Falls is this box loss that scores bounding boxes unlike many detectors that do box prediction with some Englishmen the other yada-yada they say the most commonly used l1 loss will have different scales for a small so here they basically talk about how they mix the losses they say overall our box loss is that defined as this and this now I haven't I don't know what these losses are I just assume there are some bounding box losses so when I it's not true when I say understand everything understand the things that are integral to the story of the paper right how exactly they compute bounding box losses at this point I don't care I just assume that there is some loss that I can back propagate right I what is important is that they do this Hungarian matching thing right as soon as I get that I'm like ah that was this you know this um this thing no this thing up here this thing this with the matching thing now I get it now I know uh there are always the same amount of boxes here labels here and all we need to do is somehow match them and I immediately think why is that relevant oh because when something is already matched to an object some other thing cannot be matched to the same object and that's how we you know prevent the fact that all the things predict the same thing right and so that immediately becomes clear and as I said there is usually like one or two ideas in a paper I don't assume or I don't care what their exact loss function is because I've sort of gotten the idea up here of what the loss is about alright so I hope that's clear under very closely read the things and understand the things that are necessary for the story if you find if you think something's not necessary for the story and then later end up not understanding that maybe come back and you know read it again in any case I would rather I would rather skip something and assume it's not necessary if I think so and then come back then trying to understand every everything but the things I do read I try to understand thoroughly okay then there's the architecture okay and that again I read closely and get backbone ok transformer encoder ok and now I understand much more closely a decoder ok and here I get now finally I get what this is about because n objects in parallel yada yada these input embeddings are learned positional encodings that we refer to as object queries and similarly to the encode we add them to the input at each attention layer so now they name i've already seen these object queries here and the only word i actually need from this sentence are learnt the fact that their positional encodings I just kind of ignore as soon as they say learnt I know AHA these things here are learned they have actually they're always the same for each of the images they're just over all learned okay so now I feel I understand the entire model and yeah so then they say auxiliary decoding losses and this sometimes you have to pay attention to like exhilarating because those are the things that here they say explicitly we found helpful to use auxiliary losses sometimes they won't say why they did it they will just say our loss consists of three things and you know if you look at the three things only one of the things is really a part of their story so far and that you should immediately conclude that they've put in the other things because they tried it and it didn't work right so you can also kind of get an estimate of the brittleness and so on of the system in that you see how many unnecessary things are there or how many things are not straightforward how many things aren't the easiest thing that you would do when you would go about and do what they did okay so then you this

concludes this model or method usually the student section is called like method or model or something like this and you go to experiments now the main question I have so far or I have maybe I have some more questions about the model itself that I haven't been able to pick up from this section which is not the case here but I simply keep those questions in mind and see whether they are resolved later right so I keep an awareness of what I don't understand but from here on my main issue is are they demonstrating that their story works right so they're here they're proposing a loss and a model and in my mind they now need to convince me that works and that's it's not as easy as simply to show me some numbers that they are good at some benchmark they need to show me that they get those numbers because of what they claim so here they claim well okay they propose a new architecture so what they need to convince me of is that the architecture itself makes sense right but in other papers when you propose like and when you say like oh we for example in an L STM when you build in an attention mechanism and you claim oh we you know the attention mechanism can look back at the source sequence in one step then you need to convince me that actually happens right so you need to not only need you need to perform well you need to convince me that you perform well because of what you claim your model does right so and that's often difficult and I specifically look out in the experiments for usually the question is like where are they trying to me right are or are they trying to me are they trying to cover up the fact that something doesn't work now all the experiments are always in the best light possible of course and you have to keep that in mind but a lot of times you can also already see from the experiments that okay are they doing something weird are they not showing me some obvious experiment or and that's a lot of time because is there an easier explanation for why they get the results that they get other than their explanation right and it is their job to convince you that their explanation is the correct one for these numbers and especially if there is an easier one that they haven't excluded and then I don't believe the experiments if that's the case right if there is an easier explanation for the effect I'm very skeptical but some papers have an easier job here than other papers so in this paper they basically show results on a on a task and since their paper is about hey our pipeline is just easier than other pipelines what they first of all need to do is they need to like match the numbers of other pipelines and here I see that okay in these results often in the aftermath maybe a table or something here you see like this their model other models and their model is the best model in a lot of cases now if the best thing is of course if is if their model throughout is the best the worst thing is if it's like scattered like this even if their model is the best but in every single benchmark a different configuration of their model is the best that's sort of a bad sign unless they can explicitly explain why that is and it's also not that good of a sign if these things are spread out like this like sometimes this baseline is good sometimes their model is better and so on so pay attention to that now in this paper it doesn't matter so much that's actually fine because what they're trying to show is that their model is on par and way easier and they've already made the case in what way it is easier it's easier in terms of architecture if there were to say it's much faster then after that I would expect you know an experiment in speed while these numbers are matched so but since they say it's easier I've already seen the architecture I'm convinced of that now that they show okay our numbers match actually I'm surprised they even outperform a lot of times then I'm quite happy with these experiments so also look for differences between numbers and the spread of numbers now it's not easy to say what if like point 1 is a bigger a small difference that depends on the task but if you know pay attention to these things pay attention to the fact that these results are noisy and oftentimes there is a lot more hyper parameter tuning going into the model of the paper then into the baseline model so why do you want to make your look your stuff look as good as possible and here is a little bit where the institutional credibility of someone like Facebook comes in that I tend to believe their results a bit more than other results megha but a bit more yeah also look at patterns that they don't point out in the text so if there is like a pattern if you see like an interaction between the number of parameters and the score or something like this just try to be on the lookout of that and see if you can spot something that you think or think about whether that makes sense or not in what your hypothesis would be so here we go on and okay then they go into ablations and a lot of these papers do appellations and i generally appreciate that so here they visualize that the attention mechanism in their model actually refers to different instances right encoder self attention for a set of reference points the encoder is able to separate individual instances and you can see that pretty clearly right here where and even here with the overlapping cows and this is the sort of experiment that I would expect that actually convinces me that their architecture does what it says that it does right and something like this where you see like totally overlapping things with the attention of the individual things visualized so telling me like especially this one right here the foot of the back elephant actually being focused by the attention of the bounding box of the back elephant that's the sort of experiment that convinces me that their claims like that their numbers really come from what they claim it comes from okay so at the end of the experimental section you should always ask yourself have they really convinced me that their story is true right that the improvement or winnig whenever they get an improvement or whatever they get whether is due to the story that they want to sell me or could there be like an easier explanation or does something not fit is like other are the experiments different than from what you would SPECT here okay so these are my main questions are they convincing me of their story it's not do they have state of the art numbers I don't care even though like sometimes so there is a bit of a catch I don't care about state of the art numbers now let's say you have a table like this and you have a computer vision model and one of the models is like on the C for 10 data set now if your baseline model has like a ninety one ninety two percent accuracy on C for ten when I know the state of the art is 96 I don't care right I know like I've done C for ten I know with like I don't know five six layer CNN you can reach these 91 92 93 % accuracy and to get to the 96 97 you would actually be like in the region of a wide ResNet and whatnot so I you know I know that even though you're a few points behind state of the art I know you know this is valid still so I don't care but if you were to be like at 80 percent accuracy on C for 10 then I get a bit like I like it's pretty easy to get to 90 percent plus with like a standard CNN so there I immediately start to wonder why is there an explanation now this could be like a theoretical paper that says oh we investigate MLPs and that's why we only get that number so that's that would be fine but if something is out of the ordinary like this then I pay attention but never because something isn't like the latest and greatest state of the earth that's just dumb ok and also if only evaluate what the paper claims it does right if the paper says we want to show that we are on par with current models then don't be mad if the paper doesn't outperform these models they didn't claim that right so yeah so after these ablations I'm actually pretty happy right here with the results and this right here when I saw this I didn't expect that but I read the experiment description that these are these different learned object queries and what they do and that gave me an increased understanding of how these object queries actually work right so at that point I still had like a vague I knew that these are learned but reading this and sort of looking at it studying it a bit I was like oh okay then I understood even better what they are so again when I say understand everything in the method section you can still have questions and but you just have to keep it in mind for later and then here I go on and there's this de TR for Panoptix segmentation and they here they propose like a new model so I first look at it and I'm like okay they proposed a new model they can do stuff like this now this is not object detection and again I'm not sure is this like a an add-on to the method or is was this up here just an intermediate step to this and honestly after reading that I still wasn't sure it seems like something in between of course the paper is also a bit longer than other papers it just it seems it's too long for just being a side note but it's too short for being its own thing so that was just a bit weird and I treated it as just like a oh we can also do this with our model but I didn't pay like too much attention to that okay so at the end I you know

look at conclusions now the conclusions of a paper are much often they are not nearly as informative as the introduction the conclusions they all often tend to be very generic and kind of hedging a bit against criticisms saying what would be up for future work which is again hedging against criticism because you could simply say well we didn't do this that's future work yes so again I read it but I don't really pay attention to it and then I gloss over the abstract I just would kind of scroll through the abstract if there's something that catches my eye I would look at it and if not then not and then

I basically go to the start and whenever I didn't understand something I go back I look at it again and I try to think are all my questions answered and have they sufficiently convinced me that their story is the thing that really has the effect right here and then if I now were to make a video of this I've often found it useful to just put the paper away for a while and it's I usually get the best result when I read the paper the day before and then make a video the day after or if not I'll just you know put it away do something else do some email responding programming going outside eating lunch just some kind of a break between first read or between your first couple of reads and just I don't even think about the paper I just it's kind of it's in the subconscious it kind of bruised right and I happen to think about the paper every now and then but I don't make a conscious effort to be like oh how am I gonna explain this and so on but I just found the worst videos are the ones where I immediately make the video after reading a paper and I've just discovered that if I kind of take a break and then I look at it again right I look I don't read it fully again but I if I have the feeling I've understood it I don't read it fully again but I just kind of look at it and go again through this story and I think that's even if you know wanna if you want to talk about a paper in a reading group or tell you know explain it to your friends or whatnot this is often very useful just put it away for a while let it Mellow and I find that helps a lot okay that was my process of reading this particular paper now we again this is a high quality paper so it's I find it's a pretty easy read in that I simply need to understand what they did and I'm pretty happy with their experiments I maybe next time I can find an experiment or a paper where I'm initially more skeptical and not as happy with what I find but yeah let me know if you enjoy this or if you would like to see any other explanation I don't exactly know if this is what you expect it from a video like this so let me know maybe I have misunderstood you completely or it's way too long way too detailed or way too undetailed yeah leave me a comment and I'll see you next time bye