[Trash] Automated Inference on Criminality using Face Images

hi there take a look at these faces try to decide which of these faces are criminals and which ones are law-abiding citizens I'll give you a second okay got it so if you decided that these four here are the criminals you would be correct and that makes these three the law-abiding citizens as for this one maybe if the crime is being too cool of

course none of these faces actually exists in real life these are compositions of eigenfaces of datasets of criminals and non criminals today's paper is a absolute controversy this is going to get me into so much trouble so if you see something like this in the news always always go and check now we're going to look at automated inference on criminality using face images by shouting woo and G Cheung on high level they're trying to separate criminals from non criminals using face images so basically using classifiers on ID photos this of course has generated quite the uproar I suggest we just dive into the paper and look at what's happening right here

we study for the first time automated inference on criminality based solely on still face images which is free of any biases and subjective judgments of human observers right so they say we train a bunch of models including as you can see a CNN using facial images of 1856 real persons controlled for race gender age and facial expressions nearly half of whom were convicted criminals force criminais ting between criminals and non criminals so this is the outset this is the kind of research question here now immediately you have people jumping up saying that's not possible and I would agree but I think actually there are very interesting lessons to be learned from this paper so they're saying they actually managed to do this with their classifiers actually with all of these classifiers of course deep learning being the best also some discriminating structural features for predicting criminality have been found by machine learning so they even tell you why above all the most important discovery of this research is that criminal and non-criminal face images populate two quite distinctive manifolds the variation among criminal faces is significantly greater than that of non criminal faces the two manifolds consisting of criminal and non-criminal faces appear to be concentric with the non criminal manifold lying in the kernel with the smaller span exhibiting a law of normality for faces of non criminals oh I'm gonna be canceled I don't advocate for it this is not this I'm not a fan of this just in other words the faces of general law-abiding public have a greater degree of resemblance compared with the faces of criminals or criminals have a higher degree of dissimilarity in facial appearance than non criminals so basically what they're saying is that the this kind of similarity among the non criminals in their dataset is larger than the singularity among the criminals okay so already the outset right then they go into this introduction and in the introduction we won't go in through fully but they basically introduced the concept of facial recognition they try to build up kind of an argument where they say faces are different some people have hypothesized that it's possible to infer personality traits from face facial features some studies exist that show that people agree on the perception of these traits so not the actual traits but people will kind of agree that a face looks extroverted or more agreeable people tend to agree that the appearance exists and then they sort of make the next step and say okay can facial features also be used not just for predicting the appearance but to predict the actual personality trait for validating the hypothesis on the correlations between the innate traits and social behaviors of a person and the physical characteristics of that person's face it would be hard push to find a more convincing experiment than examining the success rate of discriminating between criminals and non criminals so actually you could agree with this right since this is sort of a distinction one can make about behavior whether or not someone breaks the law or in this case is caught and convicted and so on there are like many hurdles in this in essence the statement sort of makes sense like if you could actually do this from facial features that would be very first of all very surprising and second of all very drastic people immediately jump to the conclusion that okay if such a thing were found that means you could somehow pre cognate criminality which I don't think it has to be because what could also be the case is they have a quote from Aristotle right

here it is possible to infer character from features if it is granted that body and soul are changed together by the natural affections one interpretation of me is that let's say you break the law for whatever it could be completely moral like you steal the medicine for under the old lady in your house and but you know you broke the law you know you did something that society doesn't want you to do and that will exert stress on you right you do now have to lie to people about this you now have to sort of make sure you're not caught you have to worry maybe there's a security tape or something like this and the stress will we know that stress will physically change you and that could be in turn made out by your features for example the stress of being in jail could change your physical features and since these are all convicted criminals one might think that it might be possible it might again not saying it is it might so if you if we throw away all of the kind of pre judgments on this it could be an interesting research question right could now whether we want to pursue it or not that's it a different question but the way they build this up here is that they only have the best of intentions in mind I feel like this might not be the case so they say something like this right here at the onset of this study our gut feeling is that modern tools of machine learning and computer vision will refute the validity of physiognomy although the outcomes turn out otherwise

this and this is the part where I just stopped believing them that their intentions were like all good and it's just about disproving this so we can just lay it to rest because they then is very quickly switched when they find something else non criminals are the normals and the criminals are like the that just rubs me the wrong way where you'll have to say me it's like the Ku Klux Klan going like oh no we you know we have many social gatherings and our gut feeling is that people aren't really different and the robes are actually personal protective equipment it's all actually just a community you think we all you know good intentions on every now and then we lent you guys going into this with sort of a mixed bag of feelings where you'd have a hypothetically valid research question but also even the introduction makes it very clear because it's someone over the top promising to just be neutral and be good intended not gonna fall for it

sorry they say in order to conduct their experiments they have 1856 ID photos the following criteria Chinese male between ages of 18 and 55 no facial hair no facial scars or other markings the dataset is called yes then there's two subsets sn4 non-criminals and SC for criminals the non-criminals contains ID photos of alone 126 non criminals that were acquired from the internet using the web spider tool they're from a wide gamut of professions and social status including waiters construction will bubble okay the subset of the criminals contains ID photos of 730 criminals of which 330 as are published as wanted suspects by the Ministry of Public Security of China and by the departments of Public Security for the provinces of Wang dong Jung soo-jung Ling etc the others are provided by City Police Department in China under confidentiality agreement we stress and here here's an important point we stress that the criminal phase images in SCR normal ID photos not police mug shots so they say they have violent crimes or non violent crimes and so on so they have these examples here

of those images so the top ones are the criminals and the bottom ones are the non criminals now people immediately see differences here and if you spotted that all of these have white colors and none of those have white colors then you would be correct now you're on the right path you're not actually correct but here because actually what they do is they mask away the colors so they only extract the face part and the upper neck part so these this white collar part will actually not be on the image that they analyze to control for clothing which is good but it gives you sort of an indication that the origins of the to image groups might not actually be you know the same so what you'll have is you'll have basically a database actually have two databases of criminals which are so the one database is this wanted let's call them W these are released by the police for wanted criminals then the others database is the convicted criminals let's call that C and then on the other side you have the database of non criminals and the non criminals come from the internet so you have three different databases and of course these two make will going to make up the criminals and this will make up the non criminals and the herein lies the problem right you even though the white colors are masked out you have to make sure that whatever you find isn't just a property of how you collected the data and this doesn't really come through in this paper so they do data preparation as again they mask their resize and so on and I stress again all our ID images with frontal lighting so yeah and they okay so now they test the classifiers so they say we test logistic regression KNN SVM and CNN on the image data set so for the CNN you can just input the original image but for the other classifiers you need a set of features and what they do is they concatenate three different image feature vectors so the first one is facial landmark points that you extract by some sort of tool you can extract whatever corners of mouth and so on then the second facial feature vector generated by a modular PCA and the third is the facial feature vector based on a local binary pattern histograms so these are sort of face features that people use for recognizing faces they concatenate them that gives you a feature vector you feed that into the machine learning algorithm

and they do a we perform a 10-fold cross-validation for all possible combinations of three feature different classifiers and the four types of feature vectors plus a date to driven CNN so they do a 10-fold cross-validation right which basically means you do you partition your data into ten parts you take nine to Train predict the one then you take the next that you left out and so on this kind of you get a train test split across all sorts of splits of your data which is a it's a you know it's a valid thing to do and they discover here that there CNN classifier performs at almost 90 percent accuracy as you can see here and even their SVM and the other classifiers it performed fairly well in recognizing these criminality faces so and they analyzed you know the ROC curves and the ROC curves this is a really this is a class her that works right so you can see even the the other models but especially the CNN classifier here works really well of course the question is what does it work for so they basically say alright we now have a classifier that distinguishes criminals from non criminals and I would say you have a classifier that descriptors discriminate your particular pictures of criminals from non criminals and if this were submitted to me as a reviewer I would expect that any sane author would then go and try to invalidate that so here's what you'll have to do if you want to convince me that this is not just due to how you collected your data you need to go and you need to basically say okay I have these different methods of collecting data right here now maybe I can go to the police and ask them for a picture from the same database of a non convicted knotless of a non criminal someone that was arrested but then not convicted and I can you know have someone from here that I can put in that data set and then you have to show me that your classifier will correctly predict that that's a non criminal and if it predicts it's a criminal it's due to the data set and you can also find one of the criminals but find their picture on the internet like you collect the non criminals and that will give you someone from this database in that data set and then you have to show me that your classifier correctly predicts that's a criminal you can further convince me that your classifier is neutral to this separation right here of the wanted and convicted criminals because they all should be criminals right so if your classifier is neutral to that then it basically doesn't care where it comes from so this would be a weaker argument but still one that one could investigate what do they do for validating their method here is where it gets funky so they say given

the high social sensitivities and repercussions of our topic and skeptics on physiognomy we try to exercise maximum caution before publishing our results yeah you failed in playing devil's advocate we design and conduct the following experiments to challenge the validity of the tested classifiers for the task of discriminating between criminals and non criminals all right this is it right here is where you give us where you tell us it's not because of how we collected the data which is the obvious explanation we randomly label the faces in the very sample set s as negative and positive instances with equal probability and redo all the above experiments of binary classification well how can it is this all right they're basically saying well if our classifier were not a criminality classifier that means we could invalidate it by shuffling the labels and if that comes out to 50/50 then our classifier obviously works because it's not 50/50 in this data set so basically they're great they're just validating that a classification algorithm can classify something right The Critic criticism here is never that they haven't actually trained a working classifier the criticism is what have they train the classifier for but they their entire validation procedure is basically we don't have a bug in our cohort oh the outcomes show that the randomly generated negative and positive instances cannot be distinguished at all gee who guessed a classifier on random labels doesn't generalize man in fact we go much further along the self critical path all right here it comes and carry out the same experiments for random labeling on different samples of the same size and with the same variable control only this time in the selection criteria are standard any photos of Chinese female young middle-aged or Caucasian male young middle-aged of Caucasian female young middle-aged no facial so basically if you train on a randomly labeled data set on any day okay these sort of pictures your classifier with a lot work thanks maybe that's a I think that's the academically most valid statement in the entire paper oh man in none of the three cases any of the four clusters managed to achieve a true positive rate higher than fifty three percent on randomly labeled positive and negative instances so the classifier must be valid because okay the above experiments rule out that the good accuracies of the four evaluated classifiers in phase inference on criminality are due to data overfitting no otherwise given the same sample size they would also be able to distinguish between randomly labeled positive and negative instances with significantly better chances than k but they did cross-validation the they did the cross-validation prevents the overfitting we know one criticizes that you owe these people have no idea what they're doing they have no clue of machine learning they don't know what the problems with methods are the they don't know what overfitting is and how you control for it the big jump of the true positive rate from random labeling to truth labeling onto the same set of face image can only be explained by intrinsic separable 'ti of s c and s n that is true but why are they separable that's the question ok as different source cameras generated the ID photos in the set s now that then it might be on the right track here different source cameras may be they get the idea that different data sources lead to different things they might leave their signatures that although below perception threshold in signal strength could mislead machine learning okay so they're basically saying different cameras can generate different sort of artifacts and they rule this out by basically adding noise to the images such that these artifacts would be washed out and that the noise doesn't change their results gee there was so close they were so close to actually doing something useful okay this section is where it gets even more interesting now they're trying to guess from their classifier what are the actual features that make criminals so discriminating features right here having obtained the above strong empirical evidences for the validity of automated face induced inference on criminality one cannot resist the following intriguing questions what features of a human face betray its owner's propensity for crimes um okay Shakespeare um and they basically go and explain ability route where they see were what the classifier pays attention to and it turns out a the following feature so on the left you can see where the classifier pays attention to no surprise here it pays attention to face features but they kind of parse out the following three features first of all the d the distance between the eyes in criminals tends to be smaller than in non criminals the angle between the nose and the corners of the mouth tends to be smaller in criminals than in non criminals and the curvature of the upper lip can to be higher in criminals than in non criminals so let's try just from this information to draw the ultimate criminal and non-criminal faces so first of all the non criminal let's draw the non criminal as just regular I'm not very good at this so here's the nose and then let's just draw the lips like this non criminal perfect looks like a law-abiding citizen to me criminal right here so the eyes are closer together here's the nose and then the curvature of the upper lip is higher so hmm and then the angle between the nose and the outer corners of the mouth is smaller how can I make the angle smaller could it be that if I oh yes ah that's the trick criminal ladies and gentlemen so are you telling me that all someone has to do to be a criminal is frown yeah totally valid so they're so close right but they say all these are intrinsic facial features but come on alright so they go on to

say that they have some Easter gram differences of these features that they basically say these features are what's responsible for this and then they do face clustering which is beautiful so first of all what they do is they sort of take the average faces for criminals and non criminals and these are the average faces so the top are the actual average eigenfaces and the bottom is when you can shift the facial landmarks around the seeming paradox that SC and SN can be classified but the average faces appear almost the same can sorry what a paradox these are almost the same I mean if I just overlay them one over another they're almost the same there is no difference at all I don't see a difference how what could possibly be the difference what could be the difference I don't think that these are the most honest of intentions so they basically do some clustering which I find interesting for example that they don't really explain I so map here so I so map uses the geodesic distance between two points on the manifold which is defined between the sum of the weights but we will do it all right so the kind of washee-washee I so map but they then explain k-means in great detail with formulas and again these I mean okay not machine learning people can do machine learning that's fine but they're not really into the matter here and they try k-means

clustering and they find in their opinion they find for clusters of criminals and three clusters of non criminals now why three and four and usually can do something like this by clustering and then measuring the residual variance in your data so how much does one cluster explain two clusters and so on so here you can see the curves for non criminals and criminals now they claim that the optimal number of clusters here for non criminals is three which makes no sense to me like why three what you usually want to find is kind of a kink in your curve like if it's steep and then it gets flat that means that up until then your clusters actually buy you something good and from then they basically are you so if I were to guess I would divide the criminals into two clusters and the non criminals into a single cluster because that's pretty flat certainly not the crit non-criminals into three and the criminals into four that makes no sense at all like why and they say okay these are the clusters right here and these are the pictures I've shown you at the beginning what surprised the bottom ones the non criminals are smiling and the top ones aren't gee I wonder why the method works and the interesting part here is that we're how can we justify maybe how can we say if we decide on one cluster for non criminals and two clusters for criminals what does that remind us of oh

yes that is exactly how we collected the data that is exactly the fact that we collected the non criminals with one procedure and the criminals with two different procedures gee they're classified their dataset replicates exactly how they collected the data and that convinces me that it says absolutely nothing about the actual criminality of people it's just that police even if it's ID photos they don't smile and pictures on the Internet sometimes people smile the rest of the paper is pretty much garbage they did reply to critics and they kind of take issue with a number of things so first name-calling I don't mean to name-call but it's going to happen I don't get why people call them racist because it's always say doesn't know I'm no trouble and smiley ah in our experiments we did control facial expressions but not faint micro Express the critique that our methods can be reduced to a simple discriminator of smiling versus not smiling has given us a new angle of scrutiny they say what Westerners think that this is smiling but our Chinese students and colleagues even after being prompted to consider the cue of smile failed to detect the same so basically their answer is yeah you think so but we don't and then they say instead they only find the faces in the bottom row appearing somewhat more relaxed than those in the top row hmm and then here's the crucial part all criminal ID photos are government issues but not mug shots they are normal government issue ID portraits like those - driver license in the USA in contrast most of the non criminal ID style photos are taken officially by some organizations such as real estate companies law firms etc for their website you know what I'd always says when you take your picture for a government ID please don't smile imagine if you're a law firm comes to you and say we want a picture for a website please don't smile all right this was it for this paper if you like this content please consider subscribing and sharing it out this is absolute garbage and if there is important lessons to learn here namely Occam's razor is a real problem in research people often fail to criticize themselves enough and to think is there maybe a different explanation for why I'm getting the results that I'm getting and how can I disprove that is the case make sure that the effect that I'm seeing actually comes from the place where I claim it comes from I think this is a real threat throughout all of research I've seen many papers that I've reviewed that are exactly of the same fallacy not as touchy subjects as this one but it definitely exists and I remind everyone that learn a lesson from this and have a good day