Google AI's Take on How To Fix Peer Review

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. It is time for a position paper. This paper does not have the usual visual fireworks that you see in many of these videos, however, it addresses the cornerstone of scientific publication, which is none other than peer review. When a research group is done with a project, they don’t just write up the results and chuck the paper into a repository, but instead, they submit it to a scientific venue, for instance, a journal or a conference. Then, the venue finds several other researchers who are willing to go through the work with a fine-tooth comb. In the case of double-blind reviews, both the authors and the reviewers remain anonymous to each other. The reviewers now check whether the results are indeed significant, novel, credible and reproducible. If the venue is really good, this process is very tough and thorough, and this is process becomes the scientific version of beating the heck out of someone, but in a constructive manner. If the work is able to withstand serious criticism, and ticks the required boxes, it can proceed to get published at this venue. Otherwise, it is rejected. So what we heard so far is that the research work is being reviewed, however, scientists at the Google AI lab raised the issue that the reviewers themselves should also be reviewed. Consider the fact that all scientists are expected to spend a certain percentage of their time to serve the greater good. For instance, throughout my PhD studies, I have reviewed over 30 papers and I am not even done yet. These paper reviews take place without compensation. Let’s call this issue number one for now. Issue number two is the explosive growth of the number of submissions over time at the most prestigious machine learning and computer vision conferences. Have a look here. It is of utmost importance that we create a review system that is as fair as possible - after all, thousands of hours spent on research projects are at stake. Add these two issues together, and we get a system where the average quality of the reviews will almost certainly decrease over time. Quoting the authors: “We believe the key issues here are structural. Reviewers donate their valuable time and expertise anonymously as a service to the community with no compensation or attribution, are increasingly taxed by a rapidly increasing number of submissions, and are held to no enforced standards. ” In Two Minute Papers episode number 84, so more than 200 episodes ago, we discussed the NeurIPS experiment. Leave a comment if you’ve been around back then and you enjoyed Two Minute Papers before it was cool! But don’t worry if this is not the case, this was long ago, so here is a short summary: a large amount of papers were secretly disseminated to multiple committees, who would review it without knowing about each other, and we would have a look whether they would accept or reject the same papers. Re-review papers and see if the results are the same, if you will. If we use sophisticated mathematics to create new scientific methods, why not use mathematics to evaluate our own processes? So, after doing that, it was found that at a given prescribed acceptance ratio, there was a disagreement for 57% of the papers. So, is this number good or bad? Let’s imagine a completely hypothetical committee that has no idea what they are doing, and as a review, they basically toss up a coin and accept or reject the paper based on the result of the cointoss. Let’s call them the Coinflip Committee. The calculations conclude that the Coinflip Committee would have a disagreement ratio of about 77%. So, experts, 57% disagreement, Coinflip Committee, 77% disagreement. And now, to answer whether this is good or bad: this is hardly something to be proud of — the consistency of expert reviewers is significantly closer to a coinflip than to a hypothetical perfect review process. If that is not an indication that we have to do something about this, I am not sure what is. So, in this paper, the authors propose two important changes to the system to remedy these issues: Remedy number one - they propose a rubric, a 7-point document to evaluate the quality of the reviews. Again, not only the papers are reviewed, but the reviews themselves. It is similar to the ones used in public schools to evaluate student performance to make sure whether the review was objective, consistent and fair. Remedy number two - reviewers should be incentivized and rewarded for their work. The authors argue that a professional service should be worthy of professional compensation.

Now, of course, this sounds great, but this also requires money. Where should the funds come from? The paper discusses several options: for instance, this could be funded through sponsorships, or, asking for a reasonable fee when submitting a paper for peer review, and introducing a new fee structure for science conferences. This is a short, 5-page paper that is very easily readable for everyone, raises excellent points for a very important problem, so needless to say, I highly recommend that you give it a read, as always, the link is in the video description. I hope this video will help raising more awareness to this problem. If we are to create a fair system for evaluating research papers, we better get this right. Thanks for watching and for your generous support, and I'll see you next time!