# Peer Review and the NeurIPS Experiment | Two Minute Papers #84 ## Метаданные - **Канал:** Two Minute Papers - **YouTube:** https://www.youtube.com/watch?v=a1z6GXj8QK8 - **Дата:** 03.08.2016 - **Длительность:** 4:46 - **Просмотры:** 3,701 ## Описание What is peer review and how is it done? How can we check the validity of a paper? And more importantly, how can we be sure that the peer review process is fair and consistent? We'll talk about these things and how the NIPS experiment addresses them. ____________________ The NIPS experiment: http://blog.mrtz.org/2014/12/15/the-nips-experiment.html http://www.kdnuggets.com/2016/05/embrace-random-acceptance-borderline-papers.html The showcased earlier episode video: Artistic Manipulation of Caustics - https://www.youtube.com/watch?v=K-0KJtk07YU A New Publishing Model in Computer Science by Yann LeCun: http://yann.lecun.com/ex/pamphlets/publishing-models.html WE WOULD LIKE TO THANK OUR GENEROUS PATREON SUPPORTERS WHO MAKE TWO MINUTE PAPERS POSSIBLE: David Jaenisch, Sunil Kim, Julian Josephs, Daniel John Benton, Dave Rushton-Smith, Benjamin Kang. https://www.patreon.com/TwoMinutePapers We also thank Experiment for sponsoring our series. - https://experiment.com/ The thumbnail background image was created by Quinn Dombrowski (the color of the pen was changed) - https://flic.kr/p/8HRJoc The blind man icon was created by Scott de Jonge - http://www.flaticon.com/free-icon/blind-man-silhouette_8711 Subscribe if you would like to see more of these! - http://www.youtube.com/subscription_center?add_user=keeroyz Splash screen/thumbnail design: Felícia Fehér - http://felicia.hu Károly Zsolnai-Fehér's links: Facebook → https://www.facebook.com/TwoMinutePapers/ Twitter → https://twitter.com/karoly_zsolnai Web → https://cg.tuwien.ac.at/~zsolnai/ ## Содержание ### [0:00](https://www.youtube.com/watch?v=a1z6GXj8QK8) Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. We are here to answer a simple question: what is peer review? ### [0:06](https://www.youtube.com/watch?v=a1z6GXj8QK8&t=6s) What Is Peer Review Well, in science, making sure that the validity of published results is beyond doubt is of utmost importance. To this end, many scientific journals and conferences exist where researchers can submit their findings in the form of a science paper. As a condition of acceptance, these papers shall undergo extensive scrutiny by typically 2 to 5 other scientists. This refereeing process we call peer review. ### [0:36](https://www.youtube.com/watch?v=a1z6GXj8QK8&t=36s) Single Blind Reviewing Single blind reviewing means that the names of the reviewers are shrouded in mystery, but the authors of the paper are known to them. In double blind reviews, however, the papers are anonymized, and none of the parties know the names of each other. These different kinds of blind reviews were made to eliminate possible people-related biases. There is a lot of discussion whether they do a good job at that or not, but this is what they are for. After the review, if the results are found to be correct, and the reviews are favorable enough, the paper is accepted and subsequently published in a journal and/or presented at a conference. Usually, the higher the prestige of a publication venue is, the higher the likelihood of rejection, which inevitably raises a big question: how to choose the papers that are to be accepted? As we are scientists, we have to try to ensure that the peer review is a fair and consistent process. To measure if this is the case, the NIPS experiment was born. NIPS is one of the highest quality conferences in machine learning with a remarkably low acceptance ratio, which typically hovers below 25%. This is indeed remarkably low considering the fact that many of the best research groups in the world submit their finest works here. So here is the astute idea behind the NIPS experiment: a large amount of papers would ### [1:56](https://www.youtube.com/watch?v=a1z6GXj8QK8&t=116s) Behind the Nibs Experiment be secretly disseminated to multiple committees, they would review it without knowing about each other, and we would have a look whether they would accept or reject the same papers. Re-reviewing papers and see if the results is the same, if you will. At a given prescribed acceptance ratio, there was a disagreement for 57% of the papers. This means that one of the committees would accept the paper and the other wouldn't, and vice versa. Now, to put this number into perspective, the mathematical model of a random committee was put together. This means that the members of this committee have no idea what they are doing, and as a review, they basically toss up a coin and accept or reject the paper based on the result. The calculations conclude that this random committee would have this disagreement ratio of about 77%. This is hardly something to be proud of: the consistency of expert reviewers is significantly closer to a coinflip than to a hypothetical perfect review process. So, experts, 57% disagreement, Coinflip Committee, 77% disagreement. It is not as bad as the Coinflip Committee, so the question naturally arises: where are the differences? Well, it seems that the top 10% of the papers are clearly accepted by both committees, the bottom 25% of the papers are clearly rejected, this is the good news, and the bad news is that anything between might as well be decided with a cointoss. If the consistency of peer review is subject to maximization, we clearly have to do something different. Huge respect for the NIPS organizers for doing this laborious experiment, for the reviewers who did a ton of extra work, and kudos for the fact that the organizers were willing to release such uncomfortable results. This is very important, and is the only way of improving our processes. Hopefully, someday we shall have our revenge over the Coinflip Committee. Can we do something about this? What is a possible solution? Well, of course, this is a large and difficult problem for which I don't pretend to have any perfect solutions, but there is a really interesting idea by a renowned professor about crowdsourcing reviews that I found to be spectacular. I'll leave the blog post in the comments section both for this and the NIPS experiment, and we shall have an entire episode about this soon. Stay tuned! Thanks for watching, and for your generous support, and I'll see you next time! --- *Источник: https://ekstraktznaniy.ru/video/14794*