I started to write a short post about blind peer review, but it fragmented into a bunch of barely-related musings.
TL;DR: Something something peer review.
[A] With blind peer review, there is the danger that a referee will have heard the paper at a conference or read a draft on-line.1 On balance, I think the potential benefits of scholarly communication outweigh the potential harm of maybe breaking the seal of anonymity.
Here’s the normative situation as I see it: Authors shouldn’t hide their work from other philosophers, and they should share freely. The only expectation of secrecy is in actually submitting the paper.
So it is great to share drafts with people even though they might eventually be considered as a potential referee for the paper. If they give helpful feedback, then the paper you end up submitting will be better. Yet it would be vicious just to e-mail potential referees and tell them about the paper so that they know it’s yours.
In preparing a paper for submission, one tries to remove identifying references. It’s hard to do when one has written on the topic before, but one must try.
[B] As a referee, you’re not supposed to know who the author is. I recently agreed to review a paper, started reading, and halfway through thought that I had probably heard a distant predecessor of the paper at a conference. I couldn’t clearly remember who had presented the dimly-recalled conference paper, and I wasn’t sure this was the same author in any case. I mentioned my vague recollection to the editors when I turned in my report. The editors could decide how to weigh my fuzzy non-blindness in their decision.
I’m not convinced we should struggle to make the system more anonymous than this.
[C] In discussions of a recent hoax-paper broohaha, one point that several commentators made is that peer review is meant to catch incompetently bad papers but not fraudulent ones. Referees do not make a forensic review of the research to ensure integrity. For disciplines that rely on data, a paper that reports interesting results can often be worth publishing even if its analysis is a bit nutty.
For disciplines that do not rely on data, it is less clear what it means to be a fraudulent paper. Imagine that an author makes convincing arguments but makes them insincerely. This can make a contribution to the literature in a way that a way that presenting intriguing but falsified data cannot.
[D] I once submitted a paper to a journal that practices triple-blind review.2 Shortly after submitting, I received an invitation to referee my paper. The instructions from the journal were adamant that if I knew who the author was— even if I was the author— I should decline without comment. Saying anything to the editor might contaminate the triple-blind evaluation. So I said nothing.
Given that the instructions to potential referees covered the case in which it was the referee’s own paper, this must be a regular occurrence. I wonder what would have happened if I had agreed to be a referee and had reviewed the paper.3
- I’ve blogged about this before, in the distant past.
- This means that authors do not know who is refereeing their paper, referees are not told whose paper they are reviewing, and the editors do not know the author when they invite referees.
- It would have been too brazen to laud my own paper and recommend acceptance, so I probably would have pointed out a few shortcomings and recommended that it be revised.