The Past Speaks

Some people seem to think that the answer to this question is yes, according to a recent New York Times article. See here.

some humanities scholars have begun to challenge the monopoly that peer review has on admission to career-making journals and, as a consequence, to the charmed circle of tenured academe. They argue that in an era of digital media there is a better way to assess the quality of work. Instead of relying on a few experts selected by leading publications, they advocate using the Internet to expose scholarly thinking to the swift collective judgment of a much broader interested audience.

The Shakespeare Quarterly, which traditionally relied on peer review to maintain quality, is planning to make the next issue of the journal open to comments from anybody. Anyone will be able to log in and comment on the articles in the journal. The identities of the commentators will be open.  See more here.

Replacing double-blind peer review [used by many science journals and the American Historical Review] and single-blind peer review [which is the norm in Canadian history] with web-based feedback from the public is an interesting concept. It is certainly being promoted by some smart people, such as Dan Cohen from the Center for History and New Media and GMU.

However, I’m not certain what to think of the idea of “open review” as the Shakespeare Quarterly has defined it. [There are other definitions of open review].   Allowing consumers of academic knowledge (e.g., history buffs who enjoy reading historical books in their spare time) as well as fellow producers to participate could be very helpful.  I check online reviews of restaurants before I go out to eat– I trust these reviews when I know that a) large numbers of people have reviewed each restaurant and have come to similar conclusions b) the reviews are by customers rather than people in the industry.  When a new restaurant has only one review online, I tend to be a bit skeptical of it– after all, that one could have been written by the mother of the restaurant owner or by one of his commercial rivals. In the former case, it would probably be very positive, in the latter very harsh and negative.

One of the problem with open reviews of academic articles is that an article on a specialized topic might attract only one or two reviewers even months after the article was published. Moreover, these reviewers might be people who are in a conflict of interest situation similar to that of the restaurant owner’s mother.

The unfiltered and semi-anonymous comments sections on websites like Amazon are open to abuse. Consider the case of Orlando Figes, a historian who criticzed other scholars on Amazon using a pseudonym.

Moreover, it takes specialized knowledge to determine whether an article is good or bad. Anyone with a tongue and a keyboard is entitled to judge a restaurant. When it comes to judging the tastiness of restaurant food, no individual’s opinion is more valid than that of anyone else. That’s why we leave the restaurant business largely to market forces. Over time, customers voting with their money will improve the quality of the food.  However, even in the restaurant world we rely on experts and well as public opinion polls to determine what is good– that’s why we send government experts to inspect the kitchens in the back.

Determining the quality of an article on Shakespeare is a bit more complex than finding out whether a burger tastes good. Only a few people know enough about Shakespeare to do it properly.

In 2006, the journal Nature launched an experiment in parallel open peer review — some articles that had been submitted to the regular anonymous peer-review process were put online for open, identified public comment, Amazon Reviews section style.  However, only 5% of authors agreed to participate in this experiment and only half of those articles received comments.  It was like an online review of a restaurant with only one review.  The knowledge that articles were simultaneously being subjected to the traditional peer review process in parallel may also have affected the willingness of scholars to provide feedback.

We sent out a total of 1,369 papers for review during the trial period. The authors of 71 (or 5%) of these agreed to their papers being displayed for open comment. Of the displayed papers, 33 received no comments, while 38 (54%) received a total of 92 technical comments. Of these comments, 49 were to 8 papers. The remaining 30 papers had comments evenly distributed. The most commented-on paper received 10 comments (an evolution paper about post-mating sexual selection). There is no obvious time bias: the papers receiving most comments were evenly spread throughout the trial, and recent papers did not show any waning of interest.

The trial received a healthy volume of online traffic: an average of 5,600 html page views per week and about the same for RSS feeds. However, this reader interest did not convert into significant numbers of comments.

See more here.

I note with interest that the strongest proponents of open reviewing seem to be in humanities rather than physical sciences or medicine. Perhaps this is because the consequences of a bad theory becoming popular in the humanities are less catastrophic than in medicine or engineering.  Humanities research often revolves around issues that are open to subjectivity, where opinions are a matter of taste or ideology, not cold hard facts. I’m not certainly if I would go to a doctor who relied on Wikipedia. We wouldn’t convene a jury of twelve citizens to determine whether a bridge was safe– we would hire a credentialed engineer. Interpreting the plays of Shakespeare seems to be an area in which subjectivity is more acceptable.

I know that the cult of the expert is unfashionable nowadays. It seems to be under assault from the postmodernist left and by people on the right (e.g., Sarah Palin populists). But it seems to be that it is an ideal worth fighting for.