Finding the right incentives for reviewers

Reading the Times Higher Education article on the struggles of finding expert reviewers for grant proposals brings to mind the same old problems facing academic publications and managing editors.

"We are teetering on the brink of the system breaking down," said Nigel Brown, former director of science and technology at the Biotechnology and Biological Sciences Research Council and now vice-principal of the University of Edinburgh. "It is not impossible, but it is getting quite difficult. People are under a lot of pressure in their day jobs, and it is just easier to say 'no'."

I've personally been involved in two distinct approaches to acquiring reviews: the more traditional non-payment approach while working with an association publication, and the pay-for-review approach while working in the CME industry. My impression (and this is unscientific, since I have no data to supply at this time) was that neither approach had markedly different results. The amount of the payment could of course be a significant variable, but we had inconsistent responses from reviewers regardless of the incentive offered.

The inevitable outcome in both scenarios is that the initial expansive list of potential reviewers is quickly whittled down to a core group of individuals who perform the lion's share of all reviews. In the association model, reviewers quite often came from a list of associate editors, who had agreed upon accepting the title that they would provide x number of reviews per year. However, it became clear that this obligation was tenuous at best. Further complicating matters, the association was in a highly-specialized field, greatly limiting the pool of prospective reviewers. In fact, in 2004 a paper was submitted on a topic so narrowly-focused that every potential reviewer was already listed as a co-author.

Our recourse was to drop and add associate editors as a measure of performance and dedication, although this was never explicitly communicated. A core group of individuals remained on the associate editor list year in and year out, while those who had accepted the title with little expectation of having to actually contribute were unceremoniously removed form the masthead.

In the CME scenario, in which experts in a number of fields were offered financial incentive to review, the response was perhaps much worse. The amount offered was adjusted according to the reputation of the potential reviewer, as well as our level of desperation at the time. It was commonplace to have a reviewer accept the call, only to change their minds as the review due date approached. The fact that the review process became a financial, rather than academic, transaction seemed to change the sense of duty the reviewer might possess. Indeed once the review became simply a matter of money, I got the impression that its value was now in question, "Is it really worth $500 to complete this review?" Non-paid reviewers at least appeared to perceive the review process in entirely philosophical terms.

When dealing with delinquent reviewers, the only tool in my arsenal that worked with above average results was a naked appeal to the reviewer's sense of pride (read: ego): "Your expertise in this area is essential to our editorial process..."