In the prior post we described the roles of panelists and ad hoc reviewers in the DEB merit review process and how others have highlighted the value of taking part in this process. We left off with an observation that has popped up in several comment threads on those other discussions: “I volunteered but no one ever called.” This post addresses why that happens.

Foremost among the reasons we haven’t called you back is a possibly startling admission: we don’t have an actual “reviewer database”. We have a reviewer records system but it’s an old architecture, built for a smaller scientific enterprise[i], and lacks the sort of substantial or topical content (not even research keywords) that would be needed for it to function as a database for identifying appropriate reviewers. Even with such content, because it is primarily for record-keeping and not discovery, it would simply search reviewers we’ve used in the past and wouldn’t help us to identify “new blood.” Yes, this is incredibly, woefully behind-the-times compared to what many journals use[ii]. This is an issue for which NSF is trying to find a solution.

The next major roadblock for self-identifying reviewers is timing. DEB programs need to find large numbers of reviewers in narrow windows of time. If you’re just sending us emails whenever the thought strikes you, they are likely landing on our desks in-between when we’re looking for reviewers. You can check out the review process calendar we posted previously to see when this happens for various programs. The take-home point is that we’re often looking for panelists before proposals have even been submitted (and months before a panel meets) and we are looking for specialist reviewers during just the few weeks when we are sending out proposals for “ad hoc” reviews. Hitting those periods can help put you in the right place at the right time, but is no guarantee, simply because…

Any email is one among numerous offers being passed around daily and is likely to have passed out of memory and been supplanted by other more recent offers by the time we have a need. Suffice it to say, there’s a lot of low-information-value noise in our in-boxes: CVs without context, boilerplate introductory letters where “review” appears as an afterthought, etc.

Even with a strategically-timed, well-written introduction, however, a potential reviewer may not best match our needs for a current proposal or panel and, ultimately, we don’t have anywhere to put the information and retrieve it efficiently. Any system we put in place for keeping tabs on volunteer offers is going to be 1) competing with the whole of the internet to quickly identify sufficient numbers of relevant experts and 2) filled with dead ends unless it is regularly updated. Your CV might get dropped in a shared folder and might turn up in a document search while you’re still at the same job, but it just as easily can get buried amongst the daily deluge of emails.

While advice around the academic web to “just send in your information” reflects individual experience, it is based on a perception of cause and effect when the reality is often just coincidence. You may get a call to review from someone who never saw your email if you come up in a search result. Around the time most people are first entering the potential PI/reviewer pool, they are also developing a professional web presence. And, as we said above, without a dedicated reviewer database, we put the whole internet to work to find reviewers. So, being searchable and showing up at the right moment can make sending us any sort of introduction moot. We’ll address this further in part 3.

[i] Even just 10 years ago we were dealing with only ~half the proposals we see today.

[ii] We would like to think we are pretty adept at working around this information system deficiency. Between panelists and individual (ad hoc) reviewers, DEB manages to obtain some ~10,000 separate evaluations of proposals each year. And, taking some pride in facing this adversity, we note that lots of folks never notice our lack of a proposal & reviewer matching database until they come here and we tell them.