This is part 3 of a discussion of reviewer service in DEB. In parts 1 and 2 we talked about the roles of panelists and ad hoc reviewers and how contacting us to volunteer as a reviewer doesn’t often work. With that as background we can provide some insight into actual reviewer selection and assignment in DEB.

Let’s mention a few caveats. First, all reviewers are chosen based on three key principles: expertise, interest, and lack of conflicts. In choosing a reviewer we strive for individuals who are not only highly qualified to evaluate the proposal, but are also very interested in the topic of the research (and therefore likely to do a thorough review) and who have no apparent conflicts of interests with the individuals and institutions involved in the proposal.

Second, every time a proposal is submitted, the PI has the opportunity to submit a list of potential reviewers (a “single copy document” which does not get seen by any reviewers). Program Officers try to use at least a few of those potential reviewers after first checking that they are appropriate experts and are not in conflict with anyone (or institution) involved in the proposal. BTW, if you want to make your Program Officer happy, when submitting a proposal include ~8 such expert reviewers, free of conflicts, and provide full names, institutions, and email addresses (only about 1 in 5 PIs submits this information).

Third, Program Officers often can think of a few highly appropriate reviewers for any given proposal they are managing simply because they know the literature and lots of scientists in their field. Still, this is rarely enough for what we need. Ideally, we want 3 ad hoc reviews for each full proposal, to inform the three panelist reviewers. Given a response rate of ~50%, that means requesting at least 6 ad hoc reviews

So how do we find and solicit reviewers’ participation? It’s a little different depending on the reviewer role: ad hoc reviewer or panelist.

Ad hoc reviewers

First, think of 5 or 6 keywords which in combination are specific to your research expertise.

We can wait….

Got ’em?

Good. Now open a new browser tab or window and navigate to your favorite academic search (e.g., Google Scholar, Scholar Universe, Web of Science, etc.) or even just a plain search engine.

Search on your combination of keywords.

Do you come up? If not, who does? Where are you? Are you even in the results? If you’re somewhere near the top, kudos! Otherwise….

This is how we find you in a nutshell. Except of course we don’t have your custom list of keywords; we use descriptions from proposals submitted by others. When we are identifying potential reviewers and panelists we’re looking for good matches. If you don’t even match your own description it’ll be hard to find you when we’re searching based on someone else’s project. Seriously, the most important thing you can do to make yourself available for review is to maintain a searchable presence on the web[i]. If you’re a post-doc or grad student and don’t have a lab but want to get started, use professional and academic social media to create a presence.

But that’s only step 1. Now we need to determine if you’re both an appropriate match and have a reasonable likelihood of actually responding to our request and completing a review.

What is behind the hyperlinks our search brought up? Boilerplate departmental faculty descriptions, publication lists that stop in 2009, old slide decks you never knew were out there, or an active lab website with descriptions of projects and recent publications? There’s a huge trust and confidentiality issue with proposal review — we’re not just going to send researchers’ best ideas, pre-publication data, etc. out to an unknown entity[ii] and hope it arrives in caring hands. You may come up at the top of the results but if those results look like an online ghost town, we’ll be skipping down to the next name on the list.

Once we’ve found an active online presence, we need to actually compare the content of the proposal and the potential reviewer’s expertise, check for conflicts, and then putting our judgment to the test ask ourselves “Will this person want to see this proposal?” If we think so, then there’s a good chance they’ll provide a thorough review or at least decline politely.

Panelists

As we mentioned above, finding panelists differs from matching individuals for ad hoc reviews. For one, the stakes are higher with a panelist. For a given set of proposals, an individual panelist plays a larger role in providing reviews than an individual ad hoc reviewer. An ad hoc review that is superficial, cursory, or goes totally off the rails can be ignored and those that never arrive have no weight at all. The same issues arising with a panelist, however, can undermine the discussion and evaluation of multiple proposals. While getting new faces onto panels is valuable to us, we also need to manage the risks of bringing in inexperienced participants.

How do we manage these risks?

Often, we look to people who have or previously had an award from us to become panelists. We’ve seen your proposal and report writing. Plus, while not required, we hope your sense of community includes giving back through review service.

We identify potential panelists through a professional network. Professional relationships actually play an important role when it comes to populating a panel. We often rely on suggestions from current reviewers and rotating Program Officers for new panelists since they both know the requirements of review service and have knowledge of different professional circles.

We first offer opportunities to gain experience through ad hoc review and service on smaller panels. Especially for early career faculty, panels outside of the core programs, like DDIGs, are often used to provide an introduction to reviewership.

Since we don’t have a database of reviewer/panelist expertise, each cluster in DEB has to rely on keeping and sharing notes to maintain institutional knowledge of the panelist pool: who has served, wants to serve, and might be able to provide needed expertise at specific future dates? Most members of a panel come out of this pool in response to a broad invitation noting the dates of upcoming panels. Of those available to serve, volunteers are selected based on the expected needs of the panel. This is often done well in advance of the proposal submission deadline and includes a few “swing” spots to round out the assembled expertise with later invitations. Those swing spots are where new people often enter the process, but we’re filling specific needs and so finding these folks more-or-less follows the same path as finding ad hoc reviewers.

So what can you do to become involved in review?

Drawing from the across this and the previous posts, we offer the following suggestions, in order of importance:

1) Make it easy for us to find you online in connection with your scientific work.

2) If you have no NSF review experience, talk to colleagues who are already involved in review and ask them to suggest you as a reviewer.

3) Recognize that, if you’re going to reach out to us, there’s not much we can do with your information if it’s not an immediate match to a specific need.

4) Say yes when you can, since more opportunities often follow. It’s ok if you can’t take on a review when we ask; tell us — we understand. But, please don’t ignore the request or we will get the idea that you have no interest in review.

 

[i] While we don’t ask you or your close collaborators to review your proposal, seeing those names (yours and theirs) in search results is a good sign.

[ii] With respect to having served within the NSF review system.