For a while now we’ve been pondering how to approach a delicate subject: writing reviews. This subject is something of a minefield of tensions, conflicts, opinions, and opportunities to offend, alienate, and otherwise ruffle feathers by implying, “you’re doing it wrong.” So we feel it is important to explain here why we are tackling this topic and mention some of the approaches that didn’t fly.

We receive requests from less experienced reviewers who want advice outside the trial-by-fire of an actual panel to hone their review-writing skills. And we also hear from PIs who are disappointed in the utility or quality of the reviews they have received.

To set the tone, let’s make a couple of blanket statements straight off. You are all volunteers in this effort and you do a great job and deserve immense gratitude. With that said, probably every established PI who has submitted NSF proposals has received at least one review that was… a bit less than they expected; perhaps lacking a certain degree of usefulness; or even a complete enigma.

What is going on with these sub-par reviews? Well, it’s difficult to address because we don’t think there is a singular problem. Simply put, in managing the whirlwind of research life, we may occasionally fall short in creating the ideal, critical, insightful and helpful proposal review.

Why are we addressing this here? There is quite a bit out there about writing manuscript reviews, and even some nice posts about PIs coming to grips with the reviews received. But there’s a pretty big void when it comes to discussion of actually writing proposal reviews. As a program office, we have a unique platform to start a discussion around review expectations and feel it is a topic worth talking about openly. What we hope to achieve is not a static prescription for a “good review” but for this to start a discussion, raising community awareness about the importance of proposal reviews and the need for continual improvement in writing them, regardless of how well we think we’re doing already.

Some things we ARE NOT going to do here:

• We are not going to provide you with real review examples. That would break confidentiality; even with names stripped someone might recognize the writing enough to correctly (or worse, incorrectly) ascribe the writing to an individual with unintended consequences.
• We are not going to provide you with fictional examples. There are many different ways that a review can be good or bad in the broad sense but it’s not clear that an example is needed to make the point. And, the finer points of a review often need the context of the full proposal to make sense and would therefore be lost in such an example.
• We are not seeking to create a specific template for a written review. That already exists at a very basic level in the form you access via the FastLane review module. Different directorates, divisions, programs, and specific solicitations establish and enforce varied requirements and norms for reviews under the broad scope of the same FastLane form. There is no universal, detailed guide to review expectations and we don’t think it would be possible or wise to attempt one. Even within DEB, a single template would not work for all of our solicitations.

There are many paths to a good proposal review. But the major commonalities can be distilled into a few points.

Good proposal reviews:

• are conceived with an open mind and reflect serious consideration of the proposal
• make a compelling case for the evaluation, citing evidence or examples as needed
• are well-written and direct
• contain no ad hominem criticisms or unsupported assertions

Of course the real challenge is, “How do I do it?” To help out we’ve started a list (in step-wise order[i]) of tips to guide you past the most common pitfalls on the path to completing a great review.

1. Refresh your expectations, EVERY TIME: You are asked to judge a proposal against a specific set of review criteria. These review criteria are not static. They change over time; some change almost yearly and may have changed since you last looked. They vary by organization; each successive layer of directorates, divisions, and programs has an opportunity to modify and specify review criteria; what you are asked to do in education or geosciences will differ in the specifics from what you are asked to do in DEB even though all still abide the NSF merit review criteria[ii]. They can be stacked and combined; even within a single panel some proposals will include additional review criteria relevant to a submission option or solicitation on top of those of the conceptual program at the heart of the proposal.

The best way to keep on top of this shifting body of expectations is to READ THE MATERIAL WE SEND YOU when you volunteer to review. Googling and looking through your own files can lead you to outdated information. In DEB, we send you a nicely packaged summary with the latest information and our interpretation of specific criteria all in one place with links to the source documents should you desire to read more.

2. Recognize your audiences (yes, plural): You, as a reviewer are always writing for two, and potentially three, audiences. Firstly, the review is to inform us (NSF) as to why you think the proposal is or is not worth funding; we consider this in light of others’ reviews and our award portfolio to arrive at a funding recommendation. Secondly, the review is to help the PI learn how to prepare a better proposal next time, or to improve the project if it is funded. Lastly, the written review may be used as documentary evidence that you as the reviewer gave the proposal thorough and thoughtful consideration.
3. Don’t summarize, review: Every audience for the review will also have access to the actual proposal and doesn’t need you to summarize it; we need your well-argued opinion of it. Note: Starting out with largely descriptive text may be important to organizing your thoughts as a reviewer. That’s ok. That’s great. But it’s not the end point. What we’re saying here is: take whatever part of the proposal you felt important enough to describe and fill it out by giving your opinion of it and telling us why you hold that opinion.
4. Provide clear and substantiated comments: To satisfy the distinct audiences and avoid summarizing, every point you make in your review should exhibit these 3 characteristics:
   1. It is EVALUATIVE
   2. It is SUPPORTED with appropriate details, evidence, and comparisons
   3. It provides CONSTRUCTIVE feedback in the context of the proposal

Notes: EVALUATIVE means using terms that express an opinion about the subject like “good”, “bad”, “excellent”, “inadequate”, “exemplary”, “satisfactory”, etc. SUPPORTED means following the evaluative term with an explanation like “because the [specific information in the proposal] is/is not reflective of [some important external knowledge like cited literature, best practice, or other relevant information].”

5. Be self-critical of your critical comments: A good critical comment delves below the surface of your initial reaction and constructively reflects the opportunities and constraints for addressing the issue. This step could easily be its own separate post; we’ll provide a longer discussion of this at a later date. For now though, the fast summary is: see the flaws, point them out but give them context, and don’t get hung up on small stuff. Stop and ask: why is it a flaw for this proposal? Do I fault others for this consistently? If not, why have I pointed it out now? Is there a deeper cause of this flaw that I can describe? (If so, do it.) Search out your biases. Recognize and differentiate between actual problems in the proposal and legitimate differences from how you would do the work. Make sure your suggestions for improvement are clear and reflect the constraints on space: it’s limited, and PIs can’t include “more” material on X without cutting Y. A useful suggestion needs to identify both the X that needs expansion and the Y that could stand to be cut.
6. Minimize or omit purely descriptive text: This is a repeat of point 3. We’re serious about this. It is very easy to fall into the summarizing trap – it’s much easier to write that material – and we have to remind ourselves about this constantly too. If some aspect of the proposal is important enough for you to describe, tell us how it affects your overall opinion of the proposal and why.
7. Use the FastLane review form boxes: The FastLane form has fields for your review of Intellectual Merit and Broader Impacts, and for a Summary. Use the first two to make your points (large and small, positive and negative) about the proposal. Use the summary box to explain how those points come together to form your overall opinion – reiterate the major deciding factors, explicitly note where you discount minor issues. This provides clarity for both NSF and the PI as to how the other comments translate into a single overall rating.

What other advice might you as a PI receiving a review want to give the reviewers? Is there a Golden Rule for review? If so, how could it be phrased?

Maybe some of our readers would be interested in trying to improvise a few “model review” lines, give it a go in the comments.

[i] Based on our general perception of where we get the most questions/complaints, tips 1, 4, 5, and 3 encompass the largest areas for improvement.

[ii] The current NSF merit review criteria are 2-fold (Intellectual Merit and Broader Impacts), under each criterion NSF poses 5 questions to guide reviewers’ thinking (but does not require explicit answers to each):

1. What is the potential for the proposed activity to:
2. Advance knowledge and understanding within its own field or across different fields (for Intellectual Merit); or
3. Benefit society or advance desired societal outcomes (for Broader Impacts)?
4. To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts?
5. Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale?
6. How well qualified is the individual, team, or organization to conduct the proposed activities?
7. Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities?