Many of you are probably wondering – what are the steps that my proposal goes through as it gets reviewed?  What does a typical year in DEB look like to a proposal?  Some of you may have attended an outreach presentation by a Program Officer in which there was a slide showing the preliminary proposal/full proposal cycle. The slide image looked something like this:

[i]

This is the review schedule for DEB Core Programs. There’s quite a lot of white space in there, – enough to make you wonder why review takes as long as it does.  This post (part 1 of 2) aims to address a very basic but common question we receive about the review process.

What takes so long in the first place?

Creating a funding opportunity, receiving and evaluating proposals, and making funding recommendations is not simple. The process begins long before the proposals arrive at NSF and extends well after the panel meets. Below is a general outline for the review process as practiced in DEB, focused around the panel meeting for a given funding opportunity.

8-12 months (or longer) before a panel: A funding opportunity is developed internally, then announced or updated, alerting the community to the submission requirements, and setting a closing date for receipt of proposals. Especially for new solicitations, multiple layers of review and approval are necessary within NSF.

6 months before a panel: NSF staff make estimates about proposal load based on historical performance, determine how many panels will likely be needed, find space for meetings and set panel dates (yes, this really needs to be done 6 months in advance to book a suitable room at NSF). As soon as panel dates are confirmed, program officers begin contacting and soliciting panelists based on those estimates, expected needs for reviewer expertise, and our intent to provide a demographically balanced group.

3-4 months before a panel: Proposal submissions are received (most in the last 24hrs before the due date). The next week or two is spent reviewing the submissions for compliance with the requirements of the funding opportunity and its conceptual fit to the program. This requires hands-on review of every individual proposal jacket.  Depending on the specific issues, non-compliant submissions are fixed or returned without review or offered transfer to a program of better fit.

2.5-3.5 months before a panel: If the review process includes ad hoc reviews (individual written reviews by persons not serving on the panel) in addition to panelist reviews, programs spend several weeks (overlapping somewhat with compliance checking) identifying potential reviewers, soliciting reviews, following-up with unresponsive reviewers, identifying and soliciting more reviewers, and eventually obtaining needed reviews.  Additional panelists are identified and invited based on the actual number and composition of proposals accepted for review. Panelists receive emails confirming panel participation.

2.5-2 months before a panel: Nearly all panelists have been confirmed for the panel. Panelists receive a series of emails with instructions for:

• Registering in the panel review system (through FastLane), arranging reimbursements and travel, and completing any paperwork which must be sent to NSF.
• Identifying Conflicts of Interest (COI) and preferences for review assignments from the proposal list.

Note: Instructions are sent over the course of several emails because of the process to protect review confidentiality which requires obtaining documentation before sharing proposal and reviewer details.

1.5-1 month before a panel: Based on the identified preferences and avoidance of conflicts of interest, individual proposal assignments are sent to panelists with instructions for writing individual reviews.  Once assigned reviews are received in FastLane, panelists can see reviews received from other assigned panelists.

Note: Program officers do their best to assign each panelist to her/his preferred proposals. However, sometimes fewer than 3 panelists indicate a preference for a proposal and sometimes everyone picks the same proposal. When too few indicate a preference for a proposal, program officers assign panelists to a proposal that calls upon the panelist’s breadth knowledge. Every proposal should be written so that a non-specialist scientist can understand and appreciate the proposed work. 

1 month before a panel: Panelists are completing and submitting reviews.  A reminder email is sent to panelists with the panel agenda and to ensure logistical steps noted in previous emails are complete.

~1 day before panel: All reviews are made available to non-assigned panelists (except COIs). Panelists arrive in DC area.

Panel (2-4 days): Panelists arrive at panel site and are introduced to the process and schedule.  The panel discusses proposals, rates proposals and writes panel discussion summaries. At the end of the first day, a significant portion of on-site panelists traditionally adjourn to a local restaurant for a group dinner where they talk about anything except the proposals. Discussion, rating and writing continues until complete. Panelists are given opportunity to revisit and reconsider any ratings before recommendations are recorded as final. In most cases a representative from the Assistant Director’s office visits the panel during a break for discussion about the panel experience and open Q&A.  Once the final summaries are approved, the panel adjourns and panelists depart for home.

0-0.5 months post-panel: Program Officers discuss panel advice, report to Division management on the panel outcomes and develop priorities for program recommendations.

0.5-1.5 months post-panel: Program recommendations are individually documented for each decision and reviewed for completeness. Division management provides a sign-off on the recommendation, completing the scientific merit review phase and passing the documentation to grant administration for final processing Review materials are released to the PIs. Program Officers inform PIs of the recommendations.  (Note: that we, the reviewing program, aren’t the final step is a crucial point we’ll come back to in a follow-on blog post).

Beyond: The large majority of PIs will be notified within 1-2 months post-panel whether their proposal has been declined or recommended for approval. If, however, the funding levels are uncertain when the panel ends, programs cannot decide exactly how many of their priority proposals can be funded. While most of the definite declines have been processed as well as many of the definite awards, Program Officers will contact a small number of PIs to let them know final program recommendations may be delayed until more concrete funding figures are available. In this way for most programs the “post-panel” phase can be extended well beyond the original intended completion date. We strive to have 75% of the definite decisions done in 6 months from submission.

When applied to the specific instance of the DEB Core Programs, the detailed timeline looks something like this:

Common modifiers:

Proposal volume greatly affects the time it takes to complete review. The major limiting steps are those that require individual attention to each proposal from a small program staff: checking for compliance, identifying and soliciting qualified but non-conflicted reviewers, analyzing reviews and preparing recommendations. There’s a big difference between how quickly we can process 150 versus 1500 proposals.

The length of several steps in this process can be shortened or lengthened somewhat based on the interactions with the reviewers (both ad hoc and panelist). Pre-proposals are only reviewed by the panel, which allows for a quicker decision process. Also, if the composition of the proposals can be predicted, panelists can be invited and have much of their paperwork processed earlier, shortening the process or opening a wider window for completing reviews.  However, failure of reviewers to respond to communications or complete steps can delay or slow the review process.

Summary

The big takeaway here is that with a small program streamlined to a minimal amount of panel review and a taking a dedicated and well-staffed brute force approach to all the steps requiring individual attention, a panel review process can be completed in approximately 4 months. But, differences in the responsiveness of reviewers, and the number of proposals submitted can only extend that process. Fully streamlined, the preliminary proposals take a bit over 4 months from receipt to recommendation, driven largely by volume (the shorter 4-page format helps – this  isn’t possible with a similar volume of 15-page proposals). This is also reflected in the full proposals cycle where fewer, longer proposals and a review process that requires a longer time (soliciting ad hocs), winds up taking longer.

Hopefully, that makes some sense. But why are the panels in April and October with due dates in January and August? We’ll discuss that in the next post.