DEB Numbers: FY 2016 Wrap-Up

DEB Numbers: FY 2016 Wrap-Up


Fiscal year 2016 officially closed out on September 30. Now that we are past our panels in October and early November, we have a chance to look back and report on the DEB Core Program merit review and funding outcomes for FY 2016.

This post follows the format we’ve used in previous years. For a refresher, and lengthier discussions of the hows and whys of the metrics, you can visit the 2015,  2014, and 2013 numbers.

Read on to see how 2016 compares. Continue reading

DEB & IOS Preliminary Proposal System Evaluation Update


Our independent evaluation contractor, Abt, closed the PI and Reviewer surveys in mid-November.

At present, they are working on analyzing the survey results. These results will be brought together with analyses of stakeholder interviews and programmatic (proposal, award, and review process) data that Abt has already completed to produce a full evaluation.

We are expecting the final report to be delivered to NSF by the end of February, 2017. We are looking forward to sharing the results with you as we are able.

And, to the many of you who were contacted by Abt to take part in the survey, thank you for your time and participation.

Fall 2016 DEB Panels status: “When will I have a decision?” edition


DEB’s full proposal panels finished in early November (for those full proposals submitted back in July and August). So, when will you receive review results?

Some of you may have already heard from us. Others will be hearing “soon” (as detailed below).

Right now, all of our programs have synthesized the recommendations of their panels, considered their portfolios, and come up with their planned award and decline recommendations. These are then documented, sent through administrative review, and finally signed off, “concurred,” by the head or deputy for the Division.

DEB’s first priority is processing the decline notices. We’re trying to get your reviews back to you to provide as much time as possible to consider your options for January pre-proposal submissions.

For potential awards, it’s a bit more complicated. We expect award recommendation dates to be later this year than typical. At present, NSF is operating under a temporary budget measure, called a Continuing Resolution (or CR). The current CR runs through December 9, 2016. We won’t have significant funds available to cover new grants until a longer-term funding measure is enacted.

So, while we have a prioritized list of award recommendations, we don’t yet have the funds needed to take action on those recommendations. Moreover, we don’t know how much funding we’ll actually have available so uncertainty is part of the plan. Thus, between “definite award recommendation” and “definite decline recommendation” we have a recommendation gray zone.

How are we handling this?

If your proposal fell into the definite decline group, then you’ll be getting an official notice from DEB. Once the formal decline recommendation is approved, the system updates the proposal status in FastLane and queues up a notification email. We are planning to have all declines approved by December 20, 2016. Note: our IT system sends the notification emails in batches at the end of the day[i]. Thus, if you are frequently refreshing FastLane you will likely see the news there before you get a letter from us.

If your proposal fell into the definite award group or the gray zone, you will first be getting a call or email from your Program Officer. They will be letting you know what the plan is for your particular proposal and how you can get things ready (e.g., submitting budget revisions or abstract language) for an eventual award. Formal action, including the release of reviews, cannot happen until we have funding available. However, folks in this group should also hear from their Program Officers by December 20.

After December 20, if you have not received any communication from us, first check your spam folder and then look up your proposal number and give us a call. But please remember, the lead PI for a proposal or collaborative group is the designated point of contact; if you’re a co-PI you need to get in touch with the lead PI and have them inquire.


[i] We’re not totally sure why this is, but suspect it has to do with email traffic volume and security features: discriminating an intentional batch of emails from an account taken over by a bot.

Program Announcement: DEB Core Programs & LTREB Solicitations Updates


Updated guidelines are now available for submissions under the two-stage DEB preliminary/full proposal system. Both DEB Core Programs and Long Term Research in Environmental Biology (LTREB) have been updated.

The new DEB Core Programs publication is NSF 17-512[i].

The new LTREB publication is NSF 17-513[ii].

Please read these guidelines if you plan to submit a preliminary proposal.

In this post, we’re providing a brief summary of the notable points and key changes, but this is not sufficient information to complete a submission.

Both solicitations

  • The definition of “Eligible Institutions” has been updated with limits on the eligible institution types. Institution types that do not meet this definition remain eligible as sub-awardees, but cannot be the primary grant recipient.
  • The deadline for submitting the Personnel List Spreadsheet (from a template, submitted by email) has been reduced to 1 business day (from 3 days) after the proposal deadline for both preliminary and full proposals.
  • The purpose and procedures for requesting a full proposal deferral have been updated and clarified.
  • The requirement for full proposals to provide results of prior NSF support has been clarified and emphasized.
  • The guidelines for Letters of Collaboration (to confirm cooperation or involvement of persons or organizations not receiving funding under the proposal) have been updated to clarify the purpose of, and limits on, such letters.

DEB Core Programs

  • The Core Programs solicitation now includes instructions for submission of international collaborative proposals involving eligible collaborators in the UK (via NERC) or Israel (via BSF). These instructions continue the partnerships originally advertised as Dear Colleague Letters.
  • The budget cap for the small grants (SG) option has been increased to $200,000.

LTREB

  • The Project Description page limit for RENEWAL proposals has been increased from 8 to 10 pages.

Changes Beyond the DEB Solicitations

IOS

Many of our PIs have research interests that overlap between DEB and the Division of Integrative Organismal Systems (IOS). New submission guidelines for the preliminary proposal system in IOS have also been published as NSF 17-508. Check with IOS and the IOS Blog for additional information.

NSF-wide

Please take note that the NSF general proposal guidelines have also been revised. This information is provided in the NSF Proposal & Award Policies & Procedures Guide (PAPPG), which previously comprised two publications known as the Grant Proposal Guide (GPG) and Award & Administration Guide (AAG). The new version of the PAPPG, is a single consolidated guide:  NSF 17-1. The guidelines in PAPPG 17-1 apply for proposals submitted or due, or awards made, on or after January 30, 2017. This document contains the full set of general guidelines to PIs, including everything from proposal preparation to award reporting and close-out.

A summary explanation of the new PAPPG format and changes from the previous edition of the guide can be read here: https://www.nsf.gov/pubs/policydocs/pappg17_1/sigchanges.jsp

These revisions have minimal effect on the requirements for the upcoming DEB preliminary proposal deadline (since the PAPPG comes into force on Jan 30, 2017 – a week after the pre-proposal deadline).

The guidelines in PAPPG 17-1 will apply for invited full proposals (due next August), and other proposals you may be planning to submit to DEB or other NSF programs.

For instance, starting on Jan, 30 2017 any RAPID or EAGER proposals intended for DEB would list the NSF 17-1 PAPPG program announcement number on the proposal cover page.


[i] The old solicitation NSF 15-609 is no longer accepting new proposals.

[ii] The old solicitation NSF 16-500 is no longer accepting new proposals.

Preliminary Proposal Evaluation Survey Reminder


TL;DR

Check your inbox.

Check your spam folder.

Complete the survey!

End the reminder messages.

 

Background (if the above doesn’t make sense to you).

This is about the Preliminary Proposal system in use in both NSF BIO’s Division of Environmental Biology and Division of Integrative Organismal Systems.

We are in the midst of an external evaluation of the effects of this system on the merit review process.

We posted an initial notification letter about stakeholder surveys. And, copies of this letter were sent out to everyone in the sample ahead of the formal invitations.

The formal survey invitations with the active survey links were sent out by mid-September from the evaluator, Abt Associates.

Reminder emails are also coming out and will continue to do so at regular interviews while the survey remains open and incomplete.

If you have been receiving these messages, please complete the survey. If your colleagues have been receiving these messages and have not completed the survey, encourage them to do so.

If you received an invitation to take the survey,

  • Please take the 10 or so minutes to register your responses via the link in the email.
  • Remember that these are single-use individualized links.
  • Your response matters. This isn’t a census: your invitation is part of a stratified random sample selected for inference to the population.

Thank you for your participation!

DEB Numbers: Historical Proposal Loads


Last spring we posted on the per-person success rate and pointed out several interesting findings based on a decade of DEB data. We were seeing a lot of new PIs and, conversely, a lot of PIs who never returned after their first shot. And, the vast majority of PIs who managed to obtain funding are not continuously funded.

This post is a short follow-up to take a bigger picture look at submission rates.

Since preliminary proposals entered the scene, DEB really hasn’t seen much change in the submission pattern: 75% of PIs in any year submit one preliminary proposal and the other 25% submit two (and a small number submit three ideas in a year, if one also counts full proposals to special programs).

Before the preliminary proposals were launched, we ran some numbers on how often people tended to submit. The results were that, in the years immediately prior to preliminary proposals (~2008-2011), around 75% of PIs in a year were on a single proposal submission (25% on two or more). Fewer than 5% of PIs submitted more than two proposals in a year. Further, most PIs didn’t return to submit proposals year after year (either new ideas or re-working of prior submissions); skipping a year or two between submissions was typical. These data conflicted with the perceptions and anecdotes that “everyone” submitted several proposals every year and were increasing their submission intensity. Although recent data don’t support those perceptions, we still wondered if there might be a kernel of truth to be found on a longer time scale. What is the bigger picture of history of proposal load and submission behavior across BIO?

Well, with some digging we were able to put together a data set that lets us take a look at full proposal research grant submissions across BIO, going all the way back to 1991 when, it seems, the NSF started computerized record-keeping. Looking at this bigger picture of submissions, we can see when changes have occurred and how they fit into the broader narrative of the changing funding environment.

Total BIO full research grant submissions per year (line, right axis) and proportions of individuals submitting 1, 2, 3, 4, 5, or more proposals each calendar year from 1991 to 2014. (Note: 2015 is excluded because proposals submitted in calendar year 2015 are still being processed at the time of writing.)

Total BIO full research grant submissions per year (line, right axis) and proportions of individuals submitting 1, 2, 3, 4, 5, or more proposals each calendar year from 1991 to 2014. (Note: 2015 is excluded because proposals submitted in calendar year 2015 are still being processed at the time of writing.)

 

1990s: Throughout the 1990s BIO received about 4000 proposals per year. This period of relative stability represents the baseline for more than a decade of subsequent discussions of increasing proposal pressure. Interestingly, the proportion of people submitting two or more proposals each year grew over this period, but without seeming to affect total proposal load; this could result from either increasing collaboration (something we’ve seen) or a shrinking PI pool (something we haven’t seen). At this time NSF used a paper-based process, so the cost and effort to prepare a proposal was quite high. Then….

2000s: In 2000, FastLane became fully operational and everyone switched to electronic submission. BIO also saw the launch of special programs in the new Emerging Frontiers division. In a single year, it became easier to submit a proposal and there were more deadlines and target dates to which one could potentially submit. The new electronic submission mechanism and new opportunities likely both contributed to increased submissions in subsequent years.

Following the switch to FastLane, from 2001 to 2005, total annual submissions grew to about 50% above the 1990s average and stayed there for a few years. This period of growth also coincided with an increasing proportion of people submitting 2+ proposals. Increasing numbers of proposals per person had only a limited effect on the total proposal load because of continued growth in collaboration (increasing PIs per proposal). Instead, the major driver of proposal increases was the increasing number of people submitting proposals. This situation was not unique to BIO.

This period from 2001 to 2005 was the rapid growth that sparked widespread discussion in the scientific community of overburdening of the system and threats to the quality of merit review, as summarized in the 2007 IPAMM report.

Eventually, however, the community experienced a declining success rate because BIO budgets did not go up in any way to match the 50% increase in proposal submissions. From 2005-2008 submissions/person seemed to stabilize and submissions peaked in 2006. We interpret this as a shift in behavior in response to decreasing returns for proposal effort (a rebalancing of the effort/benefit ratio for submissions). It would have been interesting to see if this held, but….

2009/2010: In 2009 and 2010, BIO was up another ~1000 proposals over 2006, reaching an all-time high of nearly 7000 proposal submissions. These were the years of ARRA, the economic stimulus package. Even though NSF was very clear that almost all stimulus funding would go toward funding proposals that had been already reviewed (from 2008) and that we wouldn’t otherwise be able to afford, there was a clear reaction from the community. It appears that the idea of more money (or less competition) created a perception that the effort/benefit relationship may have changed, leading to more proposals.

2011: We see a drop in 2011. It is plausible that this was the realization that the ARRA money really was a one-time deal, there were still many more good proposals than could be funded, and that obtaining funding hadn’t suddenly become easier. As a result, the effort/benefit dynamic could be shifting back; or, this could’ve been a one-time off year. We can’t know for sure because…

2012: Starting in 2012 IOS and DEB, the two largest Divisions in BIO, switched to a system of preliminary proposals  to provide a first-pass screening of projects (preliminary proposals are not counted in the chart). This effectively restricted the number of full proposals in the two largest competitions in BIO such that in 2012, 2013, and 2014 the full proposal load across BIO dropped below 5000 proposals per year (down 2000 proposals from the 2010 peak). The proportion of individuals submitting 2+ full proposals per year also dropped, consistent with the submission limits imposed in DEB, IOS, and MCB. PIs now submitting multiple full proposals to BIO in a given year are generally submitting to multiple programs (core program and special program) or multiple Divisions (DEB and [IOS or MCB or EF or DBI]) and diversifying their submission portfolios.

In summary, the introduction of online and multi-institutional submissions via FastLane kicked off a decade of change marked by growth in proposal submissions and per-PI submissions to BIO. The response, a switch to preliminary proposals in IOS and DEB, caused a major (~1/3) reduction in full proposals and also a shift in the proportion of individuals submitting multiple proposals each year. In essence, the pattern of proposal submission in BIO has shifted back to what it was like in the early 2000s. However, even with these reductions, it is still a more competitive context than the 1990s baseline, prior to online submissions via FastLane.

Spring 2016: DEB Preliminary Proposal Results


Notices

All PIs should have received notice of the results of your 2016 DEB Core Program preliminary proposals by now. Full proposal invitation notices were all sent out by the first week of May (ahead of schedule), giving those invited PIs a solid three months to prepare their full proposals. ‘Do Not Invite’ decisions began going out immediately thereafter and throughout the rest of May.

If you haven’t heard, go to fastlane.nsf.gov and log in. Then, select the options for “proposal functions” then “proposal status.” This should bring up your proposal info. If you were a Co-PI, check with the lead PI on your proposal: that person is designated to receive all of the notifications related to the submission.

If you are the lead PI and still have not heard anything AND do not see an updated proposal status in FastLane, then email your Program Officer/Program Director. Be sure to include the seven-digit proposal ID number of your submission in the message.

Process

All told, DEB took 1474 preliminary proposals to 10 panels during March and April of 2016. A big thank you to all of the panelists who served and provided much thoughtful discussion and reasoned recommendations. Note: if you’re interested in hearing a first-hand account of the DEB preliminary proposal panel process, check out this great post by Mike Kaspari.

Panelists received review assignments several weeks prior to the panels and prepared individual written reviews and individual scores. During the panel, each proposal was discussed by the assigned panelists and then presented to the entire panel for additional discussion and assignment to a rating category. Panels were presented two recommendation options for each preliminary proposal: Invite or Do Not Invite. Following discussion, the assigned panelists prepared a panel summary statement to synthesize the key points of the panel discussion and rationale for the assigned rating.

Both the individual written reviews and the panel summary statement are released to the PI of the preliminary proposal.

As we’ve discussed previously, the final decisions on the preliminary proposals are made by the programs with concurrence of senior management. These decisions take into account the panel recommendations, especially the substance of the discussions, as well as expectations for future award-making capacity based on the availability of funds, additional expected proposal load at the full proposal stage, and portfolio balance issues.

Results

Total Reviewed Panel Recommendations Total Invited Invite Rate
DEB Cluster Invite Do Not Invite No Consensus
SBS 289 79 210 0 85 29%
EP 440 94 346 0 101 23%
PCE 439 122 315 2 110 25%
ES 306 94 212 0 86 28%
DEB Total 1474 389 1083 2 382 26%

These numbers are consistent with our goal of inviting the most promising projects while targeting a success rate of approximately 25% for the resulting full proposals that will be submitted this summer.

Big Picture

Comparing to the previous rounds of preliminary proposals…

2012 2013 2014 2015 2016
Reviewed 1626 1629 1590 1495 1474
Invited 358 365 366 383 382
Invite Rate 22% 22% 23% 26% 26%

…we see that the system has recovered somewhat from the initial flood of submissions. Moreover, the invite rate, and subsequent full proposal success rate, has stabilized in a range that reasonably balances against the effort required to produce each submission.