Fiscal year 2015 has come to a close. With the dust settled, we can crunch the numbers on the DEB Core Program merit review and funding outcomes.

This post follows the format we’ve used in previous years. For a refresher, and lengthier discussions of the hows and whys of the metrics, you can visit the 2014 and 2013 numbers.

Read on to see how 2015 compares.

FY2015 Summary Numbers

The charts below all reflect proportions of DEB Core Program projects through each stage of the review process: preliminary proposals, full proposals, and awards.

In the review process leading to awards in FY2015:

DEB reviewed 1590 preliminary proposals received under the DEB Core Programs solicitation and LTREB solicitation in January 2014, about 25% of which were invited to the full proposal stage.

The preliminary proposal invitees were joined at the full proposal stage by 1) Direct submissions to DEB under the CAREER, OPUS, and RCN solicitations, and 2) Projects shared for co-review by another NSF program. Altogether 510 full proposals were reviewed in DEB during October and November of 2014.

From this pool of full proposals, DEB made awards to 131 projects (technically, these were 193 separate institutional awards but for analysis purposes we count collaborative groups once, i.e., as a single proposed project).

Below, we present and discuss the Division-wide success rate and some selected project demographics that were raised as concerns coming in to the preliminary proposal system. The demographic numbers are presented as proportions for comparison across the review stages. However, the progressive reduction in size of the denominators from preliminary proposals (1590) to awards (131) means each step becomes more sensitive to small absolute changes.

Success Rate

The success rate for research proposals in the DEB Core Programs remains much as it was since the preliminary proposal process was instituted; success rate is a function of the number and size of requests submitted by the research communities and appropriated funding levels.

Reminder: Elevated success rates (in grey) were due to:

• a one time ~50% increase in funding for FY2009 (the ARRA economic stimulus funding) without which success would have been ~13-15%; and,
• a halving of proposal submissions in FY2012 (the first preliminary proposal deadline replaced a second full proposal deadline for FY2012), without which success would have been ~8-9%.

 

Individual and Collaborative Projects

As seen in the figure below, there was little year-to-year change in the submission and funding success of single investigator projects.

While the proportion of single investigator preliminary proposals declined slightly, there was no decrease when it came to either full proposals or awards. As a reminder to readers: most of the apparent increase in the proportion of single investigator projects between the preliminary proposal and full proposal stages is because the primarily single-investigator proposals in the CAREER and OPUS categories are not subject to the preliminary proposal screen and thus they make up a relatively larger portion of the full proposals.  Similarly, the absence of CAREER and OPUS proposals at the preliminary proposal stage depresses the single investigator proportion of the preliminary proposal counts relative to the historical full proposal baseline.
The proportion of collaborative proposals in our award portfolio declined slightly from last year’s peak but is still above other prior years and doesn’t reverse the general upward trend over the past decade or so.
Readers may notice that the collaborative and single-investigator groupings don’t sum to 100%. The remainders are intra-institutional multi-PI arrangements; such projects are intellectual collaboration to be sure, but not a collaborative project per the NSF definition.

Early Career Scientists

As we discussed in the FY2013 Wrap-up, there are several imperfect metrics for identifying early career investigators, with the “Beginning Investigator” check-box on the cover page being the most immediately visible but also the most inconsistently applied identifier. Beginning Investigator includes everyone who has never received federal funding, and many researchers don’t, so it is not as directly related to career stage as we may want to think it. For the purposes of measuring “Beginning Investigators” we use the response in the BIO Classification Form (this poses the same question as the cover page but captures a more complete record of response than a single a check box).

According to the classification form data, beginning investigators continue to receive awards in proportion to full proposal submissions but consistently represent a smaller segment of the proposal pool at these later stages than at the preliminary proposal stage[i].

The better identifier of researcher career stage is the years since PhD of the PI.

Little changed in the profile of submitter and awardee degree ages from 2013 through 2015. Success rate does not improve markedly with degree age and generally stays between 20-30% up through 35 years post PhD. PIs more than 35 years post-PhD are typically less than 5% of the total PIs on proposals and awards: too few in number to read much into the data. Early-, mid-, and later-career PIs appear to be faring equally well in core program award decisions.

Gender & Predominantly Undergraduate Institution (PUI) Status

Last year we saw what seemed like a potentially spurious but also curious pair of year to year changes in these groups: the proportion of PUI awardees jumped up at the same time as the proportion of female PIs slumped.

We thought these changes were spurious because the absolute numbers involved were small and potential confounding factors were numerous. That is, there are a range of factors beyond the summary rating of the panel such as nuances of the reviewer input, other demographic characteristics, additional funding, overlap with others’ awards, past productivity, etc., that weigh into each award decision. If only a few proposals satisfy any one factor, even fewer proposals will satisfy combinations of factors. Thus, a decision to make an award that boosts one aspect of portfolio diversity may come with an opportunity cost of not addressing another aspect.

We thought this was curious because, with PUIs experiencing a notable jump and female PIs a notable drop, these stood out visually from other results and related to preexisting areas of concern. And, this year we see another big change but in the opposite direction.

The first key thing to note is that there is no surprising up or down change in submissions of preliminary proposals or full proposals in either group since before the preliminary proposal system. That doesn’t take away, however, from the sudden appearance of negative reciprocal changes between award proportions for these two demographics seen in FY2014 and FY2015.  On the other hand, there’s no trend (just greater variation), and there’s no direct mechanism we’ve been able to identify that would lend itself to management or modification.

Since the differences emerge only at the award stage, we continue to view this as we did last year.  Since we don’t see a similar effect from preliminary proposal review and invitation decisions (yielding the full proposals), it seems likely that this is directly related to the final decisions regarding use of limited funds across too many strong candidates.

We conclude that we are witnessing a small numbers effect; 131 awards is simply not a sufficiently large “sample” to be representative of the population across all of the potential decision-making variables and concerns. PUIs are a minority component of the proposal pool (~18%). Female PIs are a minority component of the proposal pool (~30%). Beginning Investigators are a minority component of the proposal pool (~20%). Proposals that fall into two categories are an even smaller fraction of the proposal pool (~6%) and proposals that fit into all three are even smaller yet (~1%).

Concluding Thoughts

Since the implementation of the preliminary proposal system in January 2012, we have not seen major departures from pre-2012 trajectories across these measures of our portfolio with respect to PI or institution demographics or collaborations. Three years in, we have not seen indication of Division-wide changes, especially any worsening of submission and award proportions among any of these major groups of concern. Success rates appear stable across multiple years for the first time in recent memory. However, the overall climate for awards remains poor due continued funding stagnation. If there is any bright side to that fact, funding woes at least appear to not have been amplified for any particular subgroup in recent years. But, the limited purchasing power of program budgets means our ability to support the full diversity of the research community will continue to be tested and subject to difficult trade-offs.

---

 

[i] Two points we shared in our post from this past spring on individual success rates come to mind: the first years of the preliminary proposal system have seen a large uptick in first-time submitters, and only 25% of individual PIs received any funding from DEB over a 9-year period examined in that post. Lots of PIs have never, and will never, receive funding from us. In the context of the result above, we see the learning curve. There is a high failure rate: especially among PIs without prior funding success, but even among those who have been successful in the past. While a successful preliminary proposal from a would-be first-time PI goes on to compete on even footing with full proposals from experienced PIs, we don’t expect the preliminary proposals from all unfunded PIs to have the same distribution across the quality spectrum as those from previously successful PIs.