Our friends at Dynamic Ecology posted a little while back about the NSF-wide trends in per-person success rate based on this 2014 report to the National Science Board that provided merit review process statistics across the whole agency[i]. There were several questions in the comments to that post regarding the context for the numbers and how they would look for DEB or IOS, especially since preliminary proposals were explicitly excluded from the calculations in the report to the NSB[ii].

So, we’ve put something together with DEB data to follow-up on that discussion. Our analysis sticks to the general approach of the NSF-wide report with modifications to allow inclusion of preliminary proposal data.

Part 1: Inclusion Criteria

First, let’s be clear about what we’re counting here. The NSB report’s Figure 14 illustrated a per-PI success rate based on counts of Competitive Research Grant Actions leading to Award or Decline decisions. That institutional jargon terminology specifies 3 different filters to define what was counted,

A context filter: Competitive (a stand-alone grant request) versus Non-competitive (changes to an existing grant such as a supplement or a PI move to a new institution) decision-making;

A content filter: Research (just what it sounds like, both Core and Special programs) versus Non-research (e.g., fellowships, dissertation support, travel support, conferences) activities;

An outcome filter: Awarded or Declined versus Any Other Outcome (e.g., invite, not invite, still pending a decision, returned without review, or withdrawn before a decision)

This is actually a really good set of filters for narrowing down the universe of “stuff NSF does” to questions about “bread and butter” grants. Ignoring the Any Other Outcome proposals is a good thing since those categories of proposals were never actually part of the competition in most cases across NSF. On the other hand, it complicates measurement of programs where large numbers of preliminary proposals are involved, as is our case.

Part 2: The Proposal Data

Our first table presents the big picture of proposal submissions for DEB for a period of 2006-2014 (chosen mainly because that was the span of complete years beyond which the server was getting angry with us for requesting too many records, #overlyhonestmethods). We’ve divided them up following each of the filters mentioned above and also split out the DEB-relevant sub-units. (Note: for consistency across all of the different proposal types and with the NSF-wide data, this table counts separately all proposals with unique identification numbers in FastLane. This differs from the way DEBrief usually combines separate proposals from collaborative group into a single “project” unit for counts.)

(click image to make legible)

We have discussed some of these trends before, but to quickly review the basic points:

1) Total actions spiked with the launch of the preliminary proposal system in 2012 but have since come down a bit. This was preceded by another spike in 2010 that was in part a reaction to stimulus funding in 2009 (evidenced by upward jumps in DDIGs, and Core programs from 2009-2010) and also a major spike in special programs that reflects the launch of Dimensions of Biodiversity and some other redistribution of special program responsibilities between Divisions in BIO.

2) Economic stimulus (ARRA) money in 2009 and the wiggle-room gained by clearing out some of the backlog of requests and paying down future commitments resulted in significantly elevated award counts in 2009 and 2010 that distort the longer-term pattern.

3) Incoming preliminary proposal numbers (2012-2014) have been nearly flat, as have the number of research grant award actions, especially when considering both core and special program components over the entire period.

We’re not adding per-proposal success rates to this table specifically because the preliminary proposal process crosses fiscal years and the corrections needed to account for the complexities of the process make that number very different from the straight-forward single-year data above (see endnote i). Per-proposal success rates are shown in our FY2014 wrap-up post.

Part 3: Per-Person Calculations

Each action in the table from Part 2 links to a record of between 1 and 5 persons (PIs and CoPIs) on the proposal cover page.

[Contextual tangent: we are not differentiating between core and special programs in DEB for the per-person success rate. Could it be done? Sure, but the special programs and core programs are both funding research grants and we see that applicants to one or the other quite often switch targets depending on the convenience of deadline or opportunity. Ultimately getting one or the other provides the same result, research funding.]

In total, there were 11,789 unique PIs/CoPIs associated with the 20,724 Competitive Research Grant actions in DEB between 2006 and 2014. During the same time frame, DEB made 2,671 Competitive Research Grant Awards that included a total of 2,970 unique PIs or CoPIs. Most individuals (75% of unique PI/CoPIs) who applied to DEB for funding never received a Competitive Research Award during this entire 9-year period.

The NSB report calculated PI success rate in a 3-year moving window, we’ll do that in a moment. First, we want to split it a different way to account for the stimulus (ARRA) funding in 2009; when combined with the smoothing of the window, that spike in awards winds up distorting some details we’d like to explore.

Annual Per-Person Success Rate

Fiscal Year	2006	2007	2008	2009	2010	2011	2012	2013	2014
Total Unique PIs and CoPIs Applied	2052	2057	2060	2274	3266	2733	4310	4546	3950
Unique Women Applied	503	476	488	559	791	676	1127	1211	1093
Total Unique PIs and CoPIs Awarded	352	449	424	620	601	392	424	455	414
Unique Women Awarded	93	105	114	166	163	100	110	126	110
Per-Person Success Rate	17.2%	21.8%	20.6%	27.3%	18.4%	14.3%	9.8%	10.0%	10.5%
Per-Woman Success Rate	18.5%	22.1%	23.4%	29.7%	20.6%	14.8%	9.8%	10.4%	10.1%
First Recordings of a PI	2052	1149	909	1006	1553	943	1645	1542	990
Last Recordings of a PI	526	542	471	595	1085	780	1600	2240	3950

The notable patterns here:

1) The preliminary proposal system brought in a huge increase in persons applying each year, double pre-stimulus levels.

2) 2010 was a big year, matching what we saw in the proposal load table, with a large increase in people submitting in reaction to the economic stimulus (ARRA) and following the movement of special programs into DEB.

3) These additional PIs were actually “new” people who had not submitted to DEB since at least 2006; and, we saw about 50% higher numbers of new people for each of 2010, 2012, and 2013 than typical in previous years. But, 2014 looks more like the longer-term norm.

4) The stimulus funding had a big, but temporary, effect by allowing an extra ~200 persons to be funded in both 2009 and 2010. While the effect on per PI success was large in 2009, it was much less in 2010 because of the 1000 additional applicants that year.

5) Excepting the stimulus years, the number of persons funded by research grants doesn’t show a trend or even all that much variation over this span: ~420 unique persons per year.

6) The growth in unique PIs we see includes both an absolute increase and an increasing proportion of female investigators among applicants, although the temporal range is small and the female proportion of applicants has yet to exceed 28%. At the same time, women have generally experienced a per-person success rate (17.7 %) similar to that of the general population (16.7%).

A Quick Sensitivity Check

There’s a legitimate question as to whether counting PIs and CoPIs provides the best metric of success. Perhaps we should count just PIs? This is what the NSB report does. However, at the preliminary proposal stage, with only a single proposal cover page per project team, there are many instances of collaborative PIs that appear as CoPIs as well as collaborative PIs and CoPIs that don’t appear on a cover page at all. The constraints of the FastLane submission system at the preliminary proposal stage generally lead to undercounting total participants and artificially inflating the balance of CoPIs relative to PIs[iii]. Counting only PIs causes two problems: 1) it ignores a portion of the population at the preliminary proposal stage that would have been counted on full proposals and 2) it would artificially raise the per-PI success rate under the two-stage process relative to the pre-2012 submission process. So, to reflect funding reality as best we can, we cast a wide net and include everyone from the cover pages in the calculations above. However, we can also look to see if the numbers come out any differently if we constrain our calculations to only PIs. Other than the counts being somewhat smaller, the per-person success rates are generally not changed and are tightly correlated with results shown above.

Fiscal Year	2006	2007	2008	2009	2010	2011	2012	2013	2014
Total Unique PIs-Only Applied	1299	1322	1358	1419	1973	1706	2390	2305	2140
Unique Women Applied	314	303	321	366	506	458	684	687	660
Total Unique PIs-Only Awarded	232	292	276	397	361	250	266	261	257
Unique Women Awarded	56	63	69	113	108	66	74	76	63
Per-Person SuccessRate	17.9%	22.1%	20.3%	28.0%	18.3%	14.7%	11.1%	11.3%	12.0%
Per-Woman Success Rate	17.8%	20.8%	21.5%	30.9%	21.3%	14.4%	10.8%	11.1%	9.5%

Based on the tight coupling of these measures, we continue with our analysis of per-person success using both PIs and CoPIs.

3-Year Window Per-Person Success Rate

In comparison to the annual success rate data, many of the details noted above are paved over by the 3-year window method. We’re not disparaging this method; it is quite useful, especially during steadier budget times. Because the typical grant lasts 3 years, the 3-year success rate window roughly measures the percent of the active PI population that could be continuously funded under current expectations for the size and duration of grants. However, in the case where we have multiple shocks to the system occurring over the reporting period, it can generate misperception.

	Fiscal Window	2006-2008	2007-2009	2008-2010	2009-2011	2010-2012	2011-2013	2012-2014
DEB	Unique PIs and CoPIs Applied	4110	4303	5221	5563	6798	7354	7790
	Unique PIs and CoPIs Awarded	1130	1361	1511	1475	1301	1162	1197
	Per-Person Success Rate	27.5%	31.6%	28.9%	26.5%	19.1%	15.8%	15.4%
	Per-Person Success Rate (PIs-Only)	28.2%	32.5%	29.7%	27.8%	20.7%	17.9%	17.5%
NSF-wide (From NSB Report)	Per-Person Success Rate (PIs-Only, excludes preliminary proposals)	37%	40%	40%	38%	35%	35%	TBD

In this table, the windows affected by the extra stimulus (ARRA) funding are in italics and the windows affected by the new applicants to the preliminary proposal system are in bold. The 2010-2012 window sits at the intersection of the stimulus-elevated award numbers and preliminary proposal-driven increase in applicants. What we see here is that the pre-stimulus (2006-2008) and post stimulus (2011-2013 and 2012-2014) awardee numbers are quite similar. However, the applicant numbers have grown substantially, reflecting both the influx of new PIs in response to the stimulus and to the preliminary proposal system. This increase in the number of unique PIs/co-PIs applying in a given 3 year window drives the lower per PI funding success rate.

Notably, DEB’s per-person success rate is continually lower than the NSF-wide number but does follow the same pattern across the ARRA funding windows. The exclusion of preliminary proposal PIs from the NSF-wide counts leads to the increasing disparity between DEB and NSF-wide success rates from 2010-2012 onward.

Award Distribution

We can also compare the annual and 3-year window measures to gain insight into another aspect of per-person success rate. A relevant concern we often hear is that “the same well-funded people just get funded over and over again”. If that were true, we would expect persons funded in year 1 of a window to be funded again in year 2 or year 3. So, the count of unique awardees in the 3-year window would be smaller than the sum of the annual counts of unique awardees (i.e., Dr. X is counted once in the three year window measure but twice, once in year 1 and once in year 3, by the annual measure). But, if grants were spread out (and thus, fewer PIs with overlapping/continuous funding), there would be many fewer repeat PIs so the sum of the annual counts would be much closer to the 3-year window count. In our case we have:

Fiscal Window	2006-2008	2007-2009	2008-2010	2009-2011	2010-2012	2011-2013	2012-2014
3-Year Count of Unique PI/CoPI Awardees	1130	1361	1511	1475	1301	1162	1197
Sum of Annual Unique PI/CoPI Awardee Counts	1225	1493	1645	1613	1417	1271	1293

 

What this tells us is that fewer than 10% of awarded PIs in any 3-year window are repeat awardees during that period (~1.5 – 3.1% of all PIs who apply during that period).

If we step back and consider the whole 9 year period, we still find that the majority of PIs are funded only once.

Even if they were all 5-year grants, continuous funding of a lab from DEB research grants alone is extremely unlikely for the vast majority of PIs.

 

Concluding Thoughts

1) The number of people being supported on DEB research grants (~420 persons on new grants per year) hasn’t changed much over this period, except for the temporary shock of the economic stimulus.

2) The stimulus, and a 3-year method of smoothing, really messes with the general perception of funding rates. (We actually hadn’t really thought about that much except as the one-year outlier we usually label in our posts. This was eye-opening in that regard.)

3) Funding rates, both per-person and per-proposal, are being driven down by increases in the applicant/application pool: primarily growth in actual participant numbers but some intensification of per-person activity is also possible.

4) Of 11,789 unique PI/CoPI applicants, only 2,970 (25% of all applicants) received any funding over the 9-year period examined. Of those 2,970 to receive funding, only 772 received multiple awards (26% of awardees, 6% of all applicants) that could potentially maintain continuous “funding” over this period. Any person applying to DEB’s competitive research programs is unlikely to be funded, and much less likely to maintain continuous support for a lab from this single funding source.

5) Coming back to our original motivation for this post, per-person success rates for funding in DEB were consistently ~10 percentage points lower than the NSF-wide submission and funding data in years leading up to the preliminary proposal system. The exclusion of preliminary proposals from NSF-wide statistics has only deepened the apparent magnitude of this disparity in recent years and has even altered the trajectory of PI participant counts for the agency as a whole.

[i] The 2015 version of the report, with NSF-wide numbers through fiscal year 2014 should be arriving soon.

[ii] Why are preliminary proposals excluded?

The short answer is: the records don’t neatly match up.

The longer answer is: Beyond the major issue that the entire process from receipt of preliminary proposals through decisions on the related cohort of full proposals crosses fiscal years and so defies straight-forward binning, the path from individual preliminary proposal to award can be surprisingly non-linear. Our ability to accommodate these complexities comes at the expense of our ability to enforce strong rules to ensure continuity of the data you provide to us. Collaborative proposals are a prime example. In many cases not all PIs and CoPIs are actually listed on the cover page of the preliminary proposal. When a full collaborative proposal is invited it results in several different cover pages that each contain a different set of names. There’s no guaranteed 1-to-1 mapping of PIs across the entire process. Also, the basic ability to associate a full proposal with a preliminary proposal is tied to the “institution” which is the official owner of the proposals (not the PI). So if a PI changes institutions, or a collaborative reorganizes, or any number of other things that happen quite regularly comes to pass, the system doesn’t allow the full proposal to be linked to the actual preliminary proposal record. There are also people who receive an invite but then elect not to submit a full proposal for various reasons. On top of which you also have a number of CAREER-eligible PIs who (with or without an invite) will submit CAREER based on their preliminary proposal. The twists and turns are multitude and in the choice between flexibly accepting them and rigid data quality, we generally come down on the side of broad acceptance.

[iii] This is why we ask you to submit a personnel list by email and list all of the people on the 1^st page of the preliminary proposal project description to ensure reviewers get the full info. Unfortunately, tying those names to FastLane records is not currently practical.