This post is a continuation of a discussion of the early data on the performance of Beginning Investigators and Primarily Undergraduate Institutions. Please read Part 1 before continuing for background and explanation of the terms used. If you are new to DEB Numbers posts, I suggest you read our introductory message to familiarize yourself with some of the conventions we use here to describe what can be confusing information.
When looking at the performance of a particular group of interest through the review process, success rate numbers are complicated by the trend of increasing proposal submissions that is impacting pretty much everyone across all of NSF. Falling success rates tend to distract from comparisons of the relative performance of a particular group of submitters. In the case of the two-stage review process in DEB, we heard particular concerns about the performance of Beginning Investigators and Primarily Undergraduate Institutions through the preliminary and full proposal stages. In order to evaluate how those groups are performing, we are looking at the representation of each of those groups in the portfolio of projects submitted and portfolio of awards made.
Representation of Projects Identifying the Lead PI as a Beginning Investigator (BI) and Projects Identifying the Lead Institution as a Primarily Undergraduate Institution (PUI) in the Portfolio of Project Submissions in DEB Core Program Panels FY2007-FY2013
In the first year of preliminary proposal review, both BIs and PUIs contributed to the preliminary proposal project pool at rates consistent with their submissions under the old system. At this date, it does not appear that the two-stage review mechanism either encouraged or discouraged submission from these groups to a greater degree than the general PI population. However, the submissions of both groups at the FY2013 full proposal stage constituted a smaller share of the projects. When we look at the portfolio of awards and invitations…
Representation of Projects Identifying the Lead PI as a Beginning Investigator (BI) and Projects Identifying the Lead Institution as a Primarily Undergraduate Institution (PUI) in the Portfolio of Awards or Invitations from DEB Core Program Panels FY2007-FY2013
*Note: Tentative numbers for current fiscal year under the Continuing Resolution budget plan: 80% of the 2012 budget. The FY2013 Full Proposal counts include full proposal projects submitted via CAREER, OPUS, RCN and co-reviews.
…we can see that both groups’ contribution to the portfolio of invitations coming out of the FY2012 Preliminary Proposal stage was on the low end, but within, the range seen for award portfolios since FY2007. The tentative numbers for the FY2013 Full Proposal stage put BIs firmly within the historic range for representation in the award portfolio. PUIs are on the very low end, but still within the range since 2007. Since we are still operating under a Continuing Resolution and the planned award portfolio may still change significantly, it is too early to conclusively say if the process impacted either of these groups.
One final interesting note that comes out of the current data is that while BIs represented 21.4% of the invited preliminary proposal projects, they only constituted 18.1% of the full proposal projects submitted. On the other hand, PUIs were only 12.0% of the invited preliminary proposal projects, but were 14.6% of the submitted full proposal projects. This difference means that the full proposal projects received outside of the preliminary proposal stage via CAREER, OPUS, RCN and co-review had a few more PUIs and a few less BIs relative to the regular proposal submission population.
The ultimate portfolio of awards is determined by the recommendations of the DEB Program Officers, within the constraints of the program budget, and reflecting the advice of the review panels. Here we have taken a look at the recent history and tentative FY2013 numbers for expected awards. At this point in time, the whole process appears to be avoiding disproportionate impacts on Beginning Investigators and Primarily Undergraduate Institutions. However, both of these groups have historically been minority groups in submissions to DEB and furthermore underrepresented in award portfolios compared to submissions. While we can report changes over time and compare between groups, these numbers do not provide answers to the questions that underpin evaluations of progress: What is the right mix of Beginning Investigators and Primarily Undergraduate Institutions in an award portfolio? And, how can we reach the right mix in awards? Would such a mix require a change in submission patterns and/or changes in peer review practices and/or changes in Program Officer handling of the submissions we already receive?