DEB Numbers: Revisiting performance of PI demographic groups, Part 2


This post is a continuation of a discussion of the early data on the performance of Beginning Investigators and Primarily Undergraduate Institutions.  Please read Part 1 before continuing for background and explanation of the terms used. If you are new to DEB Numbers posts, I suggest you read our introductory message to familiarize yourself with some of the conventions we use here to describe what can be confusing information.

When looking at the performance of a particular group of interest through the review process, success rate numbers are complicated by the trend of increasing proposal submissions that is impacting pretty much everyone across all of NSF.  Falling success rates tend to distract from comparisons of the relative performance of a particular group of submitters.  In the case of the two-stage review process in DEB, we heard particular concerns about the performance of Beginning Investigators and Primarily Undergraduate Institutions through the preliminary and full proposal stages.  In order to evaluate how those groups are performing, we are looking at the representation of each of those groups in the portfolio of projects submitted and portfolio of awards made.

Representation of Projects Identifying the Lead PI as a Beginning Investigator (BI) and Projects Identifying the Lead Institution as a Primarily Undergraduate Institution (PUI) in the Portfolio of Project Submissions in DEB Core Program Panels FY2007-FY2013

Numbers2b.1

In the first year of preliminary proposal review, both BIs and PUIs contributed to the preliminary proposal project pool at rates consistent with their submissions under the old system.  At this date, it does not appear that the two-stage review mechanism either encouraged or discouraged submission from these groups to a greater degree than the general PI population.  However, the submissions of both groups at the FY2013 full proposal stage constituted a smaller share of the projects.  When we look at the portfolio of awards and invitations…

Representation of Projects Identifying the Lead PI as a Beginning Investigator (BI) and Projects Identifying the Lead Institution as a Primarily Undergraduate Institution (PUI) in the Portfolio of Awards or Invitations from DEB Core Program Panels FY2007-FY2013

Numbers2b.2

*Note: Tentative numbers for current fiscal year under the Continuing Resolution budget plan: 80% of the 2012 budget.  The FY2013 Full Proposal counts include full proposal projects submitted via CAREER, OPUS, RCN and co-reviews.

…we can see that both groups’ contribution to the portfolio of invitations coming out of the FY2012 Preliminary Proposal stage was on the low end, but within, the range seen for award portfolios since FY2007.  The tentative numbers for the FY2013 Full Proposal stage put BIs firmly within the historic range for representation in the award portfolio.  PUIs are on the very low end, but still within the range since 2007. Since we are still operating under a Continuing Resolution and the planned award portfolio may still change significantly, it is too early to conclusively say if the process impacted either of these groups.

One final interesting note that comes out of the current data is that while BIs represented 21.4% of the invited preliminary proposal projects, they only constituted 18.1% of the full proposal projects submitted.  On the other hand, PUIs were only 12.0% of the invited preliminary proposal projects, but were 14.6% of the submitted full proposal projects.  This difference means that the full proposal projects received outside of the preliminary proposal stage via CAREER, OPUS, RCN and co-review had a few more PUIs and a few less BIs relative to the regular proposal submission population.

The ultimate portfolio of awards is determined by the recommendations of the DEB Program Officers, within the constraints of the program budget, and reflecting the advice of the review panels.  Here we have taken a look at the recent history and tentative FY2013 numbers for expected awards.  At this point in time, the whole process appears to be avoiding disproportionate impacts on Beginning Investigators and Primarily Undergraduate Institutions.  However, both of these groups have historically been minority groups in submissions to DEB and furthermore underrepresented in award portfolios compared to submissions.  While we can report changes over time and compare between groups, these numbers do not provide answers to the questions that underpin evaluations of progress: What is the right mix of Beginning Investigators and Primarily Undergraduate Institutions in an award portfolio?  And, how can we reach the right mix in awards?  Would such a mix require a change in submission patterns and/or changes in peer review practices and/or changes in Program Officer handling of the submissions we already receive?

3 thoughts on “DEB Numbers: Revisiting performance of PI demographic groups, Part 2

  1. “What is the right mix of Beginning Investigators and Primarily Undergraduate Institutions in an award portfolio?”

    As an opening gambit, I’d suggest that the proportions of awards should roughly reflect the proportion of submissions. This is a very common way to check if there’s systematic discrimination, after all. (E.g., “15% of the population are X, but only 3% are represented in our awards.)

    Currently, it looks like there is about a 3% gap between submissions and awards for both these categories.

    One suggestion: currently, the main difference between marking a full proposal as an RUI and other proposals has been that RUI proposals are allowed a 5 page impact statement. This has always truck me as counter-intuitive. Researchers at undergraduate institutions usually have less time for research, and they are asked to write EVEN LONGER grant proposals than everyone else.

    • Zen,

      Thank you for rolling this our way. We are listening to the discussion, but we wanted to make a couple of points.

      The numbers of Beginning Investigators (BI) and Primarily Undergraduate Institutions (PUI) are probably biased downwards. BIs are, by NSF definition, people who have never received an NSF research grant (e.g., GRFs and DDIGs do not count). Not everyone chooses to indicate his/her status, and not everyone who has received funding has necessarily received very much. Likewise, not every PUI is recorded as such in the NSF database. Many schools without graduate programs in DEB fields have other departments eligible for NSF funding with plenty of Ph.D. students. The institution is not necessarily classed a PUI, but maybe a primarily undergrad program should count? Program directors usually consider these cases, but the databases at NSF do not; the success rates reflect only the “official” cases.

      The Research at Undergraduate Institutions (RUI) solicitation (NSF 00-144) allows the PI a five page opportunity to explain what this grant would do for research and education at her/his institution. Many RUI-eligible schools (those offering fewer than 10 doctoral degrees a year in any fields supported by NSF) have boilerplate RUI Impact Statements to which a PI can simply add a few specifics. The RUI program is an NSF-wide solicitation so changing that five page document is beyond DEB’s immediate purview. Note that although there is a special RUI solicitation, there are no special RUI funds. These projects are funded from program core budgets; a PI at a RUI school can (and many do!) submit a non-RUI proposal if they so choose and it does not change the pool of money that might fund their research. In any case, the program directors probably do know what kind of institution submitted the proposal!

  2. Pingback: DEB Numbers: FY2013 Wrap-Up | DEBrief

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s