Post-Panel Decision Making: What exactly is this “portfolio balance” I keep hearing about?

Program Officers frequently remind panelists of two things: 1) panel discussions are confidential and 2) the panel provides advice to the program; it doesn’t make decisions. Thus, what you see on the rating board is not the final outcome. The typical rejoinder to the second item is: so how do you get from the board to a final outcome? To us, that question sounds like an excellent basis for a blog post.

Once full proposal panels are done and reviewers have made their recommendations, our work is far from over. Program Officers incorporate the panel’s advice with other considerations to manage a variety of short- and long-term factors affecting scientific innovation and careers. Sure, funding the best science is paramount, but most programs receive many more deserving proposals than they can support. We use the term “Portfolio Balance” to describe the strategic considerations that program officers incorporate into these funding decisions. Below, we highlight several axes of the portfolio (in alphabetical order) and outline the driving thoughts behind each one:

  • Award diversity: Programs fund a variety of special awards such as CAREERs, RAPIDs, EAGERs, Research Coordination Networks, OPUS, Small Grants, and Dissertation Improvement Grants. These serve a variety of roles in diversifying the types of projects supported by the Foundation in ways rarely found in a regular grant.
  • Career Stage diversity: How should a program distribute support among PIs at different career stages? Beginning investigators bring new ideas but may have weaker grantsmanship. Mid-career scientists offer experience and a track record, and may merit special consideration if changing research direction. Late career scientists need opportunities to synthesize their work to create a legacy for their community. Postdoctoral awards create special opportunities for beginning scientists to pursue novel and independent projects.
  • Demographic diversity: How can NSF help diversify the scientific workforce and address various demographic imbalances? Many studies have shown that diversity in the workforce generates new ideas and approaches. Different people see different aspects of a topic through their experiences and educational backgrounds; more homogeneous research teams may miss novel and unexpected insights that lead to innovative solutions. Broader impacts often include activities designed to broaden participation in science.
  • Geographic diversity: How can a program ensure the opportunities and benefits of research reach the diverse geographic regions of the country? Innovative research is done in diverse institutions located outside of the major research hubs. In EPSCoR states, which generally receive a smaller portion of federal research dollars, leveraging opportunities can amplify the impact of an award while co-funding can stretch our program budget.
  • Institutional diversity: Not all stellar scientists are at the few major research universities. And, neither are all the students who will become the great researchers of the future. How can we direct limited research support to ensure opportunities are not limited to a select few? Funding projects from diverse institutions, including primarily undergraduate colleges and universities, minority-serving institutions, and regional universities, allows a broader range of faculty and students to participate in and strengthen the scientific enterprise.
  • Intellectual diversity: How do specific projects reinforce, build upon or challenge the results and knowledge generated by the diversity of other projects in the same broad domain? Program officers may try to balance research in areas that are currently “hot” with other topics of importance. Co-review with other programs provides another way to broaden the program’s domain and promote novel application of tools developed in other fields.
  • Laboratory Diversity: Where is the balance between investing in new/unfunded labs versus sustaining established enterprises? There are always new labs, labs running out of funds, labs with funding gaps, and labs with existing funding from us or elsewhere. We often consider PIs’ current funding status in making our decisions; it’s not an outright disqualifier to be well funded at the moment but it is an important consideration in distributing our funds.
  • Risk diversity: Does the program fund at least some work that is intellectually risky? Because progress in science depends on the willingness to challenge the norm, program officers often consider relative degrees of risk and innovation in their funding decisions. Some individuals argue that panels are overly conservative in their recommendations, but program officers make the final decisions and reflect carefully on the nature and magnitude of risks versus the potential payoffs for their field.

Because the distribution of submitted proposals can vary over time, portfolio balance requires both a short term and a long-range vision. NSF staff consider the overall present and future health of the research communities they serve at a depth not generally visible to individual scientists. The recommendations of the reviewers are by far the most important factor; the best of the best are likely to be funded. Discriminating among the next group of outstanding proposals usually involves consideration of one or more of the above factors leading up to that phone call saying you have been recommended for funding.


DEB Numbers: FY2015 Wrap-Up

Fiscal year 2015 has come to a close. With the dust settled, we can crunch the numbers on the DEB Core Program merit review and funding outcomes.

This post follows the format we’ve used in previous years. For a refresher, and lengthier discussions of the hows and whys of the metrics, you can visit the 2014 and 2013 numbers.

Read on to see how 2015 compares.

Continue reading

A reminder to check your FastLane Profiles

For any demographic analysis or comparison, NSF is reliant on the self-reported characteristics of participants in all phases of proposals and awards. Completion of the profiles is voluntary but critical for linking demographic data to proposal, funding, and review patterns. And, importantly, your profile provides the contact information that we use to reach out to you. So if your email address and institutional information are not up to date you may miss out on funding opportunities or critical notifications that affect your eligibility for funding.

So, is your FastLane PI profile complete, up to date, and error-free?

What about your OTHER FastLane profile? When was the last time you completed your Reviewer information? Continue reading

DEB Numbers: Analysis of Broader Impacts

A recent paper in Bioscience by a AAAS Fellow (Sean Watts), an Einstein Fellow (Melissa George), and an NSF Program Director (Doug Levey) explores how the Broader Impacts Criterion was applied and reported in DEB proposals between 2000 and 2010. A major conclusion is that activities aimed at recruiting and mentoring students from underrepresented groups are proposed more than twice as often as they are eventually reported by PIs; of all the types of broader impact activities, broadening participation is by far the toughest to achieve. This result and others are discussed in the context of a recent review of the Merit Review Criteria by the National Science Board and resulting revisions to the Proposal & Award Policies & Procedures Guide (PAPPG).

DEB Numbers: FY2014 Wrap-Up

At the end of 2013, we presented DEB submission and award portfolio data examining the initial results of the preliminary proposal process, leading to DEB’s FY2013 awards. In this post, we provide a follow-up for the second round of awards funded under the preliminary proposal system in FY2014. Continue reading