Preliminary Proposal Evaluation Update for DEB and IOS: Surveys Arriving Soon


Dear Researcher,

We are writing today to alert you that you may soon receive an invitation by e-mail to participate in a short survey conducted by Abt Associates on behalf of the US National Science Foundation (NSF). This is a legitimate request and we invite you to participate.

This survey is part of an independent evaluation of the two-step merit review process (preliminary proposal system) implemented by the Division of Environmental Biology (DEB) and Division of Integrative Organismal Systems (IOS). Some of you may have also received an invitation to complete a broader merit review satisfaction survey within the past year. These are complementary but completely separate activities.

The goal of this survey is to examine the level of satisfaction with the two-step merit review process (preliminary proposal system) pilot in DEB and IOS and to estimate the workload associated with preparing and reviewing proposals. The survey is being sent to a sample of DEB and IOS applicants and reviewers and to a comparison group from similar NSF programs which have not adopted the two-step process. This approach will enable us to understand the relative advantages and limitations of the change as well as to capture everyone’s perspective.

While participation in the survey is voluntary, we hope that you will take a few minutes to share your views. Research community input is vital to developing the best approaches to handle the large number of grant applications without compromising the quality of review and that is why your participation is very important. Thank you in advance for your help.

Regards,

Paula Mabee, Division Director Heinz Gert de Couet, Division Director
Division of Environmental Biology Division of Integrative Organismal Systems
National Science Foundation National Science Foundation

Note: Pursuant to 5 CFR 1320.5(b), an agency may not conduct or sponsor, and a person is not required to respond to an information collection unless it displays a valid OMB control number. The OMB control number for this collection is 3145-0215. Public reporting burden for this collection of information is estimated to average less than 10 minutes per response, including the time for reviewing instructions. Send comments regarding this burden estimate and any other aspect of this collection of information, including suggestions for reducing this burden, to: Reports Clearance Officer, Office of the General Counsel, National Science Foundation, Suite 1265, 4201 Wilson Boulevard, Arlington, VA 22230.

DEB Numbers: Historical Proposal Loads


Last spring we posted on the per-person success rate and pointed out several interesting findings based on a decade of DEB data. We were seeing a lot of new PIs and, conversely, a lot of PIs who never returned after their first shot. And, the vast majority of PIs who managed to obtain funding are not continuously funded.

This post is a short follow-up to take a bigger picture look at submission rates.

Since preliminary proposals entered the scene, DEB really hasn’t seen much change in the submission pattern: 75% of PIs in any year submit one preliminary proposal and the other 25% submit two (and a small number submit three ideas in a year, if one also counts full proposals to special programs).

Before the preliminary proposals were launched, we ran some numbers on how often people tended to submit. The results were that, in the years immediately prior to preliminary proposals (~2008-2011), around 75% of PIs in a year were on a single proposal submission (25% on two or more). Fewer than 5% of PIs submitted more than two proposals in a year. Further, most PIs didn’t return to submit proposals year after year (either new ideas or re-working of prior submissions); skipping a year or two between submissions was typical. These data conflicted with the perceptions and anecdotes that “everyone” submitted several proposals every year and were increasing their submission intensity. Although recent data don’t support those perceptions, we still wondered if there might be a kernel of truth to be found on a longer time scale. What is the bigger picture of history of proposal load and submission behavior across BIO?

Well, with some digging we were able to put together a data set that lets us take a look at full proposal research grant submissions across BIO, going all the way back to 1991 when, it seems, the NSF started computerized record-keeping. Looking at this bigger picture of submissions, we can see when changes have occurred and how they fit into the broader narrative of the changing funding environment.

Total BIO full research grant submissions per year (line, right axis) and proportions of individuals submitting 1, 2, 3, 4, 5, or more proposals each calendar year from 1991 to 2014. (Note: 2015 is excluded because proposals submitted in calendar year 2015 are still being processed at the time of writing.)

Total BIO full research grant submissions per year (line, right axis) and proportions of individuals submitting 1, 2, 3, 4, 5, or more proposals each calendar year from 1991 to 2014. (Note: 2015 is excluded because proposals submitted in calendar year 2015 are still being processed at the time of writing.)

 

1990s: Throughout the 1990s BIO received about 4000 proposals per year. This period of relative stability represents the baseline for more than a decade of subsequent discussions of increasing proposal pressure. Interestingly, the proportion of people submitting two or more proposals each year grew over this period, but without seeming to affect total proposal load; this could result from either increasing collaboration (something we’ve seen) or a shrinking PI pool (something we haven’t seen). At this time NSF used a paper-based process, so the cost and effort to prepare a proposal was quite high. Then….

2000s: In 2000, FastLane became fully operational and everyone switched to electronic submission. BIO also saw the launch of special programs in the new Emerging Frontiers division. In a single year, it became easier to submit a proposal and there were more deadlines and target dates to which one could potentially submit. The new electronic submission mechanism and new opportunities likely both contributed to increased submissions in subsequent years.

Following the switch to FastLane, from 2001 to 2005, total annual submissions grew to about 50% above the 1990s average and stayed there for a few years. This period of growth also coincided with an increasing proportion of people submitting 2+ proposals. Increasing numbers of proposals per person had only a limited effect on the total proposal load because of continued growth in collaboration (increasing PIs per proposal). Instead, the major driver of proposal increases was the increasing number of people submitting proposals. This situation was not unique to BIO.

This period from 2001 to 2005 was the rapid growth that sparked widespread discussion in the scientific community of overburdening of the system and threats to the quality of merit review, as summarized in the 2007 IPAMM report.

Eventually, however, the community experienced a declining success rate because BIO budgets did not go up in any way to match the 50% increase in proposal submissions. From 2005-2008 submissions/person seemed to stabilize and submissions peaked in 2006. We interpret this as a shift in behavior in response to decreasing returns for proposal effort (a rebalancing of the effort/benefit ratio for submissions). It would have been interesting to see if this held, but….

2009/2010: In 2009 and 2010, BIO was up another ~1000 proposals over 2006, reaching an all-time high of nearly 7000 proposal submissions. These were the years of ARRA, the economic stimulus package. Even though NSF was very clear that almost all stimulus funding would go toward funding proposals that had been already reviewed (from 2008) and that we wouldn’t otherwise be able to afford, there was a clear reaction from the community. It appears that the idea of more money (or less competition) created a perception that the effort/benefit relationship may have changed, leading to more proposals.

2011: We see a drop in 2011. It is plausible that this was the realization that the ARRA money really was a one-time deal, there were still many more good proposals than could be funded, and that obtaining funding hadn’t suddenly become easier. As a result, the effort/benefit dynamic could be shifting back; or, this could’ve been a one-time off year. We can’t know for sure because…

2012: Starting in 2012 IOS and DEB, the two largest Divisions in BIO, switched to a system of preliminary proposals  to provide a first-pass screening of projects (preliminary proposals are not counted in the chart). This effectively restricted the number of full proposals in the two largest competitions in BIO such that in 2012, 2013, and 2014 the full proposal load across BIO dropped below 5000 proposals per year (down 2000 proposals from the 2010 peak). The proportion of individuals submitting 2+ full proposals per year also dropped, consistent with the submission limits imposed in DEB, IOS, and MCB. PIs now submitting multiple full proposals to BIO in a given year are generally submitting to multiple programs (core program and special program) or multiple Divisions (DEB and [IOS or MCB or EF or DBI]) and diversifying their submission portfolios.

In summary, the introduction of online and multi-institutional submissions via FastLane kicked off a decade of change marked by growth in proposal submissions and per-PI submissions to BIO. The response, a switch to preliminary proposals in IOS and DEB, caused a major (~1/3) reduction in full proposals and also a shift in the proportion of individuals submitting multiple proposals each year. In essence, the pattern of proposal submission in BIO has shifted back to what it was like in the early 2000s. However, even with these reductions, it is still a more competitive context than the 1990s baseline, prior to online submissions via FastLane.

DEB Numbers: Are aquatic ecologists underrepresented?


Editor’s note: This post was contributed by outgoing rotating Program Officer Alan Wilson and is a write-up of part of a project performed by DEB summer student Zulema Osorio during the summer of 2015.

Generalizability has been fundamental to the major advances in environmental biology and is an important trait for current research ideas proposed to NSF.  Despite its significance, a disconnect between terrestrial and aquatic ecological research has existed for several decades (Hairston 1990).

For example, Menge et al. (1990) quantitatively showed that authors heavily (~50%-65%) cite more studies from their representative habitat but that terrestrial ecologists are less likely to include citations from aquatic systems than the converse.  Failure to broadly consider relevant literature when designing, conducting, and sharing findings from research studies not only hinders future scientific advances (Menge et al. 2009) but may also compromise an investigator’s chances for funding[i] when proposing research ideas.

More recently, there have been anecdotal reports from our PI community that freshwater population or community ecology research is under-represented in NSF’s funding portfolio.  To explore the potential bias in proposal submissions and award success rates for ecological research associated with focal habitat, we compared the submissions and success rates of full proposals submitted to the core Population and Community Ecology (PCE) program from 2005-2014 that focused on terrestrial systems, aquatic systems, or both (e.g., aquatic-terrestrial linkages, modeling, synthesis).  Data about focal ecosystems were collected from PI-reported BIO classification forms.  To simplify our data analysis and interpretation, all projects (including collaboratives) were treated only once.  Also, the Division of Environmental Biology (DEB) switched to a preliminary proposal system in 2012.  Although this analysis focuses only on full proposals, the proportion of preliminary and full proposal submissions for each ecosystem type were nearly identical for 2012-2014.  Some projects (2.7% of total projects) provided no BIO classification data (i.e., non-BIO transfers or co-reviews) and were ignored for this project.  Finally, several other programs inside (Ecosystem Science, Evolutionary Processes, and Systematics and Biodiversity Science) and outside (e.g., Biological Oceanography, Animal Behavior, Arctic) of DEB fund research in aquatic ecosystems.  Thus, our findings only relate to the PCE portfolio.

In total, 3,277 core PCE projects were considered in this analysis. Means + 1 SD were calculated for submissions and success rates across 10 years of data from 2005-2014. Terrestrial projects (72% ± 2.8% SD) have clearly dominated projects submitted to the core PCE program across all ten years surveyed (Figure 1).  Aquatic projects accounted for 17% (± 2.6% SD) of the full proposal submissions while projects that include aspects of both aquatic and terrestrial components accounted for only 9% (± 1.6% SD) (Figure 1).  The full proposal success rate has been similar across studies that focused on terrestrial or aquatic ecosystems (calculated as number of awards ÷ number of full proposal submissions; Figure 2; terrestrial: 20% ± 6.9% SD; aquatic: 18% ± 6.5% SD).  Proposal success rate dynamics for projects that focus on both ecosystems are more variable (Figure 2; 16% ± 12.7% SD), in part, due to the small population size (9.5% of the projects considered in this study).

Figure 1. Submission history of full proposals submitted to the core PCE program from 2005-2014 for terrestrial (brown), aquatic (blue), or both ecosystems (red). Proposals were classified based on PI-submitted BIO classification forms. Note that some projects did not provide BIO classification data. These projects were ignored for this analysis and explain why yearly relative data may not total 100%.

Figure 1. Proportion of full proposals submitted to PCE based on focal ecosystem from 2005 to 2014.

Figure 2. Success rate of full proposals submitted to PCE based on focal ecosystem from 2005 to 2014.

Figure 2. Success rate of full proposals submitted to PCE based on focal ecosystem from 2005 to 2014.

In summary, anecdotal PI concerns of fewer funded aquatic proposals in PCE are consistent with available data but are an artifact of fewer aquatic proposal submissions.  Although funding rates for all full PCE proposals have generally varied from 2005-2014 (mean: 19.9% ± 6.4% SD; range: 11%-29%) as a function of available funds and the number of proposals considered, terrestrial- and aquatic-focused research proposals have fared similarly for the past decade.  PCE, like the rest of DEB and NSF, is motivated to have a diverse portfolio and encourages ecologists from varied institutions and backgrounds to submit ideas that study interesting, important questions that will generally move the field of population and community ecology forward.

Figures

Figure 1. Submission history of full proposals submitted to the core PCE program from 2005-2014 for terrestrial (brown), aquatic (blue), or both ecosystems (red).  Proposals were classified based on PI-submitted BIO classification forms.  Note that some projects did not provide BIO classification data.  These projects were ignored for this analysis and explain why yearly relative data may not total 100%.

Figure 2. Success rate history of full proposals submitted to the core PCE program from 2005-2014 for terrestrial (brown), aquatic (blue), or both ecosystems (red).  Proposal success rate is calculated for each ecosystem type as the number of awards ÷ number of full proposal submissions.   Proposals were classified based on PI-submitted BIO classification forms.

References

Hairston, Jr., N. G. 1990. Problems with the perception of zooplankton research by colleagues outside of the aquatic sciences. Limnology and Oceanography 35(5):1214-1216.

Menge, B. A., F. Chan, S. Dudas, D. Eerkes-Medrano, K. Grorud-Colvert, K. Heiman, M. Hessing-Lewis, A. Iles, R. Milston-Clements, M. Noble, K. Page-Albins, R. Richmond, G. Rilov, J. Rose, J. Tyburczy, L. Vinueza, and P. Zarnetska. 2009. Terrestrial ecologists ignore aquatic literature: Asymmetry in citation breadth in ecological publications and implications for generality and progress in ecology. Journal of Experimental Marine Biology and Ecology 377:93-100.

[i] Generalizability “within its own field or across different fields” is a principal consideration of the Intellectual Merit review criterion: http://www.nsf.gov/pubs/policydocs/pappguide/nsf16001/gpg_3.jsp#IIIA

Reminder: Project Reports Required for New and Continuing Funding


Note: This is an edited version of a post originally appearing on May 20, 2014.

This is a critical reminder for anyone who currently has a continuing grant[i] OR has been (hopes to be) recommended for funding in the remainder of fiscal year 2016 (i.e. now through October 1, 2016).

We need you to complete your reports now for projects funded in prior years, up to and including FY 2015, in order to release FY 2016 funds. Continue reading

Spring 2016: DEB Preliminary Proposal Results


Notices

All PIs should have received notice of the results of your 2016 DEB Core Program preliminary proposals by now. Full proposal invitation notices were all sent out by the first week of May (ahead of schedule), giving those invited PIs a solid three months to prepare their full proposals. ‘Do Not Invite’ decisions began going out immediately thereafter and throughout the rest of May.

If you haven’t heard, go to fastlane.nsf.gov and log in. Then, select the options for “proposal functions” then “proposal status.” This should bring up your proposal info. If you were a Co-PI, check with the lead PI on your proposal: that person is designated to receive all of the notifications related to the submission.

If you are the lead PI and still have not heard anything AND do not see an updated proposal status in FastLane, then email your Program Officer/Program Director. Be sure to include the seven-digit proposal ID number of your submission in the message.

Process

All told, DEB took 1474 preliminary proposals to 10 panels during March and April of 2016. A big thank you to all of the panelists who served and provided much thoughtful discussion and reasoned recommendations. Note: if you’re interested in hearing a first-hand account of the DEB preliminary proposal panel process, check out this great post by Mike Kaspari.

Panelists received review assignments several weeks prior to the panels and prepared individual written reviews and individual scores. During the panel, each proposal was discussed by the assigned panelists and then presented to the entire panel for additional discussion and assignment to a rating category. Panels were presented two recommendation options for each preliminary proposal: Invite or Do Not Invite. Following discussion, the assigned panelists prepared a panel summary statement to synthesize the key points of the panel discussion and rationale for the assigned rating.

Both the individual written reviews and the panel summary statement are released to the PI of the preliminary proposal.

As we’ve discussed previously, the final decisions on the preliminary proposals are made by the programs with concurrence of senior management. These decisions take into account the panel recommendations, especially the substance of the discussions, as well as expectations for future award-making capacity based on the availability of funds, additional expected proposal load at the full proposal stage, and portfolio balance issues.

Results

Total Reviewed Panel Recommendations Total Invited Invite Rate
DEB Cluster Invite Do Not Invite No Consensus
SBS 289 79 210 0 85 29%
EP 440 94 346 0 101 23%
PCE 439 122 315 2 110 25%
ES 306 94 212 0 86 28%
DEB Total 1474 389 1083 2 382 26%

These numbers are consistent with our goal of inviting the most promising projects while targeting a success rate of approximately 25% for the resulting full proposals that will be submitted this summer.

Big Picture

Comparing to the previous rounds of preliminary proposals…

2012 2013 2014 2015 2016
Reviewed 1626 1629 1590 1495 1474
Invited 358 365 366 383 382
Invite Rate 22% 22% 23% 26% 26%

…we see that the system has recovered somewhat from the initial flood of submissions. Moreover, the invite rate, and subsequent full proposal success rate, has stabilized in a range that reasonably balances against the effort required to produce each submission.

DEB 2016 Summer Meetings Schedule


Meeting season is upon us. Here’s a quick overview of the where, when, and who for finding your DEB representatives at annual meetings this summer. Note: Lists of expected attendees are tentative and subject to change. Check back for updates and additional details of scheduled sessions and other outreach activities as they become available.

 

Society of Wetland Scientists’ 2016 Annual Meeting

http://swsannualmeeting.org/

31 May – 4 June 2016; Corpus Christi, Texas

Liz Blood (Ecosystems)

 

EEID (Ecology and Evolution of Infectious Disease)

http://eeid.cornell.edu/eeid-2016/

3 – 5 June 2016; Ithaca, New York

Sam Scheiner (Evolutionary Processes); Karen Alroy (Science Associate); Diana Weber (AAAS S&T Policy Fellow)

 

ASLO Summer Meeting

https://www.sgmeet.com/aslo/santafe2016/default.asp

5 – 10 June 2016; Santa Fe, New Mexico

Alan Tessier (DDD); Lou Kaplan (Ecosystems); Maria Gonzalez (Population and Community Ecology); Tim Kratz (Macrosystems & NEON Science); Mike Vanni (Postdoctoral Fellows program in DBI)

Event: NSF Funding Opportunities in Aquatic Sciences; Date: Tuesday, 7 June; Time: 12:00 – 13:30

 

ASM Microbe 2016

http://www.asmmicrobe.org/

16 – 20 June 2016; Boston, MA

Matt Kane (Ecosystems); Leslie Rissler (Evolutionary Processes)

 

Evolution 2016 (ASN/SSE/SSB)

http://www.evolutionmeetings.org/evolution-2016—austin-texas.html

17 – 21 June 2016; Austin, Texas

Paula Mabee (DD); George Gilchrist, Paco Moore, Leslie Rissler, Sam Scheiner (Evolutionary Processes); Gordon Burleigh (Systematics and Biodiversity Science)

Event: NSF information session; Date: Monday, 20 June; Time: 12:00 – 13:00

 

Botany 2016

http://www.botanyconference.org/

30 July – 3 August 2016; Savannah, Georgia

Gordon Burleigh, Joe Miller & Simon Malcomber (Systematics and Biodiversity Science)

ESA Ecology 2016

http://esa.org/ftlauderdale/

7 – 12 August 2016; Ft Lauderdale, Florida

Alan Tessier (DDD); Doug Levey & Betsy Von Holle (Population and Community Ecology); Liz Blood, Henry Gholz & Karina Schäfer (Ecosystems); Janice Bossart (Evolutionary Processes); Cheryl Dybas (Public Affairs); John Adamec (Staff)

Booth: #333

Event: Funding Agency Information Session; Date: Monday, 8 August; Time: 11:30-13:15

 

North American Ornithological Conference 2016

http://americanornithology.org/content/north-american-ornithological-conference-2016

16 – 20 August 2016; Washington, DC

Doug Levey (Population and Community Ecology)

 

ecoSummit 2016

http://www.ecosummit2016.org/

29 August – 1 September 2016; Le Corum, Montpellier, France

Karina Schäfer (Ecosystems)

 

Entomology 2016 (XXV International Congress of Entomology)

http://ice2016orlando.org/

25 – 30 September 2016; Orlando, Florida

Janice Bossart (Evolutionary Processes)

Sprucing Up the Place


We’ve been blogging here on DEBrief since February 2013. Since our initial pilot, we’ve gotten noticed not just by you but by our colleagues here at NSF.  Once we paved the way, the rest of the BIO directorate joined in as well and now there are 5 NSF BIO blogs: one each for DEB, IOS, MCB, DBI, and the Assistant Director’s office. We are even integrated with the BIO twitter account (@NSF_BIO).

With all the enthusiasm here in BIO, and positive comments from you, for community outreach through these blogs, we’ve been given some new tools to work with. These will enable DEBrief and the other BIO blogs to look more like the collective cross-division effort that they are. So, over the next couple of weeks (and probably much longer) we’ll be trying out some changes to our look.

The most important part of DEBrief, our content, is not changing. We’ll still be bringing you the latest in funding opportunity updates, analysis of our programs, and discussions of the merit review and decision-making process.

The first change you’re likely to notice is a new custom domain name (debblog.nsfbio.com). The old URL: nsfdeb.wordpress.com will not go away. All existing links, etc. will still work. The idea is to add to this another more informative and easier to find address.

We are also looking to debut an update to our color scheme. We’re no longer limited by the color choices embedded in our theme selection, and are going to work on incorporating the NSF.gov colors.

Lastly, this is a great time for feedback. What works? What doesn’t? Are there parts of the blog site that you want to see improved? Is there something missing you want to see on DEBrief? Did we break anything in the process of implementing the updates? You are always welcome to leave a comment or send us an email at DEBQuestions@nsf.gov.