Over the past several years, DEB program officers have examined the costs and benefits of our review process in the face of increasing proposal submissions, increasing project costs, and flat program budgets. During this time the success rate of proposals has steadily declined. We recently reached the point where, for most of our programs, semi-annual review cycles with hundreds of full proposals competing for the limited pool of funds were unsustainable. In this posting, we will lay out the three top issues that concerned the DEB staff.

1) The decline in the quality of review. The heavy burden of submissions to the core programs led to a steady decline in the quality of review on every front. Prior to the pre-proposal process, program officers were generally handling 40-60 projects per review cycle. Few of us were able to find the time to carefully read most of those proposals, which meant that we were less informed when debating our decisions. Second, securing at least two ad hoc reviews by subject experts outside of the panel generally required 6 or more requests, which translated into >12,000 requests per year. Thus, the burden on the reviewer community was heavy and system was stressed to the point of breaking; completed reviews were relatively few and far between.  Finally, with 140-180 proposals per panel, discussions had to be short and panelists were pressed to provide thorough panel summaries. Overall, everybody was working hard, but most program officers felt that we did not have enough time and information to make the careful decisions that our limited budget required.

2) A community working harder than ever for less money. A second concern was about the effort that beginning faculty members were exerting in trying to compete for the small pool of available funding. We routinely heard about assistant professors investing time in meticulous preparation of full proposals at the expense of writing manuscripts — an especially thorny tradeoff, given that the pressure to write proposals seemed to rise inversely with the probability that those proposals would be funded. We heard from many PIs that a decline at the pre-proposal stage would open up several months that could be spent collecting data and writing papers before the next deadline, hopefully making PIs more competitive for the next round of pre-proposal submissions.

3) The churn at the top. All DEB programs operate on an annual budget. Until 2011, all DEB core programs managed two deadlines per year by dividing that annual budget into roughly two halves, one for a fall panel and one for a spring panel. By necessity, that meant that about half the number of projects was funded from each panel as under the current system of a single deadline. With funding rates under 10%, there were and continue to be far more highly ranked proposals than can be funded. From 2008-2011, approximately 1/3 of DEB projects were funded on first submission, and approximately 1/3 were funded on second submission. In some clusters, the majority of those funded on second submission were highly ranked proposals that were declined six months earlier, largely due to lack of funds. Thus a downside of maintaining two funding deadlines was a delay in funding of top ranked proposals by forcing a second submission, even if few revisions were required.

Rapid resubmission for proposals that required more substantial revision was not common. Overall, 85% of DEB resubmissions came in one year or more after their first debut. Thus, in the move to a single annual deadline, we have tried to reduce the unnecessary resubmission of highly ranked proposals, with the trade-off in reduced flexibility in deadlines.

DEB permanent staff and rotators discussed changes in the review cycle over approximately five years. During the last two years, large volumes of data about our PIs and their submissions were analyzed and models were made to anticipate costs and benefits of many different alternatives. The current pre-proposal system is the result of those considerations. Overall, our first year with the new system has gone fairly smoothly, but DEB continues to collect a variety of data that will be used to examine possible consequences of the current system. These data will be discussed at length in future posts to DEBrief. We would welcome ideas from the community — and discussion of those ideas on this blog — that might better balance the trade-offs among various submission schemes.

DEB Science Staff