Our last post, tracing the fate of proposals over the two-stage review process, worked with data that had been pulled together for our Committee of Visitors (COV) in 2015. For those of you unfamiliar with NSF COVs, the short version:

A COV is a special type of panel established, in our case, under the Advisory Committee for BIO. Instead of reviewing proposals, however, the COV reviews the entire merit review and award process. They look at individual reviews and panel summaries provided by you, our management of the merit review and decision-making process, and the resulting portfolio of awards. Every program at NSF undergoes a COV review every three years.

We’ll leave the in-depth discussion of the COV process to another time, but thought you would be interested in some related follow-up.

The DEB COV met last June. It was chaired by Dr. Steward Pickett and the Advisory Committee for BIO was represented by Dr. Paul Turner. After the committee completed its work, a report was transmitted to the BIO Advisory Committee for approval.

Dr. Turner presented the 2015 DEB COV report to the NSF Advisory Committee for BIO at their fall meeting (29 Sept. 2015). The report, and a response document prepared by BIO to outline our planned follow-up actions, were accepted by the Advisory Committee. Both documents have been published to the nsf.gov website. The COV report, and BIO response are both available for download and sharing.

The COV report endorsed our plan to formally evaluate the preliminary proposal process in DEB and IOS via a third-party provider. At the time of the COV we were working to develop a solicitation for a contractor to carry out the work. We were successful in soliciting several bids^[i]. After review of the bids, a 1-year contract was awarded to Abt Asssociates out of Cambridge, MA to conduct a study. This organization has extensive experience as third-party program evaluators for government clients and scientific research programs. The contract began in March, 2016 and is well underway. The project team is presently developing the necessary survey instruments and beginning to work with program data. The project is scheduled for completion by February 2017. We look forward to sharing the results with you as we are able.

We plan to post additional updates to DEBrief as project milestones are reached. However, this is also a first heads-up that Abt may reach out to you to by email to participate in the evaluation. Please make sure your FastLane contact info is up to date so you don’t miss it.

In the interim, the COV report also covered the first two years of preliminary proposals and might be of interest.

[i] The federal contracting process is a bit like the proposal process but operates under a much more extensive set of regulations.