We’ve previously posted about what we are looking for in strong individual reviews of proposals. After receiving individual reviews, most proposals handled by DEB are brought to a panel meeting. After discussing a proposal, the panel prepares a document called the Panel Summary. In this post, we describe Panel Summaries, our goals in what we want them to communicate, and the steps DEB has recently taken to improve them.

What are Panel Summaries? (short version)

A Panel Summary is the written record of the review panel discussion of a proposal.

When you hear back from DEB about a proposal, you typically receive several documents in FastLane:

• all individual reviews (generally at least 3 from panelists and ad hoc reviewers),
• a context statement describing the program and review process employed,
• a Panel Summary and/or a Program Officer comment explaining the program decision.

The Panel Summary is the justification of the panel’s recommendation to the Program and to the PI. It is the most important document the PI receives. It acts as a bridge between the reviews and the panel’s recommendation, helping the PI to understand how and why the panel came to its decision.

What are we hoping to see in a well-written Panel Summary?

The single most important point to keep in mind for crafting a useful Panel Summary is that it needs to provide evaluative statements about a proposal and to justify those statements with specific details and feedback. However, this is not easy to do given that variation in proposals, reviews, panel discussions, and panelists’ writing styles all contribute to the Panel Summary. [This is why Panel Summaries are one of the items we’ve been monitoring and seeking to better manage and improve under the preliminary proposal system. More on this below.]

A good Summary is clear and concise in regards to the panel evaluation. It provides a consensus advisory statement from the panel to NSF about the merits of a particular proposal after consideration and discussion of all viewpoints. As with individual reviews, the panelists are asked to consider the proposal in light of the NSF merit review criteria and any additional criteria applicable to a specific program or funding opportunity. Program Officers and staff also provide feedback during the panel meeting to ensure that Summaries are complete and compliant with policies (e.g., confidentiality).

As far as style and approach, our previous advice on crafting individual reviews applies here too. But keep in mind that a Panel Summary differs from an individual review in that it is a summary of panel discussion, not of the individual reviews – This is another important point. The other major difference between an individual review and a Panel Summary is the context in which they are written. While your individual review is written prior to the panel, alone, and in an environment of your choosing, Panel Summaries are written in the midst of a panel with several other panelists who must sign-off on the final product providing advice.

What steps are we taking to encourage useful Panel Summaries?

For many years now, DEB has been providing panelists with a template for completing Panel Summaries. Ongoing evaluation has motivated us to modify the template and provide additional instructions to panelists on the purpose of a Panel Summary.

The purpose of the new template is to provide greater clarity for both the panelists and the PIs as to what we expect to see in each of the sections of a Summary.

DEB Panel Summary Template provided to panelists during Fall of 2015.

The new template features familiar headings that outline the major points to be considered by the panel and couples those with brief prompts (in red italics) that are intended to be kept in the document so that both panelists and PIs will have constant reminders of what we are asking of panelists in each section of the template.

In addition to this template, which will be provided to each panelist as a document file, panelists will also receive hard-copy guidance documents for their panel work-spaces that reiterate our verbal instructions about writing strong and complete Panel Summaries. This guidance document includes short example phrasings, and call-out boxes to highlight common issues with content, style, and formatting. You can read it for yourself below:

DEB Panel Summary Guidance handout for Fall of 2015

By putting these documents out here, we are hoping you, our community members, who are both our PIs and panelists, can be partners with us in maintaining awareness of what we are looking for in high quality Panel Summaries. We think establishing clarity and mutual understanding of the role of panel summaries before you find yourself in panel or receiving a decision from a Program Officer will contribute toward a culture that demands and provides high quality review documentation for everyone.