Those of you who have served on a BIO panel at any point in the last few years may recall being informally polled by representatives of the BIO Directorate leadership about your feelings, interest, and concerns about conducting panel meetings by teleconference, web-conference, video-conference or some combination thereof dubbed “virtual panels”. And, some of you may even have served on a virtual review panel: DEB has hosted a few, IOS has hosted several more, and various interdisciplinary programs have made use of them too.
A recent article from AIBS by Gallo, Carpenter, and Glisson (2013) compares measures of the review process (reviewer recruitment and panel discussions) and panel outputs (review scores) of in-person and virtual peer review panels for grant review process managed under contract by AIBS, finding “few differences are evident” between the two.
Press Release here: http://www.aibs.org/peer-review/news/study-explores.html
Full article here: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0071693
We wanted to share some of our own observations and feedback we’ve already received from you. Our goal is to start a discussion on this topic.
Upsides of Virtual Panels
- Expands panelist pool to those who cannot or do not wish to travel (thus, potentially more inclusive)
- No travel cost to NSF (incl. expenses of travel and time making arrangements)
- No time lost traveling by panelists
- Smaller carbon footprint
- Smaller physical footprint (fewer large meeting rooms needed at NSF)
Downsides of Virtual Panels
- No informal interaction with other panelists about things other than panel (e.g., at dinner or in hallway chats)
- Lower quality opportunities (though not fewer) for panelists to learn about happenings at NSF and to provide feedback to management
- Remote panelist locations are often in a distraction-heavy environment; and, it can be difficult to locate a panelist who is delayed in returning from a break
- Longer meetings: Pace of review and discussion slower, less incentive to focus and finish
- Managing conflicts of interest is more difficult when you can’t just step out of the room
- No ability for subgroups of panelists to caucus with a program officer about proposal details – lowers quality of panel deliberations
- Harder to feel like an integral part of an important process when you’re sitting at your same ol’ desk, like you always do
- Size: It wasn’t examined in the above study, but the general feeling (boosted by non-panel experience with the various technologies) is that there are ceilings on the number of panelists (roughly 12) that can come together virutally and how long they can convene at a stretch (approx. 1-2 full days or shorter sessions over a longer period). This necessitates fewer proposals per panel. In some fields there may be sufficient subdivisions within a program to make this work, but it also means less calibration of ratings across the several panels. DEB proposals often cross several such subdivisions making them difficult to implement and the repeated collection of sufficient expertise in small panelist groups even more difficult.
- Non-verbal communication is lost: Even with video, voice, and screen-sharing, some aspects of interpersonal communication are not replicated; however, might this be a good thing in certain circumstances?
- Technology: The more “interactive” the system used (for instance, online meeting space with video and voice), the greater the front-loaded effort to get all participants up to speed, the greater connection requirements for the remote participant, and the greater chance for disruptions compared to lower tech options (for instance, a conference phone line). However, we’ve heard reports that support for the lower tech means (e.g., landlines in faculty offices, desktop speakerphones) is beginning to disappear.
We’re interested in hearing what other considerations come to mind based on your thoughts about or experience with virtual meetings and how that might be good, bad, or just something to think about for panels.