Those of you who have served on a BIO panel at any point in the last few years may recall being informally polled by representatives of the BIO Directorate leadership about your feelings, interest, and concerns about conducting panel meetings by teleconference, web-conference, video-conference or some combination thereof dubbed “virtual panels”. And, some of you may even have served on a virtual review panel: DEB has hosted a few, IOS has hosted several more, and various interdisciplinary programs have made use of them too.
A recent article from AIBS by Gallo, Carpenter, and Glisson (2013) compares measures of the review process (reviewer recruitment and panel discussions) and panel outputs (review scores) of in-person and virtual peer review panels for grant review process managed under contract by AIBS, finding “few differences are evident” between the two.
Press Release here: http://www.aibs.org/peer-review/news/study-explores.html
Full article here: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0071693
We wanted to share some of our own observations and feedback we’ve already received from you. Our goal is to start a discussion on this topic.
Upsides of Virtual Panels
- Expands panelist pool to those who cannot or do not wish to travel (thus, potentially more inclusive)
- No travel cost to NSF (incl. expenses of travel and time making arrangements)
- No time lost traveling by panelists
- Smaller carbon footprint
- Smaller physical footprint (fewer large meeting rooms needed at NSF)
Downsides of Virtual Panels
- No informal interaction with other panelists about things other than panel (e.g., at dinner or in hallway chats)
- Lower quality opportunities (though not fewer) for panelists to learn about happenings at NSF and to provide feedback to management
- Remote panelist locations are often in a distraction-heavy environment; and, it can be difficult to locate a panelist who is delayed in returning from a break
- Longer meetings: Pace of review and discussion slower, less incentive to focus and finish
- Managing conflicts of interest is more difficult when you can’t just step out of the room
- No ability for subgroups of panelists to caucus with a program officer about proposal details – lowers quality of panel deliberations
- Harder to feel like an integral part of an important process when you’re sitting at your same ol’ desk, like you always do
Other Concerns/Trade-offs
- Size: It wasn’t examined in the above study, but the general feeling (boosted by non-panel experience with the various technologies) is that there are ceilings on the number of panelists (roughly 12) that can come together virutally and how long they can convene at a stretch (approx. 1-2 full days or shorter sessions over a longer period). This necessitates fewer proposals per panel. In some fields there may be sufficient subdivisions within a program to make this work, but it also means less calibration of ratings across the several panels. DEB proposals often cross several such subdivisions making them difficult to implement and the repeated collection of sufficient expertise in small panelist groups even more difficult.
- Non-verbal communication is lost: Even with video, voice, and screen-sharing, some aspects of interpersonal communication are not replicated; however, might this be a good thing in certain circumstances?
- Technology: The more “interactive” the system used (for instance, online meeting space with video and voice), the greater the front-loaded effort to get all participants up to speed, the greater connection requirements for the remote participant, and the greater chance for disruptions compared to lower tech options (for instance, a conference phone line). However, we’ve heard reports that support for the lower tech means (e.g., landlines in faculty offices, desktop speakerphones) is beginning to disappear.
We’re interested in hearing what other considerations come to mind based on your thoughts about or experience with virtual meetings and how that might be good, bad, or just something to think about for panels.
DEBrief 
Some other potential downsides to virtual reviews are:
1. Lack of physical presence reduces ability of reviewers to maintain attention to speaker, understand nuances, inflections, etc. Videoconferencing may mitigate this to some degree but language barriers are real and the even best technology leaves much to be desired in terms of atmosphere.
2. It is easy to spot someone in a room goofing off (checking email, snoozing, etc.) while this is much more difficult virtually. Even if someone appears engaged, they may be distant and effectively out of touch.
3. While this may not be the case with all reviewers, there is a level of performance anxiety (a good thing) insofar as when you are speaking to a group, you need to be organized and well prepared. Discombobulated interactions may reduce the level of preparedness.
4. Related to the other points, it is often easier to commit 2 or 3 days of full time, intense, all-on, clear-the-decks attention when travel to an off-site meeting is entailed than to carve out time in your own office. There is always the temptation to multi-task.
As an aside, the Canadian Institutes of Heath Research are proposing an all-out switch to virtual reviews next year (with a College of Reviewers). See: http://www.cihr-irsc.gc.ca/e/44761.html
This is causing significant concern among the Canadian research community for some of these reasons as well as some mechanistic changes (such as collecting 5 reviews per application, assignment of reviews based on key-words, etc).
Thanks for the comment. Point 3 especially is one we haven’t heard put that way before.
A question: What is the average cost of a panel, and where does the money come from? How would the savings (if any) be used?
Panels are paid for out of the same source as awards and are about 1% of that budget. Panel costs are mainly the cost of transporting the panelists to NSF and the panelist per diem: which covers meals, lodging, and an honorarium. So the cost of a panel is dependent on the number of panelists, which is in turn dependent on the number of proposals and how they are distributed for review. Estimating the potential savings is difficult because of the factors noted in the post. We would save on the travel costs and meals and lodging, but remote panelists still receive a reduced per diem. Virtual panels may need to be longer and broken into a greater number of smaller meetings, generating cumulatively more panelist-days, which would cut into those savings.
I have not done a virtual panel myself, but have talked to half a dozen or so people who have for NSF and probably twice that who have for other agencies. I have yet to anyone describe the experience in a positive light. The common theme I hear is the juggling of “home” and “away” at the same time makes the task more difficult. In many ways it is not much different from having a scientific conference in your city – instead of playing one role (panelist, conference goer) you are forced to wear multiple hats.
Maybe you skip an hour to teach or have to leave at a certain time to pick up your kids. Maybe something in the lab pulls you away for a little bit here or there. In the evenings there are the “home” tasks you need to do, rather than a hotel to retreat to in order to review for the next day. I’m not saying it is impossible to commit to the panel when joining virtually, but I can’t imagine a scenario in which you will get the same commitment from 10-20 people that you do when they are physically in DC. Add that to the lack of interaction in the room and I think you have a significantly less engaged panel.
I would be interested to hear from the POs whether they observe a difference in the number of people willing to weigh in on a discussion for which they have not reviewed the proposal. This is one (indirect) measure of engagement at the table. I’ll admit that I am terrible at ignoring the discussion and working on my summaries, but mostly because I enjoy hearing about the science and am willing to ask questions of the other reviewers to help me put my proposals into the bigger context of the panel. If I were on a virtual panel I think it would be far easier to turn down the volume when I’m not a reviewer and bang out my summaries so I can spend my evenings with my family.