Caveat: This post is based on the research and analysis of Kara Shervanick, a 2013 Summer Student in DEB. She did valuable work but her time was relatively brief for this complex information gathering and analysis process. This work does provide some context for understanding DDIG program outcomes, however, we point out that the small sample size limits the power of these analyses.

See our other recent posts on the DDIG program here and here.

Summary

Receipt of a DDIG award does not appear correlated with remaining in research oriented careers over time. Having a DDIG award, however, is associated with a more successful research-oriented career for the success metrics of: number of additional NSF grants, total number of publications and citations, and h-index. The sampled data did not provide clear evidence for gender associated with the success of awarded or declined applicants. In the limited instances of apparent differences associated with gender, no clear mechanism emerges.

Study Background

In the summer of 2013 we conducted a retrospective analysis of the DEB DDIG program, making use of the dedicated effort of a temporary summer student intern. The goal of the project was to see if there were discernable correlations between DDIG award receipt and subsequent career trajectories.

Our student researcher set out to collect and analyze DEB and publicly available data to address the following questions about the DDIG applicant pool:

(i)            Does receiving a DDIG award correlate with who leaves or remains in a research-oriented career?

(ii)           Does gender correlate with who leaves or remains in a research-oriented career?

(iii)          Does having a DDIG correlate with a more successful research career?

(iv)         Of those in research-oriented careers, does the potential effect of receiving a DDIG award differ based on gender?

Since the total number of DDIG applicants is quite large, the data difficult to assemble, and the time in which to complete the project limited, the scope of the data collection and analysis was limited to samples of the DDIG student applicants from 5 (2008), 10 (2003), 15 (1998), and 20 (1993) years ago.

The percentage awarded each year ranged from 28.6% to 32.5% and averaged 30.8% over the four years, in line with the 1983-2013 average noted in our previous post.

There was no apparent gender bias in the awards and female students were represented in the awards in proportion with the proposal pool in each year.

Success, of course, is subjective. However, for the purposes of this study, we looked at several imperfect proxies: future awards, and publication and citation metrics.

Sample Selection

For each year, 30 awardees were randomly selected from the list of students receiving a DDIG; these comprised the “awarded” samples.

With any such retrospective analysis, there is the problem of distinguishing correlation for causation. Receiving a DDIG award might improve the career of an individual, or the most talented people might receive awards and would have gone on to have more successful careers had there been no DDIG program. To minimize this potential bias, the “declined” comparison group was not random but consisted of the 30 declined proposals with the highest average reviewer score in each year. Students who later received DDIG support on a resubmission were excluded from consideration in the declined group for any year. Thus the mean score of the proposals in the selected declined group is higher than the mean for all declined proposals and is nearer the mean score of the awarded samples than the bulk of declined proposals.

For each of the 240 student applicants (sample of 30 across two outcomes and four years), NSF award search, Web of Knowledge, publicly available webpages, and direct contact were used to identify: additional number of NSF grant awards, total number of publications, total number of citations, citations per paper, h-index, and current position and institution or company.

Does a DDIG award correlate with retention in research-oriented careers?

Position title and employer were considered in assigning each individual’s current position to a “research” or “non-research” category. Except for a few individuals who couldn’t be found or left the workforce entirely, nearly all of the people who left research are still in the broader environmental and scientific workforce as educators and private sector professionals.

Award status did not appear correlated with whether or not an applicant remained in a research-oriented career in any of the reviewed years. However, we do see a consistent pattern at 10, 15, and 20 years which suggests some small but real correlation with retention in research careers may be present. And, though the data doesn’t directly address this, losses from the research pool seem to stop after 10 years post-PhD.

Is applicant gender correlated with retention in research-oriented careers?

Only one year, 1993, showed a significant difference in this respect. There is no clear explanation for why 1993 is so different from the other years for gender differences. There were fewer females participating in the DDIG program in 1993 compared to the later years, so it is more sensitive to small differences. However, none of the women in this cohort were missing data on current position and the positions were similar (private industry, government, education, administration) to those occupied by both men and women in non-research careers.

One thing we don’t see in the samples is an exodus of female scientists from the workforce entirely (regardless of funding outcome). For the most part, neither the men nor women in our sample seem to have given up research.

Does receiving a DDIG correlate with a more successful research career?

Within the limits of this analysis, it appears so.

For all measures at 10, 15, and 20 years awardees appear to have more success than those who applied for but did not receive DDIG funding. There definitely is an association between DDIG funding and positive research career outcomes.

Although we generally see larger differences in the older cohorts, this data doesn’t tell us how, if at all, these differences grow over time for individual year groups or whether the trajectories are changing from year to year. However, the small differences between awarded and declined students in the 2008 cohort suggests to us that both groups are starting from the same point and the DDIG program is not simply selecting those already “ahead in the game” of academic output.

On the flip side, the lack of differences at the 5-year mark could be seen as supporting the idea that the program simply selects those who would have done well in research without the DDIG program rather than providing a direct early career boost.

Of those in research-oriented careers, does the potential effect of receiving a DDIG award differ based on gender?

For those remaining in research, there were few detectable effects of funding outcome and gender on career metrics. None were particularly surprising. In general, for both genders, awardees were more successful than declined students and in many of the metrics males had a slight advantage over females for the year and funding outcome. However the differences were neither sufficiently large nor consistent to suggest real relationships between gender, DDIG award status, and career outcomes.