Quantifying and Assessing Trends on the National Science Foundation's Broader Impact Criterion

Abstract

The American Innovation and Competitiveness Act (S.3084) reapproved the National Science Foundation's (NSF) merit review criteria i.e. Intellectual Merit and Broader Impacts, called for an update of the policy guidelines for NSF staff members and merit review process participants, and emphasized the importance of transparency and accountability. Evaluating Project Summaries based on Intellectual Merit and Broader Impacts has been the standard of maintaining excellence and accountability since 1997. Intellectual merit consists of proposing activities that advance knowledge, while Broader Impacts statements, "describe the potential of the proposed activity to benefit society and contribute to the achievement of specific, desired societal outcomes." While Intellectual Merit has been widely understood since its inception, the Broader Impacts criterion has undergone continuous discussion among Principle Investigators (PI), panel reviewers, NSF Committees of Visitors, and the broader scientific community.To better understand Broader Impacts and how they could be quantified, I conducted an exploratory study during a 10-week internship in the Engineering Education and Centers (EEC) division of the Engineering (ENG) Directorate at NSF. This study used a convergent parallel mixed method design, collecting both quantitative and qualitative data, simultaneously, to answer two research questions 1) What trends are Program Officers seeing in the Broader Impacts criterion and 2) Which Broader Impacts statements are being addressed in Project Summaries submitted to the National Science Foundation.The quantitative approach consisted of examining 82 awarded Project Summaries in the EEC division to obtain a quantifiable assessment of the extent to which PIs who applied to EEC addressed the Broader Impacts suggestions outlined in NSF's Proposal and Award Policies and Procedures Guide. The qualitative approach involved interviews of four program officers from the EEC division regarding the trends in addressing the Broader Impacts criterion. The data were collected in parallel strands, independently from each other, and were brought together for comparison.Results from the examined abstracts indicate that PIs were more likely to have Broader Impacts statements around increasing public scientific literacy, public engagement in science and engineering, and addressing issues of developing a diverse STEM workforce. While the interview data from Program Officers indicated that PIs were more inclined to write "pie in the sky" statements around Broader Impacts. Program Officers also indicated that reviewers tend to place more weight on Intellectual Merit than Broader Impacts even though they are informed to put equal weight in both criteria. There exists no quantitative metric for measuring or assessing Broader Impacts statements PIs propose in their NSF awarded grants. This study is an exploratory attempt to unpack what is currently being funded using awarded Project Summaries and outline tensions around addressing Broader Impacts by propo...

Analysis of Panel Summaries of Proposals Submitted to the S-STEM Program

“…Comments about the approach and project plan also tended to differentiate awarded from declined proposals; although positive aspects of both awarded and declined proposals were noted, reviewers pointed out a number of disadvantages for declined proposals (and none for awarded proposals). While for many NSF programs, reviewers tend to more strongly emphasize intellectual merit than broader impacts, indicating a better understanding of or connection to that criterion [8], the panel summaries we examined attended more strongly to broader impacts. We believe this is due, in part, to the nature of the S-STEM program and its emphasis on improving outcomes for students from low-income backgrounds.…”