Credibility and use of scientific and technical information in policy making: An analysis of the information bases of the National Research Council’s committee reports

Abstract

Often researchers are disappointed by the limited extent to which peer reviewed STEM research seems to contribute directly to high level public policy decision-making. However, does the perception of the limited use of formal scientific and technical information (STI) accord with empirical reality? How does the choice of various types of information relate to the use and impacts of science policy reports and recommendations? While there is a prodigious literature on the use of formal information in decisionmaking, our focus is on the use of STI in science, technology and innovation (S&T) policy, a domain in which there is virtually no empirical literature. This study examines the use and impacts of STI in the context of a single, but arguably quite important, S&T policy domain: the US National Research Council (NRC) reports. This is an especially important target institution for analysis because NRC committees have extensive information access and resources, as well as decision-makers who are well equipped to deal with a variety of information types, including STI. To understand the information ingredients of high-level S&T policymaking and advice, we have coded information about the report, policy area, committee and reviewers, STI, and use of the report by Congress. Results indicate that STI widely used in the NRC report-writing process, but, although nearly half of all NRC reports are explicitly conveyed to Congress, STI use does not figure significantly in this conveyance. These findings imply different internal and external credibility orientations.

When Is Science Used in Science Policy? Examining the Importance of Scientific and Technical Information in National Research Council Reports

“…The NAS has not often attracted the interest of policy scholars. To our knowledge, our previous study (Youtie et al, ) is the only analysis of the NAS based chiefly on systematic quantitative data.…”

“…Our previous work (Youtie et al, ) included bibliometric analysis and content analysis of 589 NRC reports in order to determine the extent of use of STI in the reports, with STI defined as citations to refereed scientific and technical journals. To predict STI use, our study focused on various characteristics of the reports (e.g., size, year of publication), variables pertaining to the policy content areas, attributes of committee members (e.g., sectoral affiliation of committee member from committee biographies, sectoral affiliation of reviewer from reviewer organizational information), and nature of the authorization for the studies.…”

“…The authors’ own research, both in this study and a recently published companion publication (Youtie, Bozeman, Jabbehdari, & Kao, ) focuses directly on the topic of STI use in science policy. The earlier study, based on quantitative analysis of bibliometric content of National Academy of Sciences reports, focused on citations of scholarly articles, patents, and other National Academy reports.…”

“…The aim of the current study is to supplement these earlier findings with data from semi‐structured interviews. Since the interviews were conducted well after the quantitative results, they were informed by those earlier results and the interviews were, at least in part, designed explicitly to address some of the limitations and unresolved questions from Youtie et al (). Here, we are interested in addressing a variety of factors that may pertain to the degree of use of STI in NRC reports and also the extent to which NRC reports are used by policy makers.…”

“…Reflection from experience seems a warranted approach given the paucity of directly relevant research and the fact that previous experience with NRC work was one of the primary factors giving rise to this research project, including this paper. The second basis for informing the present research is our previous related study (Youtie et al, ). In the prior paper we explored some of the same topics but with very different data.…”