Assessing Research Methodology

Abstract

This paper reports the development and application of a method for evaluating the methodology of social science research. In this study, 11 social scientists rated the methodology of 126 federally sponsored, social science research projects. Written documentation from each research project was divided into three packets, representing three stages of the research process: proposal, interim report, and completed research. Each project was independently rated at each of the three stages by two different reviewers, using a comprehensive methodology review instrument created for this purpose. A series of factor analyses of the ratingsfor 75 evaluative methodological features reveals a multidimensional structure underlying the evaluations. This structure appears to become more elaborate as research progresses toward completion. The implications of the structuring of methodological assessments are discussed for research funding procedures, research monitoring, the conduct of research, and the training of social scientists. basic tenet of science holds that the procedures by which a conclusion is reached determine whether that conclusion should be treated as demonstrated and generalizable knowledge. ~ ~ -Assessment of research procedures is largely a normative matter, depending upon currently accepted standards and practice.