The use of naturalistic narrative data is increasing but without increasing methodological rigor. Assumptions underlying such methods prohibit an investigator from interrupting the natural habitat by use of such standard laboratory procedures as equal observation intervals for all subjects, equal behavior units, or other artificial equalization of the behavioral data stream. Data analytic techniques commonly used in standard laboratory research present problems in the analysis of naturalistic data of this type inasmuch as most assume equal observation intervals, equal unit intervals, or some other fundamental character of equality. These problems are particularly acute with respect to estimates of analyst and observer agreement. This paper addresses these problems and discusses the advantages and disadvantages of several possible solutions.The collection and analysis of naturalistic observational data present some unique problems in determining observer and coder reliability and agreement. A number of these problems, especially as they relate to category observation schedules, have been extensively reviewed have also addressed these problems as they relate to ethnographic research. However, the particular problems surrounding determination of observer and analyst agreement in narrative and observational records have not been well explored. Furthermore, the use of naturalistic narrative data is increasing without concomitant increase in methodological rigor.The purposes of this paper are (a) to describe problems that researchers encounter when working in this area, (b) to identify various solutions that have been used, (c) to report results of some recent studies of observer and analyst agreement, (d) to propose alternate strategies for calculating analyst agreement, and (e) to compare and contrast this strategy with others previously used. The focus of this paper is on interobserver and interanalyst agreement. The issues raised and the methods discussed, however, would be equally applicable to intraobserver and intraanalyst agreement. PROBLEMS IN DETERMINING OBSERVER AND ANALYST AGREEMENTThe methodology associated with narrative observation provides a framework for considering the unique difficulties encountered with this type of observation in estimating Requests for reprints should be sent to M. M. Scott,
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
customersupport@researchsolutions.com
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.