An Inventory of Evaluation Studies of Information Technology in Health Care

Abstract

We found interesting developments in evaluation research in the last 20 years. For example, there has been a strong shift from medical journals to medical informatics journals. With regard to methods, explanatory research and quantitative methods have dominated evaluation studies in the last 20 years. Since 1982, the number of lab studies and technical evaluation studies has declined, while the number of studies focusing on the influence of information technology on quality of care processes or outcome of patient care has increased. We interpret this shift as a sign of maturation of evaluation research in medical informatics.

Factors of accepting pain management decision support systems by nurse anesthetists

“…The organizational barrier involves the following: (1) Organizational structure is a tool and instrument used by enterprises to achieve their objectives and to dominate and coordinate decision-making activities [53], and it may include type and size (number of beds), culture, politics, hierarchy, autonomy, planning and control systems, strategy, management and communications, leadership, top management support, and medical staff sponsorship [42], and (2) organizational environment, in the context of the environment in which the organization exists, includes external competition, influence of government policies, characteristics of target audience, and source of funds required for unit operations [42,54,55]. Kaplan [56] argued that the nature of a healthcare institution can be examined from its structure and environment.…”

Design and Evaluation in eHealth: Challenges and Implications for an Interdisciplinary Field

“…For example, controlled trials may be ideal for studying the impact of eHealth systems on measures of clinical outcome or efficiency, but they are poorly suited to exploring social, contextual, or technical barriers to adoption and certainly will have little to offer developers designing a new Web interface. Conversely, think aloud methods may be extremely useful for assessing the usability of a decision support tool but say very little about its clinical validity or effectiveness [46-48]. The value attributable to different forms of evidence thus varies depending on the context in which it is used, although adherence to high standards of data collection, analysis, and reporting is a universal objective.…”

“…The move toward more holistic training in medical informatics advocated by bodies such as the American Medical Informatics Association represents one step to achieving these goals [5,9], and there is evidence of a trend toward increasing pluralism in the objects of evaluation projects, which may signal a move toward greater interdisciplinarity [48]. Pockets of transdisciplinary working are emerging as eHealth becomes a target of research, for example, within academic units of human-computer interaction and science and technology studies, while the field of information science has a long tradition of research exploring socioeconomic and organizational influences on technology development and adoption, from which eHealth researchers and developers have much to learn [49].…”

Attending unintended transformations of health care infrastructure

“…Further, “the study of failures, partial successes, and changes in project definition or outcome” are recommended for future research [ibid]. Ammerwerth et al confirm maturation in evaluation research in medical informatics as studies of more technical aspects of implementation have declined and shifted to an increase in studies on the quality of care processes and patient outcomes [15 p.55]. Leroy and Beuscart advocate the importance of usability evaluation by including final users in real settings in order to detect severe usability problems [16].…”