A methodology for evaluation of knowledge-based systems in medicine

Clarke, Kevin A.; O’Moore, R. R.; Smeets, R.; Talmon, Jan L.; Brender, Jytte; McNair, Peter; Nykänen, Pirkko; Grimson, Jane; Barber, Barry

doi:10.1016/0933-3657(94)90040-x

Cited by 41 publications

(14 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This would be very significant, because exact risks can then be calculated for the deployment of an intelligent medical system in clinical use. Numerator (11) where is the probability at the lower boundary of the element, and is the probability at the upper boundary of the element. Numerator (12) Dealing with one partial beta function at a time, where either or can be substituted for Applying the binomial expansion (13) Change the limits on the integral from zero to , to zero to one, by letting , which implies , and letting…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Receiver operating characteristic analysis for intelligent medical systems-a new approach for finding confidence intervals

Tilbury

Eetvelt

Garibaldi

et al. 2000

IEEE Trans. Biomed. Eng.

View full text Add to dashboard Cite

Intelligent systems are increasingly being deployed in medicine and healthcare, but there is a need for a robust and objective methodology for evaluating such systems. Potentially, receiver operating characteristic (ROC) analysis could form a basis for the objective evaluation of intelligent medical systems. However, it has several weaknesses when applied to the types of data used to evaluate intelligent medical systems. First, small data sets are often used, which are unsatisfactory with existing methods. Second, many existing ROC methods use parametric assumptions which may not always be valid for the test cases selected. Third, system evaluations are often more concerned with particular, clinically meaningful, points on the curve, rather than on global indexes such as the more commonly used area under the curve. A novel, robust and accurate method is proposed, derived from first principles, which calculates the probability density function (pdf) for each point on a ROC curve for any given sample size. Confidence intervals are produced as contours on the pdf. The theoretical work has been validated by Monte Carlo simulations. It has also been applied to two real-world examples of ROC analysis, taken from the literature (classification of mammograms and differential diagnosis of pancreatic diseases), to investigate the confidence surfaces produced for real cases, and to illustrate how analysis of system performance can be enhanced. We illustrate the impact of sample size on system performance from analysis of ROC pdf's and 95% confidence boundaries. This work establishes an important new method for generating pdf's, and provides an accurate and robust method of producing confidence intervals for ROC curves for the small sample sizes typical of intelligent medical systems. It is conjectured that, potentially, the method could be extended to determine risks associated with the deployment of intelligent medical systems in clinical practice.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Substituting (15) into (12) and then substituting into (11) and simplifying Numerator (16) Substituting (16) into (3) gives the expression for each element of the vector, as shown in the equation at the top of the next of the page, which simplifies to (6). Similarly, for [by substituting (16) into (4) and simplifying to give (7)].…”

Section: Discussionmentioning

confidence: 99%

Receiver operating characteristic analysis for intelligent medical systems-a new approach for finding confidence intervals

Tilbury

Eetvelt

Garibaldi

et al. 2000

IEEE Trans. Biomed. Eng.

View full text Add to dashboard Cite

show abstract

“…The need to evaluate the system during its development stems from the fact that the results of the evaluation of each stage of the software development are used to improve development on the next cycle [17]. This results in a continuous improvement of the quality of the system.…”

Section: Evaluating the Technical Aspectsmentioning

confidence: 99%

Evaluation of the Decision Support Systems

Jallal¹,

Mouna²,

Naima³

et al. 2017

JCC

View full text Add to dashboard Cite

MDSSs (medical decision support systems) are computer applications providing clinicians with suitable information about a patient's clinical situation, as well as knowledge relevant to the situation, appropriately filtered and presented, to improve the quality of care and the patients' health. As is the case with any intervention aiming at influencing medical practices and their impact on the patients' health, it seems appropriate to question their real value through a methodology for reliable evaluation. Evaluation methods are necessarily dependent on the questions raised and it is common for different questions to lead to different methodologies and different processes of data collection and analysis. In the light of this, better consideration and better implementation of the evaluation processes of these systems are major criteria on which their future depends.

show abstract

“…Furthermore, these conventional systems have generally provided neither remote video-monitoring functions for the mobile handheld devices of caregivers, nor a customized and configurable framework for intelligent home healthcare and video-monitoring systems. To provide the flexibility and extensibility for various demands on intelligent healthcare systems, knowledge-based system technologies have demonstrated the advantages of extensibility and flexibility for a wide variety of practical applications [22,23]. Consequently, many researchers have recently applied the concepts of knowledge-based systems to many applications, such as scene analysis [24], document analysis for office automation [25,26], automatic traffic monitoring [27], facial analysis for human-computer interaction [28], clinical decision supporting systems [29,30], and healthcare information systems [31,32].…”

Section: Introductionmentioning

confidence: 99%

An Intelligent Knowledge-Based and Customizable Home Care System Framework with Ubiquitous Patient Monitoring and Alerting Techniques

Chen

Chiang

et al. 2012

Sensors

View full text Add to dashboard Cite

This study develops and integrates an efficient knowledge-based system and a component-based framework to design an intelligent and flexible home health care system. The proposed knowledge-based system integrates an efficient rule-based reasoning model and flexible knowledge rules for determining efficiently and rapidly the necessary physiological and medication treatment procedures based on software modules, video camera sensors, communication devices, and physiological sensor information. This knowledge-based system offers high flexibility for improving and extending the system further to meet the monitoring demands of new patient and caregiver health care by updating the knowledge rules in the inference mechanism. All of the proposed functional components in this study are reusable, configurable, and extensible for system developers. Based on the experimental results, the proposed intelligent homecare system demonstrates that it can accomplish the extensible, customizable, and configurable demands of the ubiquitous healthcare systems to meet the different demands of patients and caregivers under various rehabilitation and nursing conditions.

show abstract

A methodology for evaluation of knowledge-based systems in medicine

Cited by 41 publications

References 9 publications

Receiver operating characteristic analysis for intelligent medical systems-a new approach for finding confidence intervals

Receiver operating characteristic analysis for intelligent medical systems-a new approach for finding confidence intervals

Evaluation of the Decision Support Systems

An Intelligent Knowledge-Based and Customizable Home Care System Framework with Ubiquitous Patient Monitoring and Alerting Techniques

Contact Info

Product

Resources

About