Using the Critiquing Approach to Cope with Brittle Expert Systems

Abstract

We have conducted a series of studies aimed at understanding how to design cooperative problem-solving systems to deal with situations in which the computer is not fully competent (the "brittleness" problem). Results from an initial empirical study showed that an expert system acting as an automated assistant induced new errors in such situations, but that this did not occur when the system was designed to critique user's performance on the task. Guided by these results, a more complete critiquing system was then designed and evaluated. On cases where the computer was fully competent, outcome errors were completely eliminated (a 30-60% reduction in errors). On a case for which the system was brittle (less than fully competent), misdiagnosis rates were still reduced by 31%, giving an overall significance of p < 0.000003 across all Post-Test Cases, further supporting our initial study regarding the potential value of critiquing as an effective role for an expert system. A detailed analysis of the influence of performance further indicates the ways in which different classes of critics detect errors and influence the user's problem-solving. Man-Machine St&, 2, 429-453. Kennedy, M. (1991). Errors in antibody identification. h m u n o h e w , Z, 20-20. complexities of multiple-solution problems: A case study.

Task “Versus” Work Domain Analysis Techniques: A Comparative Analysis

“…• Strategy Guides: explicit templates for selecting an appropriate water or heat input strategy, accompanied by an overlay on the interface to guide control actions in keeping with that strategy and/or intelligent 'critics' (Guerlain, 1995) to aid in strategy implementation. • Expectation Indicators: The prevalence and importance of the expected value for a parameter given system state, strategy and goal, implies that marking expectations explicitly might be helpful.…”

Operator Object Function Guidance for a Real-Time Unmanned Vehicle Scheduling Algorithm

“…Though fast and able to handle complex computation far better than humans, computer optimization algorithms are notoriously 'brittle' in that they can only take into account those quantifiable variables identified in the design stages that were deemed to be critical [19,20]. In a command and control situation such as supervising multiple UVs-where events are often unanticipated-automated planners have difficulty accounting for and responding to unforeseen problems [21,22]. Additionally, operators can become confused when working with automation, unaware of how the 'black box' automated planner came to its solution.…”

Assessing operator strategies for real-time replanning of multiple unmanned vehicles

“…Numerous previous experiments have shown the benefits of human-guided algorithms for search, such as in vehicle-routing problems [1,2,31], scheduling UxVs [5], or trade space exploration for large scale design optimization [3]. Events are often unanticipated and automated planners are typically unable to account for and respond to unforeseen problems [18,29].…”