Software Productivity of Field Experiments Using the Mobile Agents Open Architecture with Workflow Interoperability

Clancey, William J.; Lowry, Michael; Nado, Robert A.; Sierhuis, Maarten

doi:10.1109/smc-it.2011.10

Cited by 3 publications

(5 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experience with large-scale projects (e.g., aircraft, highly automated naval vessels, spacecraft, self-driving vehicles) and their ensuing failures indicates that evaluation of automation is crucial early in design, before construction, and not just during iterative prototyping involving empirical assessment in authentic work settings (Clancey 2020;Clancey, et al, 2011).…”

Section: Methodsmentioning

confidence: 99%

Recommendations for the Empirical Assessment of Human-AI Work Systems: A Contribution to AI Measurement Science

Hoffman¹,

Klein²,

Clancey³

2021

Preprint

View full text Add to dashboard Cite

This Report is a companion to the Report titled "Requirements for the Evaluation of Human-AI Work Systems." Whereas that Report focused on the minimum necessary empirical requirements for the assessment of AI systems, this Report provides additional recommendations and technical details to assist the developers of AI systems. Recommendations are presented covering study design, research methods, measurement, statistical analyses, and online experimentation. This guidance should be applicable to all research intended to evaluate the effectivity of AI systems.

show abstract

Section: Methodsmentioning

confidence: 99%

Recommendations for the Empirical Assessment of Human-AI Work Systems: A Contribution to AI Measurement Science

Hoffman¹,

Klein²,

Clancey³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The task is removed from its "real world" context. In the empirical assessment of HAI systems, the task presented to participants should be one that is representative of the tasks that are conducted in the work domain for which the AI has been created (see Buçinca et al 2020;Clancey 2020;Clancey et al 2011;Hernandez-Orallo 2017b). If the evaluation is sterilized, the results may be irrelevant.…”

Section: The Tasksmentioning

confidence: 99%

“…2020; Clancey 2020; Clancey et al. 2011; Hernandez‐Orallo 2017b). If the evaluation is sterilized, the results may be irrelevant.…”

Section: The Tasksmentioning

confidence: 99%

“Minimum Necessary Rigor” in empirically evaluating human–AI work systems

Klein,

Hoffman,

Clancey

et al. 2023

AI Magazine

Self Cite

View full text Add to dashboard Cite

The development of AI systems represents a significant investment of funds and time. Assessment is necessary in order to determine whether that investment has paid off. Empirical evaluation of systems in which humans and AI systems act interdependently to accomplish tasks must provide convincing empirical evidence that the work system is learnable and that the technology is usable and useful. We argue that the assessment of human–AI (HAI) systems must be effective but must also be efficient. Bench testing of a prototype of an HAI system cannot require extensive series of large‐scale experiments with complex designs. Some of the constraints that are imposed in traditional laboratory research just are not appropriate for the empirical evaluation of HAI systems. We present requirements for avoiding “unnecessary rigor.” They cover study design, research methods, statistical analyses, and online experimentation. These should be applicable to all research intended to evaluate the effectiveness of HAI systems.

show abstract

“…Point of Contact: William J. Clancey References: [6][7][8][9] Web: http://ti.arc.nasa.gov/tech/cas/work-systems-designevaluation/mobile-agents/ The time-delay of communications with Mars explorers will preclude having the relatively immediate, direct collaboration support that was provided by the CapCom communications officer during the Apollo missions. The CapCon served as the intermediary through which all voice communication was transmitted between the spacebased crew and the ground-based mission control personnel.…”

Section: Mobile Agentsmentioning

confidence: 99%

“…The agents are "mobile" because they are on movable platforms (spacesuit backpacks, all-terrain vehicles, robots). They enable workflow automation and interoperability of arbitrary hardware and software systems used during EVAs [8]. The agents facilitate collaboration among the humans, hardware, and software systems.…”

Section: Mobile Agentsmentioning

confidence: 99%

Collaborative systems for NASA science, engineering, and mission operations

Keller

Clancey

Deans

et al. 2011

2011 International Conference on Collaboration Technologies and Systems (CTS)

Self Cite

View full text Add to dashboard Cite

We describe a set of ten collaborative systems projects developed at NASA Ames Research Center over the past ten years. Our goal is to design new information technologies and collaboration tools that facilitate the process by which NASA engineers, scientists, and mission personnel collaborate in their unique work settings. We employ information management, artificial intelligence, and participatory design practices to build systems that are highly usable, augment human cognition, and support distributed NASA teams. NASA settings and applications serve as valuable testbeds for studying collaboration and developing new collaborative technologies.

show abstract

Software Productivity of Field Experiments Using the Mobile Agents Open Architecture with Workflow Interoperability

Cited by 3 publications

References 5 publications

Recommendations for the Empirical Assessment of Human-AI Work Systems: A Contribution to AI Measurement Science

Recommendations for the Empirical Assessment of Human-AI Work Systems: A Contribution to AI Measurement Science

“Minimum Necessary Rigor” in empirically evaluating human–AI work systems

Collaborative systems for NASA science, engineering, and mission operations

Contact Info

Product

Resources

About