User-centric image segmentation using an interactive parameter adaptation tool

Pauplin, Olivier; Caleb-Solly, Praminda; Smith, James E.

doi:10.1016/j.patcog.2009.03.007

Cited by 10 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Within the fields of design and optimisation via techniques such as Interactive Evolutionary Algorithms [28], the importance of interfaces that support people to take a more active role in guiding search, and the effects on facilitating and prolonging engagement has been extensively studied e.g. [29][30][31]. Amershi et al survey the role of humans in interactive machine learning [32], looking at different forms of user interaction, and also considering how people can provide more than just data labels.…”

Section: Related Workmentioning

confidence: 99%

Visual analytics for collaborative human-machine confidence in human-centric active learning tasks

Legg

Smith

Downing

2019

Hum. Cent. Comput. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

Active machine learning is a human-centric paradigm that leverages a small labelled dataset to build an initial weak classifier, that can then be improved over time through human-machine collaboration. As new unlabelled samples are observed, the machine can either provide a prediction, or query a human 'oracle' when the machine is not confident in its prediction. Of course, just as the machine may lack confidence, the same can also be true of a human 'oracle': humans are not all-knowing, untiring oracles. A human's ability to provide an accurate and confident response will often vary between queries, according to the duration of the current interaction, their level of engagement with the system, and the difficulty of the labelling task. This poses an important question of how uncertainty can be expressed and accounted for in a human-machine collaboration. In short, how can we facilitate a mutually-transparent collaboration between two uncertain actors-a person and a machine-that leads to an improved outcome? In this work, we demonstrate the benefit of human-machine collaboration within the process of active learning, where limited data samples are available or where labelling costs are high. To achieve this, we developed a visual analytics tool for active learning that promotes transparency, inspection, understanding and trust, of the learning process through human-machine collaboration. Fundamental to the notion of confidence, both parties can report their level of confidence during active learning tasks using the tool, such that this can be used to inform learning. Human confidence of labels can be accounted for by the machine, the machine can query for samples based on confidence measures, and the machine can report confidence of current predictions to the human, to further the trust and transparency between the collaborative parties. In particular, we find that this can improve the robustness of the classifier when incorrect sample labels are provided, due to unconfidence or fatigue. Reported confidences can also better inform human-machine sample selection in collaborative sampling. Our experimentation compares the impact of different selection strategies for acquiring samples: machine-driven, human-driven, and collaborative selection. We demonstrate how a collaborative approach can improve trust in the model robustness, achieving high accuracy and low user correction, with only limited data sample selections.

show abstract

Section: Related Workmentioning

confidence: 99%

Visual analytics for collaborative human-machine confidence in human-centric active learning tasks

Legg

Smith

Downing

2019

Hum. Cent. Comput. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, in practice a nice synergy has been observed -in fact the ability to guide search actually increases user engagement, and so delays the onset of fatigue [335]. Methods for directly affecting variation are usually designed with the goal of increasing the rate of adaptation towards good solutions by removing some of the black-box elements of search.…”

Section: Interaction In the Variation Processmentioning

confidence: 99%

“…Pauplin et al [335] described the development of an interactive tool used by quality control engineers. The task is to create customised software that automatically detects and highlights defects in images of processed items.…”

Section: Example Application: Automatic Elicitation Of User Preferencesmentioning

confidence: 99%

Introduction to Evolutionary Computing

Eiben

Smith

2015

Natural Computing Series

698

230

View full text Add to dashboard Cite

“…However, this could well be improved further by also factoring in the level of confidence that the user currently possesses when interacting with the system. It has also been shown for a variety of different applications that by incorporating user engagement via multi-modal interactions, a user can effectively provide "hints" that implicitly control the search process [24,18,22,15,14].…”

Section: Introductionmentioning

confidence: 99%

Predicting User Confidence During Visual Decision Making

Smith

Legg

Matovic

et al. 2018

ACM Trans. Interact. Intell. Syst.

Self Cite

View full text Add to dashboard Cite

People are not infallible consistent "oracles": their confidence in decisionmaking may vary significantly between tasks and over time. We have previously reported the benefits of using an interface and algorithms that explicitly captured and exploited users' confidence: error rates were reduced by up to 50% for an industrial multi-class learning problem; and the number of interactions required in a design optimisation context was reduced by 33%. Having access to users' confidence judgements could significantly benefit intelligent interactive systems in industry, in areas such as Intelligent Tutoring systems, and in healthcare. There are many reasons for wanting to capture information about confidence implicitly. Some are ergonomic, but others are more 'social'-such as wishing to understand (and possibly take account of) users' cognitive state without interrupting them. We investigate the hypothesis that users' confidence can be accurately predicted from measurements of their behaviour. Eye-tracking systems were used to capture users' gaze patterns as they undertook a series of visual decision tasks, after each of which they reported their confidence on a 5-point Likert scale. Subsequently, predictive models were built using "conventional" Machine Learning approaches for numerical summary features derived from users' behaviour. We also investigate the extent to which the deep learning paradigm can reduce the need to design features specific to each application, by creating "gazemaps"-visual representations of the trajectories and durations of users' gaze fixations-and then training deep convolutional networks on these images. Treating the prediction of user confidence as a two-class problem (confident/not confident), we attained classification accuracy of 88% for the scenario of new users on known tasks, and 87% for known users on new tasks. Considering the confidence as an ordinal variable, we produced regression models with a mean absolute error of ≈0.7 in both cases. Capturing just a simple subset of non-task-specific numerical features gave slightly worse, but still quite high accuracy (eg. MAE ≈1.0). Results obtained with gazemaps and convolutional networks are competitive, despite not having access to longer-term information about users and tasks, which was vital for the 'summary' feature sets. This suggests that the gazemapbased approach forms a viable, transferable, alternative to hand-crafting features for each different application. These results provide significant evidence to confirm our hypothesis, and offer a way of substantially improving many interactive artificial intelligence applications via the addition of cheap non-intrusive hardware and computationally cheap prediction algorithms.

show abstract

User-centric image segmentation using an interactive parameter adaptation tool

Cited by 10 publications

References 12 publications

Visual analytics for collaborative human-machine confidence in human-centric active learning tasks

Visual analytics for collaborative human-machine confidence in human-centric active learning tasks

Introduction to Evolutionary Computing

Predicting User Confidence During Visual Decision Making

Contact Info

Product

Resources

About