Empirica: a virtual lab for high-throughput macro-level experiments

Almaatouq, Abdullah; Becker, Joshua; Houghton, James P.; Paton, Nicolas; Watts, Duncan J.; Whiting, Mark E.

doi:10.3758/s13428-020-01535-9

Cited by 51 publications

(26 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To achieve this, a web-based experiment platform was developed where ophthalmologists evaluated fundus retinal images under 3 different conditions—solo, ARIA answer, and ARIA explanation — described below. The platform was developed based on the Empirica framework [ 39 ]. Figure 3 displays the main screens of the web platform used in this experiment.…”

Section: Methodsmentioning

confidence: 99%

Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial

Noriega¹,

Meizner²,

Camacho³

et al. 2021

JMIR Form Res

Self Cite

View full text Add to dashboard Cite

Background The automated screening of patients at risk of developing diabetic retinopathy represents an opportunity to improve their midterm outcome and lower the public expenditure associated with direct and indirect costs of common sight-threatening complications of diabetes. Objective This study aimed to develop and evaluate the performance of an automated deep learning–based system to classify retinal fundus images as referable and nonreferable diabetic retinopathy cases, from international and Mexican patients. In particular, we aimed to evaluate the performance of the automated retina image analysis (ARIA) system under an independent scheme (ie, only ARIA screening) and 2 assistive schemes (ie, hybrid ARIA plus ophthalmologist screening), using a web-based platform for remote image analysis to determine and compare the sensibility and specificity of the 3 schemes. Methods A randomized controlled experiment was performed where 17 ophthalmologists were asked to classify a series of retinal fundus images under 3 different conditions. The conditions were to (1) screen the fundus image by themselves (solo); (2) screen the fundus image after exposure to the retina image classification of the ARIA system (ARIA answer); and (3) screen the fundus image after exposure to the classification of the ARIA system, as well as its level of confidence and an attention map highlighting the most important areas of interest in the image according to the ARIA system (ARIA explanation). The ophthalmologists’ classification in each condition and the result from the ARIA system were compared against a gold standard generated by consulting and aggregating the opinion of 3 retina specialists for each fundus image. Results The ARIA system was able to classify referable vs nonreferable cases with an area under the receiver operating characteristic curve of 98%, a sensitivity of 95.1%, and a specificity of 91.5% for international patient cases. There was an area under the receiver operating characteristic curve of 98.3%, a sensitivity of 95.2%, and a specificity of 90% for Mexican patient cases. The ARIA system performance was more successful than the average performance of the 17 ophthalmologists enrolled in the study. Additionally, the results suggest that the ARIA system can be useful as an assistive tool, as sensitivity was significantly higher in the experimental condition where ophthalmologists were exposed to the ARIA system’s answer prior to their own classification (93.3%), compared with the sensitivity of the condition where participants assessed the images independently (87.3%; P =.05). Conclusions These results demonstrate that both independent and assistive use cases of the ARIA system present, for Latin American countries such as Mexico, a substantial opportunity toward expanding the monitoring capacity for the early detection of diabetes-related blindness.

show abstract

Section: Methodsmentioning

confidence: 99%

Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial

Noriega¹,

Meizner²,

Camacho³

et al. 2021

JMIR Form Res

Self Cite

View full text Add to dashboard Cite

show abstract

“…It also allows for some of the nodes to be replaced by autonomous agents (Shirado and Christakis 2017). Similar tools such as Empirica (Almaatouq et al 2020) have been used to study collective intelligence in large networked groups. These platforms facilitate the study of several important modulators of human behavior and collective intelligence; however, they leave out real-time embodied interaction between individuals.…”

Section: Related Workmentioning

confidence: 99%

HuGoS: a virtual environment for studying collective human behavior from a swarm intelligence perspective

et al. 2021

View full text Add to dashboard Cite

Swarm intelligence studies self-organized collective behavior resulting from interactions between individuals, typically in animals and artificial agents. Some studies from cognitive science have also demonstrated self-organization mechanisms in humans, often in pairs. Further research into the topic of human swarm intelligence could provide a better understanding of new behaviors and larger human collectives. This requires studies with multiple human participants in controlled experiments in a wide variety of scenarios, where a rich scope of possible interactions can be isolated and captured. In this paper, we present HuGoS—‘Humans Go Swarming’—a multi-user virtual environment implemented using the Unity game development platform, as a comprehensive tool for experimentation in human swarm intelligence. We demonstrate the functionality of HuGoS with naïve participants in a browser-based implementation, in a coordination task involving collective decision-making, messaging and signaling, and stigmergy. By making HuGoS available as open-source software, we hope to facilitate further research in the field of human swarm intelligence.

show abstract

“…We called a network together with a specific starting position an environment. The experiment was implemented using a customized version of the Empirica framework [41] and consisted of three consecutive stages (see Figure 1a). In the first stage, participants were asked to watch the solution of a previous player.…”

Section: Taskmentioning

confidence: 99%

Hybrid social learning in human-algorithm cultural transmission

Brinkmann¹,

Gezerli²,

Kleist³

et al. 2021

Preprint

View full text Add to dashboard Cite

Humans are impressive social learners. Researchers of cultural evolution have studied the many biases that enable solutions and behaviours to spread socially from one human to the next, selecting from whom we copy and what we copy. In a digital society, algorithmic and human agents both contribute to transmission of knowledge. One hypothesis is that machines may influence the patterns of social transmission not only by providing a means for spreading human behavior but also by providing novel behaviors themselves. We propose that certain algorithms might show (either by learning or by design) different behaviors, biases and problem-solving abilities than their human counterparts. This may in turn foster better decisions in environments where diversity in problem-solving strategies is beneficial. In this study, we ask whether machines with complementary biases to humans could boost cultural evolution in a lab-based planning task, where humans show suboptimal biases. We conducted a large behavioral study and an agent-based simulation to test the performance of transmission chains with human and machine players. In half of the chains, an algorithmic bot replaced a human participant. We show that the bot boosts the performance of immediately following participants in the chain, but this gain is lost for participants further down the transmission chain. Our findings suggest that machines can potentially improve performance, but human bias can hinder machine solutions from being preserved, especially under conditions of uncertainty or high cognitive load. Our results suggest that the conditions for hybrid social learning and cultural evolution may be limited by task environment and human biases.

show abstract

Empirica: a virtual lab for high-throughput macro-level experiments

Cited by 51 publications

References 56 publications

Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial

Screening Diabetic Retinopathy Using an Automated Retinal Image Analysis System in Independent and Assistive Use Cases in Mexico: Randomized Controlled Trial

HuGoS: a virtual environment for studying collective human behavior from a swarm intelligence perspective

Hybrid social learning in human-algorithm cultural transmission

Contact Info

Product

Resources

About