Searching for explanations: testing social scientific methods in synthetic ground-truthed worlds

Schmidt, Aurora; Cameron, Christopher J.; Corey, Lowman,; Brulé, Joshua; Deshpande, Amruta J.; Fatemi, Seyyed A.; Barash, Vladimir; Greenberg, Ariel M.; Costello, Cash; Sherman, Eli; Bhattacharya, Rohit; McQuillan, Liz; Perrone, Alexander; Kouskoulas, Yanni; Fink, Clay; Zhang, Mengjie; Shpitser, Ilya; Macy, Michael W.

doi:10.1007/s10588-021-09353-w

Cited by 2 publications

(1 citation statement)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At the end of each test, the research teams submitted results to the test and evaluation team. More information on the methods used to develop these submissions can be found in Graziul et al (2022), Schmidt et al (2022), andVolkova et al (2022). The test and evaluation team collected any necessary data (for example, historical projects from the simulations) and evaluated the research teams' performance.…”

Section: Program Structurementioning

confidence: 99%

The Ground Truth program: simulations as test beds for social science research methods

Naugle

Russell

Lakkaraju

et al. 2022

Comput Math Organ Theory

View full text Add to dashboard Cite

Social systems are uniquely complex and difficult to study, but understanding them is vital to solving the world’s problems. The Ground Truth program developed a new way of testing the research methods that attempt to understand and leverage the Human Domain and its associated complexities. The program developed simulations of social systems as virtual world test beds. Not only were these simulations able to produce data on future states of the system under various circumstances and scenarios, but their causal ground truth was also explicitly known. Research teams studied these virtual worlds, facilitating deep validation of causal inference, prediction, and prescription methods. The Ground Truth program model provides a way to test and validate research methods to an extent previously impossible, and to study the intricacies and interactions of different components of research.

show abstract

Section: Program Structurementioning

confidence: 99%

The Ground Truth program: simulations as test beds for social science research methods

Naugle

Russell

Lakkaraju

et al. 2022

Comput Math Organ Theory

View full text Add to dashboard Cite

show abstract

What can simulation test beds teach us about social science? Results of the ground truth program

Naugle

Krofcheck

Warrender

et al. 2022

Comput Math Organ Theory

View full text Add to dashboard Cite

The ground truth program used simulations as test beds for social science research methods. The simulations had known ground truth and were capable of producing large amounts of data. This allowed research teams to run experiments and ask questions of these simulations similar to social scientists studying real-world systems, and enabled robust evaluation of their causal inference, prediction, and prescription capabilities. We tested three hypotheses about research effectiveness using data from the ground truth program, specifically looking at the influence of complexity, causal understanding, and data collection on performance. We found some evidence that system complexity and causal understanding influenced research performance, but no evidence that data availability contributed. The ground truth program may be the first robust coupling of simulation test beds with an experimental framework capable of teasing out factors that determine the success of social science research.

show abstract

Searching for explanations: testing social scientific methods in synthetic ground-truthed worlds

Cited by 2 publications

References 12 publications

The Ground Truth program: simulations as test beds for social science research methods

The Ground Truth program: simulations as test beds for social science research methods

What can simulation test beds teach us about social science? Results of the ground truth program

Contact Info

Product

Resources

About