Gaming the System: An Agent-Based Model of Estimation Strategies and their Effects on System Performance

Abstract

Parameter estimates in large-scale complex engineered systems (LaCES) affect system evolution, yet can be difficult and expensive to test. Systems engineering uses analytical methods to reduce uncertainty, but a growing body of work from other disciplines indicates that cognitive heuristics also affect decision-making. Results from interviews with expert aerospace practitioners suggest that engineers bias estimation strategies. Practitioners reaffirmed known system features and posited that engineers may bias estimation methods as a negotiation and resource conservation strategy. Specifically, participants reported that some systems engineers “game the system” by biasing requirements to counteract subsystem estimation biases. An agent-based model (ABM) simulation which recreates these characteristics is presented. Model results suggest that system-level estimate accuracy and uncertainty depend on subsystem behavior and are not significantly affected by systems engineers' “gaming” strategy.

A Review & Framework for Modeling Complex Engineered System Development Processes

“…Researchers typically utilize qualitative methods-including interviews [106], ethnography [107], case studies [108], and grounded theory [109]-to answer questions of "what happens," "how it happens," and "why it happens" in organizations. Prominent examples with engineering organizations include assessments of the Three Mile Island [110] and Challenger [111] disasters, though studies also utilize such methods to broaden and deepen understandings of non-catastrophic activities in engineering organizations [2], [51], [74], [104], [112], [113].…”

“…5) Large-Scale Complex Engineered Systems: This practice orientation gives way to another definition seeking to bridge theory and practice. Large-scale complex engineered systems (abbreviated LSCES or LaCES) are "engineering projects with significant cost and risk, extensive design cycles, protracted operational timelines, a significant degree of complexity, and dispersed supporting organizations" [51]. The term grew out of a confluence of experts at the NSF, the National Aeronautics and Space Administration (NASA), academia, and industry to address the challenges described in Sec.…”

“…Theories have begun to form around structure [34], [99], modularity [100]- [102], quality [34], [77], innovation [103], and the limitations of systems engineering for CES [70]. A small contingent of work also studies how uncertainty affects system performance via game theory [51], [104], [105].…”

“…Systems engineering research has explored alternative system formulations such as Value-Driven Design [38], [39] while engineering design research has formulated Multidisciplinary Design Optimization [40], each of which lends itself to abstract representations of CES development. Likewise, organizational research is beginning to explore computational methods of representing organizational processes [41]- [46] as engineering follows suit [2], [47]- [51].…”

A computational simulation-based framework for estimating potential product impact during product design

“…ABM differs from more traditional differential modelling in that it does not seek for a top-down governing equation for modelling the system behaviour, rather it builds the system from the bottom-up with individual agents (Vicsek 2002). ABM has been used to model segregation (Schelling 1971), wind farm landowner relations (Syal, Ding & MacDonald 2020), social behaviour (Smaldino et al 2012), decision making (Meluso & Austin-Breneman 2018) and product adoption (Kiesling et al 2012;Rai & Robinson 2015), among many other applications (Squazzoni 2010).…”