Complexity Should Not Be in the Eye of the Beholder: How Representative Complexity Measures Respond to the Commonly-Held Beliefs of the Literature

Abstract

In an increasingly interconnected & cyber-physical world, the ability to coherently measure and manage complexity is vital for the engineering design and systems engineering community. To this end, numerous measures have been promulgated in the literature, yet these measures differ in terms of their intellectual foundations and perspectives, with limited cross-validation among them. In this paper, we propose a framework for benchmarking the status quo of existing complexity measurement approaches in terms of their alignment with the commonly-held beliefs in the literature. We discover that the literature broadly suggests an understanding of complexity based on a system’s size, number of interconnections, and architectural structure. We adopt a design of experiments approach and synthetically create system architectures to mimic the variation across these dimensions. We then use these architectures as a shared test-bed to document the response of four complexity measures that are representative of the predominant perspectives of the literature. We do this by evaluating the change in measurement of a complexity measure as we incrementally varied the levels of one system architecture property believed to affect complexity while keeping the others constant. We find that none of the measures fully satisfy the commonly-held beliefs of the literature and provide a discussion on the underlying factors that lead to these discrepancies. We note that multiple independent discussions coexist in the literature, with little cohesion and communication across the groups, suggesting that further research is required to understand the interactions and influences among these communities. For this purpose, our rigorous, structured, and grounded in literature benchmarking approach can serve as a testbed for development and verification of future architectural assessment tools and measures.

The impact of system representation choices on architecting insights

“…Measures draw on a variety of disciplines to provide a quantifiable measurement of a system's complexity, such as but not limited to, computer science, 4,5,48,74–76 graph theory, 7,24,24,77–80 modern physics, 6,81 measurements of how a mind captures knowledge, 10,82 or developed from engineering design theory 8,83,84 . Extensive work has been undertaken to improve the validity of these measures by testing how well these measures capture some of the basic theory of the complexity phenomena, but there has been mixed success 85 . We used the Sinha structural complexity measure to measure the complexity of the system 6,35 which is the one that is most often used in recent EDSE work 53,86,87 …”

“…8,83,84 Extensive work has been undertaken to improve the validity of these measures by testing how well these measures capture some of the basic theory of the complexity phenomena, but there has been mixed success. 85 We used the Sinha structural complexity measure to measure the complexity of the system 6,35 which is the one that is most often used in recent EDSE work. 53,86,87 Complexit y Sinha =…”