An Overview of Systems Engineering Challenges for Designing AI-Enabled Aerospace Systems

Raz, Ali K.; Blasch, Erik; Guariniello, Cesare; Mian, Zohaib

doi:10.2514/6.2021-0564

Cited by 14 publications

(7 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The complete list of required VTAVD than forms an augmentation to the MSR checklist completion criteria, to be (subsequently) formally established as the reference Data Baseline (DBL) as an outcome of the PDR MSR, and to be subject to CM and design control, as per the extant SE FBL, ABL and PBL at each life-cycle iteration. This concept correlates with Figure 4 from Raz et al (2021), where "Data Architecture" is recognised as an addition to the Functional, Allocated and Physical architectures.…”

Section: Phase Outcomessupporting

confidence: 61%

A Conceptual Framework for the SE of AI‐Intensive Systems (SE4AI) – Considering Data Through the Life‐Cycle

Bhalla,

Cook,

Harvey

2023

INCOSE International Symp

View full text Add to dashboard Cite

This paper considers the challenge of the Systems Engineering of AI‐Intensive Systems (SE4AI) with a particular focus on their high reliance on data for model design verification, model training and model validation. The paper opens by describing the contextual background and reviewing contemporary AI/ML development approaches, identifying these to be largely software based “bottom‐up” driven by technological advancements, lacking a holistic top‐down “end‐to‐end” systems‐based perspective. Concepts and principles from systems thinking, modelling and simulation, and situation awareness are then briefly described in part two, as exaptive enabling building blocks, for integration into best‐practice evolutionary SE life‐cycle processes. A proposed tailored evolutionary SE life‐cycle process, with a focus on model design verification, model training and model validation data is then described in part three, through an example life‐cycle iteration, through each phase, in terms of the phase objective, the proposed refinement to the phase activities to cater for ML systems and the proposed phase outcomes. Related initiatives of direct relevance to this research are identified in part four, followed by a summary and conclusion.

show abstract

Section: Phase Outcomessupporting

confidence: 61%

A Conceptual Framework for the SE of AI‐Intensive Systems (SE4AI) – Considering Data Through the Life‐Cycle

Bhalla,

Cook,

Harvey

2023

INCOSE International Symp

View full text Add to dashboard Cite

show abstract

“…Typical systems engineering approaches to testing generally include developmental tests at earlier stages of a technology's development, and then operational testing as a system matures. While this approach is reasonable for deterministic systems, it is simply not going to be sustainable for systems with embedded AI, as noted by others (Raz et al, 2021 ; Wojton et al, 2021 ).…”

Section: Testing Evaluation Verification and Validation (Tevv) Of Ai ...mentioning

confidence: 94%

“…developmental tests at earlier stages of a technology's development, and then operational testing as a system matures. While this approach is reasonable for deterministic systems, it is simply not going to be sustainable for systems with embedded AI, as noted by others (Raz et al, 2021;Wojton et al, 2021). One major issue is the constant updating of software code that is a necessary byproduct of agile software development.…”

Section: Figurementioning

confidence: 96%

Revising human-systems engineering principles for embedded AI applications

Cummings

2023

Front. Neuroergonomics

View full text Add to dashboard Cite

The recent shift from predominantly hardware-based systems in complex settings to systems that heavily leverage non-deterministic artificial intelligence (AI) reasoning means that typical systems engineering processes must also adapt, especially when humans are direct or indirect users. Systems with embedded AI rely on probabilistic reasoning, which can fail in unexpected ways, and any overestimation of AI capabilities can result in systems with latent functionality gaps. This is especially true when humans oversee such systems, and such oversight has the potential to be deadly, but there is little-to-no consensus on how such system should be tested to ensure they can gracefully fail. To this end, this work outlines a roadmap for emerging research areas for complex human-centric systems with embedded AI. Fourteen new functional and tasks requirement considerations are proposed that highlight the interconnectedness between uncertainty and AI, as well as the role humans might need to play in the supervision and secure operation of such systems. In addition, 11 new and modified non-functional requirements, i.e., “ilities,” are provided and two new “ilities,” auditability and passive vulnerability, are also introduced. Ten problem areas with AI test, evaluation, verification and validation are noted, along with the need to determine reasonable risk estimates and acceptable thresholds for system performance. Lastly, multidisciplinary teams are needed for the design of effective and safe systems with embedded AI, and a new AI maintenance workforce should be developed for quality assurance of both underlying data and models.

show abstract

“…For the T&E of systems with DMAIL components, the current SE approaches, are necessary but remain insufficient. Sufficient evidence can be found in AI literature where T&E approaches such as the black box testing, parametric variations, and assumption testing are employed in AI algorithmic development, yet the T&E challenge remain in transition of AI into practical systems (Raz et al 2021). A recent paper on unsolved problems in ML safety describes four major problem areas for ML (i.e., robustness, monitoring, alignment, and external safety) which collectively point to lack of systems engineering in DMAIL (Hendrycks et al 2022).…”

Section: B Methods For Desing Test and Evaluation Of Intelligent Engi...mentioning

confidence: 99%

“…The SE T&E of data-driven DMAIL components embedded in systems is particularly challenging because they lack both well-defined requirements and narrowly defined test cases, and they also do not meet mathematically described COS criteria (Raz et al 2021). Furthermore, the evolution and behavior of these DMAIL components are not fully characterized throughout their life cycle.…”

Section: Introductionmentioning

confidence: 99%

Explainable AI and Counterfactuals for Test and Evaluation of Intelligent Engineered Systems

Raz,

Miller,

Chang

et al. 2023

INCOSE International Symp

View full text Add to dashboard Cite

Systems Engineering (SE) based test and evaluation (T&E) approaches have proven crucial for successful realization of most modern‐day complex systems and system‐of‐systems solutions. The design, test, and evaluation of engineered systems with many technologically advanced and complex components hinges on well‐structured integration and life cycle evolution process models, clearly defined requirements, along with controllability, observability, and stability (COS) of components that transcend to the system‐level. However, integration of machine learning (ML), deep learning (DL) and artificially intelligent (AI) components invalidates some SE foundations as the DL algorithms are primarily data‐driven with opaque decision‐making constructs. As a result, the current SE T&E approaches, although necessary, have become insufficient to evaluate the adoption of ML/DL/AI methods in engineered systems. This paper proposes two new approaches—explainable AI (xAI) and counterfactual testing and evaluation (cT&E)—as addition to the SE tool set for T&E of intelligent engineered systems. An example of SE T&E considerations is provided based on a conceptual aircraft control system implementation comparing as a classical controller and a DL‐based reinforcement learning controller.

show abstract

An Overview of Systems Engineering Challenges for Designing AI-Enabled Aerospace Systems

Cited by 14 publications

References 9 publications

A Conceptual Framework for the SE of AI‐Intensive Systems (SE4AI) – Considering Data Through the Life‐Cycle

A Conceptual Framework for the SE of AI‐Intensive Systems (SE4AI) – Considering Data Through the Life‐Cycle

Revising human-systems engineering principles for embedded AI applications

Explainable AI and Counterfactuals for Test and Evaluation of Intelligent Engineered Systems

Contact Info

Product

Resources

About