Using System Entity Structures to Model the Elements of a Scenario in a Research Flight Simulator

Durak, Umut; Pruter, Insa; Gerlach, Torsten; Jafer, Shafagh; Pawletta, Thorsten; Hartmann, Sven

doi:10.2514/6.2017-1076

Cited by 12 publications

(5 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the work, the authors aim to answer the question of what needs to be simulated for synthetic data and scenario generation in the simulation engineering process of an AI-based system. The used methods are a simulation-based data generation process adapted from EASA's first usable guidance for Level 1 machine learning applications and the scenario-based approach using SES, which is explained more thoroughly in the publications of Durak [11], [12] as well as Karmokar [13]. The work in [10] is succeeded with [14], which discusses behavioral modeling for scenario-based testing in aviation and introduces an enhanced approach for scenario-based testing called Operational Domain Driven Testing.…”

Section: Related Workmentioning

confidence: 99%

Iterative Scenario-Based Testing in an Operational Design Domain for Artiﬁcial Intelligence Based Systems in Aviation

Lukic¹,

Sprockhoff²,

Ahlbrecht³

et al. 2023

ASIM WS 2023 Proceedings Langbeiträge

View full text Add to dashboard Cite

The use and development of Artificial Intelligence (AI) based systems is becoming increasingly prominent in different industries. The aviation industry is also gradually adopting AI-based systems, for instance, with Machine Learning algorithms for flight assistance. There are several reasons why adopting these technologies poses additional obstacles in aviation compared to other industries. One reason are the strong safety requirements which lead to obligatory and thorough assurance activities such as testing to obtain certification. Therefore, a systematic approach is needed for developing, deploying, and assessing test cases for AI-based systems in aviation. This paper proposes a method for iterative scenario-based testing for AI-based systems. The method contains three major parts: First, a high-level description of test scenarios; second, the generation and execution of these scenarios; and last, monitoring of parameters during scenario execution. Parameters are refined, and the steps are repeated iteratively. The method forms a basis for developing iterative scenariobased testing solutions. As a domain-specific example, a practical implementation of this method is illustrated. For an object detection application used on an airplane, flight scenarios, including multiple airplanes are generated from a descriptive scenario model and executed in a simulation environment. The parameters are monitored using a custom Operational Design Domain monitoring tool and refined in the process of iterative scenario generation and execution. The proposed iterative scenariobased testing method helps in generating precise test cases for AI-based systems while having a high potential for automation.

show abstract

Section: Related Workmentioning

confidence: 99%

Iterative Scenario-Based Testing in an Operational Design Domain for Artiﬁcial Intelligence Based Systems in Aviation

Lukic¹,

Sprockhoff²,

Ahlbrecht³

et al. 2023

ASIM WS 2023 Proceedings Langbeiträge

View full text Add to dashboard Cite

show abstract

“…To address these issues, the American Institute of Aeronautics and Astronautics (AIAA) Modeling and Simulation Technical Committee (MSTC) launched a working group towards the development of a standard simulation scenario definition language for the aviation domain [16]. Within the scope of this effort, Durak et al proposed using System Entity Structure for specifiying a domain specific languge for simulation scenario definition [17]. SES is a high-level ontology which was introduced for knowledge representation of decomposition, taxonomy and coupling of systems [18].…”

Section: Formalizing Concept Of Operationmentioning

confidence: 99%

“…Attributes are then used to define the scenario parameters. The details of the language are presented in recent papers [24,25] of Durak and colleagues. To improve the understanding of a formalized ConOps, a representative excerpt of an OSDL is given in Fig.8.…”

Section: Fig 7 Transformation From Formalized Conops Description (Opmentioning

confidence: 99%

Formally Bounding UAS Behavior to Concept of Operation with Operation-Specific Scenario Description Language

Torens

Durak

Nikodem

et al. 2019

AIAA Scitech 2019 Forum

Self Cite

View full text Add to dashboard Cite

Previous work introduced an approach for formally describing the concept of operations for unmanned aircraft. For this purpose, an existing language for simulation scenario description was adapted. In the context of the specific operation category, an upcoming European regulation for the operation of unmanned aircraft, the description and acceptance of the concept of operations plays a major role for flight approval on a per mission basis. This paper extends the previous approach further with combining the formalized description of the concept of operations with our existing approach for runtime monitoring. Monitoring the behavior at runtime can be used to enforce certain limits on the behavior. Therefore, the concept of operations is an ideal input for the monitoring approach. As a basis for the information relevant for the concept of operations the official annex to the guidelines document for the specific operation risk assessment is used, as well as an internal concept of operations document for a DLR research unmanned aircraft system.

show abstract

“…Addressing this problem the American Institute of Aeronautics and Astronautics (AIAA) Modeling and Simulation Technical Committee (MSTC) recently launched a working group towards the development of a standard simulation scenario definition language for the aviation domain, namely AIAA Simulation Scenario Development Working Group (AIAA SSDWG). Within this context, the current practices, problems and opportunities in scenario development [10][11][12][13], the requirements of a standard simulation scenario definition language [14][15][16] and possible technical solutions [17][18][19][20] are currently being discussed.…”

Section: A Competency-based Training Scenario Design Toolmentioning

confidence: 99%

“…Scenario development is a data engineering activity in which the data elements of a scenario are modeled. Following the simulation data modeling approach proposed by Zeigler and Hammonds [21], Durak et al [19] proposed a scenario definition language based on System Entity Structure (SES). SES can be introduced as an high-level ontology for knowledge representation of decomposition, taxonomy and coupling of systems [22].…”

Section: A Competency-based Training Scenario Design Toolmentioning

confidence: 99%

Simulating the Unexpected: Challenge-Centric Simulator Scenario Design for Advanced Flight Crew Training

Niedermeier

Buch

Mohrmann

et al. 2018

2018 AIAA Modeling and Simulation Technologies Conference

Self Cite

View full text Add to dashboard Cite

In 2013 the International Civil Aviation Organization (ICAO) published a report on the implementation of Evidence-Based Training (EBT). EBT aims at improving the overall capabilities of a pilot trainee in certain competencies such as decision-making, communication, situational awareness and others. This results in a shift in training focus from maneuvers to the training of competencies, which in turn sets different requirements for the desired simulator training scenarios. This paper presents a training scenario design approach being driven by the expected impact on the crew rather than focusing on specific maneuvers or system failures. The so-called "Challenge-Centric" design approach was applied in the EU project Man4Gen. It forms the basis for the development of a scenario design kit that shall facilitate the design of training scenarios for Evidence-based Training. A. The Need for Competency-Based TrainingFor many decades full flight simulators have proven to be a powerful tool for training flight crews in airline operations. This is true for the training of specific basic flying skills, the handling of particular system failures as well as the training of decision-making and problem solving processes in a complex operationally relevant scenario. In the early days of simulator training, each scenario was intended to train certain anticipated events and failures, often referred to as Maneuver-Based Training (MBT). This training is still necessary to properly prepare flight crews for critical events that must be recognized rapidly. That is why it was set as regulatory standard. However, as aircraft and operations become increasingly complex, recent accidents and incidents are showing that crews are having difficulty with a different type of challenge in the cockpit: managing unexpected, complex and ambiguous situations. Examples of such cases are Qantas 32 [1] and Air Asia 8501 [2].Maneuver-based Training does not necessarily prepare flight crews well for these new challenges. In addition, the relevancy of certain scenario elements required by the regulators has changed due to the evolution of modern aircraft and their component reliability. This led to the development of Evidence-based Training by the International Air Transport Association (IATA) Training and Qualification Initiative as defined in the ICAO Document 9995 "Manual of Evidence-based Training" [3]. EBT aims at improving the overall capabilities of a pilot trainee in certain competencies such as decision-making, communication, situational awareness and others. This results in a shift in training focus from maneuvers, to the training of competencies, which in turn sets different requirements for the desired simulator training scenarios. Inherent to scenarios which trigger specific competencies is that they are unfamiliar and pose the crew with a novel situation in which they can practice (or demonstrate) their competencies without being able to have memorized the exact solution for that scenario. The major challenge lies in making several scenarios ...

show abstract

Using System Entity Structures to Model the Elements of a Scenario in a Research Flight Simulator

Cited by 12 publications

References 9 publications

Iterative Scenario-Based Testing in an Operational Design Domain for Artiﬁcial Intelligence Based Systems in Aviation

Iterative Scenario-Based Testing in an Operational Design Domain for Artiﬁcial Intelligence Based Systems in Aviation

Formally Bounding UAS Behavior to Concept of Operation with Operation-Specific Scenario Description Language

Simulating the Unexpected: Challenge-Centric Simulator Scenario Design for Advanced Flight Crew Training

Contact Info

Product

Resources

About