Application usability levels: a framework for tracking project product progress

Abstract

Halford et al: Application Usability LevelsThe space physics community continues to grow and become both more interdisciplinary and more intertwined with commercial and government operations. This has created a need for a framework to easily identify what projects can be used for specific applications and how close the tool is to routine autonomous or on-demand implementation and operation. We propose the Application Usability Level (AUL) framework and publicizing AULs to help the community quantify the progress of successful applications, metrics, and validation efforts. This framework will also aid the scientific community by supplying the type of information needed to build off of previously published work and publicizing the applications and requirements needed by the user communities. In this paper, we define the AUL framework, outline the milestones required for progression to higher AULs, and provide example projects utilizing the AUL framework. This work has been completed as part of the activities of the Assessment of Understanding and Quantifying Progress working group which is part of the International Forum for Space Weather Capabilities Assessment.The existing frameworks each focus on tracking a particular type of product, and so do not fully meet the needs of the heliophysics community. Space physics products include observational data, derived indices, modeled outputs, and more. These products are often used together for different purposes. Each user will have different requirements for the application in terms of the type of product, robustness, and accuracy.The unique needs of the space weather community led to the modification of existing research-to-application communication frameworks to create the Application Usability Level (AUL) framework. Applying AULs to model and data analysis efforts can benefit space physics research. These benefits include improving access to collaborators, project transparency, and communication of project results. As the requirements and user interests for each application are unique, the AUL framework uses specifically-tuned metrics. For instance, a research user interested in upper atmospheric coupling may want to know the flux and characteristic energy of precipitating electrons. Similarly, a satellite industry partner may want to predict satellite drag during a geomagnetic storm. A single research project may be able to provide both users with the products they need. However, the different outputs will require different metrics, implementation strategies, and time frames for implementation. Since the AUL framework is highly adaptable, it can help a single research project meet and track both of these user needs.The AUL framework can bolster communication between researchers, users, funding bodies, and stakeholders. Using a standard framework provides a clear path for users and researchers to follow. This improves efficiency assuring that all components from the researchers' project to the user needs are considered. It enables communication about a proposal's dev...

Developing the LDi and LCi Geomagnetic Indices, an Example of Application of the AULs Framework

“…Direct communication with the researchers is encouraged for any additional information about the project or the application. Halford et al (2019) introduced the AULs framework as a structured approach for tracking progress of a project toward an application. Figure 1 (from ; Halford et al, 2019) summarizes the AUL levels and phases, providing a diagram with the progress of a project toward an application moving from AUL 1 to AUL 9 passing through three main phases: discovery, development, and implementation.…”

“…Since ESA is one of the main users, we have checked that AULs framework terminology fits with the ECSS Glossary. Halford et al (2019). The progress of a project toward an application moves from AUL 1 to AUL 9, passing through three main phases: discovery, development, and implementation.…”

“…Nowadays both indices are approved for autonomous use at the Spanish Space Weather Service (http://www.senmes.es/) and at the ESA Space Weather Portal (http://swe.ssa.esa.int/). The development of these indices can be used as an example of the Application Usability Level (AUL) framework introduced by Halford et al (2019) to easily identify which projects can be used for specific applications and how suitable the tool is for autonomous routine or on-demand implementation. Although developed before the AUL framework, this project and the associated interaction between the University of Alcala and REE and ESA may be used as an example of how one would apply the AUL framework for a specific application.…”

The STONE Curve: A ROC‐Derived Model Performance Assessment Tool

“…A different threshold settings might be most favorable if other considerations outweigh discrimination, such as minimizing false alarms or maximizing a particular skill score. Focusing on user needs during the validation of a model is a foundational element of Application Usability Levels (Halford et al, 2019) and should always be considered when assessing a model's performance. If the user is most concerned about optimizing one of these other features of the data‐model comparison for their particular decision‐making needs, then a model event identification threshold should be chosen that best addresses that need.…”

The soft X-ray Neupert effect as a proxy for solar energetic particle injection

“…Using the definition of Halford et al (2019), the approach described here is currently in Phase 1 of application usability. The choice of interplanetary transport parameters also remains uncertain and a thorough parameter study is yet to be performed.…”