Easy and Efficient Hyperparameter Optimization to Address Some Artificial Intelligence “ilities”

Abstract

Artificial Intelligence (AI), has many benefits, including the ability to find complex patterns, automation, and meaning making. Through these benefits, AI has revolutionized image processing among numerous other disciplines. AI further has the potential to revolutionize other domains; however, this will not happen until we can address the "ilities": repeatability, explain-ability, reliability, use-ability, trust-ability, etc. Notably, many problems with the "ilities" are due to the artistic nature of AI algorithm development, especially hyperparameter determination. AI algorithms are often crafted products with the hyperparameters learned experientially. As such, when applying the same algorithm to new problems, the algorithm may not perform due to inappropriate settings. This research aims to provide a straightforward and reliable approach to automatically determining suitable hyperparameter settings when given an AI algorithm. Results, show reasonable performance is possible and end-to-end examples are given for three deep learning algorithms and three different data problems.

Tuning Hyperparameters for DNA-based Discrimination of Wireless Devices

“…Hyperparameter determination is an emerging discipline in AI and includes a multitude of methods. A general taxonomy of these approaches is presented in [11]. These can largely be separated into model-free and model-based approaches [29].…”

“…Model-based approaches employ a wrapper on an outer loop around the algorithm of interest and determine settings to explore in a concerted search strategy. From the families of model-based approaches listed in [11], we consider:…”

“…BO similarly considers ℎ , and ; but additionally, ℎ , is in the bounded set ℋ , which can be continuous or integer valued. BO notes that ( ) is stochastic in nature due to the randomness in the results as a function of the random selection of training data [11]. For BO, let = ( ) and let { , ℎ } =1 be a sequence of and ℎ pairs.…”

“…This work compares four hyperparameter optimization methods for WPAN security using the GRLVQI algorithm as the representative classifier. The four methods are BO, leveraging the process of [11], Stochastic Approximation (SA) [6], and DoE [5]. An extension of CRISP-DM is used to create a repeatable process for this purpose using experimentally collected Z-Wave RF Fingerprints.…”

A Preliminary Look at Heuristic Analysis for Assessing Artificial Intelligence Explainability

“…This concern is due to the complexity in AI decision-making processes which are often un-interpretable, unexplainable, or un-alignable from a human user perspective [1] [2]. In AI and automation, a user typically asks three questions when encountering unexpected or unexplainable results [ Collectively, these questions get at a larger problem which precludes the wider adoption of AI algorithms, the failures of AI to address the "ilities" [4] [5] [6]: reliability, repeatability [7] [1], replicability [1], trust-ability [8], functionality [9], portability [9], usability [9], maintainability [9], and explainability [2]. Of particular interest in this paper, and to programs such as DARPA's eXplainable AI (XAI) project [2], is addressing the explainability of AI to reduce the opacity of algorithms and decrease the amount of unexpected or unexplained results [2].…”