The Secrets of Machine Learning: Ten Things You Wish You Had Known Earlier to Be More Effective at Data Analysis

Abstract

Despite the widespread usage of machine learning throughout organizations, there are some key principles that are commonly missed. In particular: 1) There are at least four main families for supervised learning: logical modeling methods, linear combination methods, case-based reasoning methods, and iterative summarization methods. 2) For many application domains, almost all machine learning methods perform similarly (with some caveats). Deep learning methods, which are the leading technique for computer vision problems, do not maintain an edge over other methods for most problems (and there are reasons why). 3) Neural networks are hard to train and weird stuff often happens when you try to train them. 4) If you don't use an interpretable model, you can make bad mistakes. 5) Explanations can be misleading and you can't trust them. 6) You can pretty much always find an accurate-yet-interpretable model, even for deep neural networks. 7) Special properties such as decision making or robustness must be built in, they don't happen on their own. 8) Causal inference is different than prediction (correlation is not causation). 9) There is a method to the madness of deep neural architectures, but not always. 10) It is a myth that artificial intelligence can do anything.

Into the Black Box: What Can Machine Learning Offer Environmental Health Research?

“…The widget claimed air pollution levels were safe, 15 even as people in the area were watching the ash build up on their cars. 8 According to Carlson, computer scientists are currently experimenting with ways to open up deep neural networks and other black boxes to expose their internal calculations or to produce interpretable models with comparable accuracy. Meanwhile, any model's accuracy depends, in large part, on the quantity and quality of the data to which it is exposed and differences between training data and real-world data.…”

“…8 Carlson claimed that "modifying a single pixel can completely alter an algorithm's understanding of an image" and a small decal stuck to a stop sign "can fool even a modern industrial computer vision system for self-driving vehicles." 8 Making sure that machine-learning algorithms used in environmental health have sufficient access to high-quality data is now a priority for the field. "Nothing in AI is going to work if you do not pay attention to data quality," says Woychik, adding that the NIEHS is highly focused on developing sustainable systems for generating data that can be easily shared with researchers around the world.…”

“…AI algorithms can be difficult to train, and many of them are "black boxes"-meaning their internal calculations are either proprietary information or too complex for people to understand. 8 Scientists may justifiably wonder if a black box will behave as expected when it processes real-world data or if it will pick up on confounding signals that compromise its predictions.…”

Assessment of deep neural networks for the diagnosis of benign and malignant skin neoplasms in comparison with dermatologists: A retrospective validation study

“…In addition, hospital archives have an insufficient number of images of benign disorders, which can reduce the diagnostic accuracy in the case of images from the general population with benign disorders. (4) Data leakage: This occurs when the training and test dataset are not split completely (train-test contamination) or when training uses data from the future (target leakage; e.g., cellulitis prediction based on antibiotics usage) [28]. In the case of data leakage, the model uses inessential additional information for classification [29].…”

Improving Mental Health Services: A 50-Year Journey from Randomized Experiments to Artificial Intelligence and Precision Mental Health

“…However, I would caution potential users that there are significant limitations associated with AI that are discussed later in this paper. Rudin and Carlson (2019) present a non-technical and well written review of how to utilize AI and of some of the problems that are typically encountered.…”