A Machine Learning Model Helps Process Interviewer Comments in Computer-assisted Personal Interview Instruments: A Case Study

Abstract

During data collection, field interviewers often append notes or comments to a case in open text fields to request updates to case-level data. Processing these comments can improve data quality, but many are non-actionable, and processing remains a costly manual task. This article presents a case study using a novel application of machine learning tools to assist in the evaluation of these comments. Using over 5,000 comments from the Medical Expenditure Panel Survey, we built features that were fed to a machine learning model to predict a grouping category for each comment as previously assigned by data technicians to expedite processing. The model achieved high top-3 accuracy and was incorporated into a production tool for editing. A qualitative evaluation of the tool also provided encouraging results. This application of machine learning tools allowed a small but worthwhile increase in processing efficiency, while maintaining exacting standards for data quality.

"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning

“…While ad recommender systems might be easily tested online with a small fraction of users, other applications require significant simulation testing depending on safety, security, and scale issues [49,70]. Common applications of ML, such as medicine [77], customer service [19], and interview processing [6] , have their own studies. Our work expands on the literature by identifying common challenges across various applications and reporting on how MLEs handle them.…”

“…Moreover, we did not focus on the differences between practitioners' workflows based on their company sizes, educational backgrounds, or industries. While there are interview studies for specific applications of ML [6,19,77], we see further opportunities to study the effect of organizational focus and maturity on the production ML workflow. There are also questions for which interview studies are a poor fit.…”