Open-ended questions in surveys are often manually coded into one of several classes (or categories). When the data are too large to manually code all texts, a statistical (or machine) learning model must be trained on a manually coded subset of texts. Uncoded texts are then coded automatically using the trained model. The quality of automatic coding depends on the trained statistical model, and the model relies on manually coded data on which it is trained. While survey scientists are acutely aware that the manual coding is not always accurate, it is not clear how double coding affects the classification errors of the statistical learning model. We investigate several budget allocation strategies when there is a limited budget for manual classification: single coding versus various options for double coding where the number of training texts is reduced to maintain the fixed budget. Under fixed budget, double coding improved prediction of the learning algorithm when the coding error is greater than about 20–35%, depending on the data. Among double-coding strategies, paying for an expert to resolve differences performed best. When no expert is available, removing differences from the training data outperformed other double-coding strategies. When there is no budget constraint and the texts have already been double coded, all double-coding strategies generally outperformed single coding. As under fixed budget, having an expert to solve disagreement in training texts improves accuracy most, followed by removing differences.
Text answers to open-ended questions are typically manually coded into one of several codes. Usually, a random subset of text answers is double-coded to assess intercoder reliability, but most of the data remain single-coded. Any disagreement between the two coders points to an error by one of the coders. When the budget allows double coding additional text answers, we propose employing statistical learning models to predict which single-coded answers have a high risk of a coding error. Specifically, we train a model on the double-coded random subset and predict the probability that the single-coded codes are correct. Then, text answers with the highest risk are double-coded to verify. In experiments with three data sets, we found that this method identifies two to three times as many coding errors in the additional text answers as compared to random guessing, on average. We conclude that this method is preferred if the budget permits additional double-coding. When there are a lot of intercoder disagreements, the benefit can be substantial.
Objectives
Pharmacy professionals are required to take all necessary steps to protect commonly misused drugs such as opioids at their pharmacies to minimize the risk of diversion. The aim of this study is to assess Canadian pharmacy professionals’ knowledge and compliance with federal and provincial regulations using the computer-based educational platform Pharmacy5in5.
Methods
A Narcotic Inventory module was created and reviewed by experts representing provincial and federal regulators. Descriptive statistics were used to analyze users’ performance in quizzes. Binomial regression and logistic regression models were used to investigate the effect of demographic factors on users’ performance. P-values less than 0.05 were considered statistically significant.
Key findings
The analysis included data collected over a period of three months. A total of 792 users accessed the Narcotic Inventory module on the Pharmacy5in5 website between July 2019 and November 2019. Most of the users were licenced pharmacists (64%), female (72%), received their training in Canada (68%), and were practising in Ontario (80%). Users performed best on the quiz addressing the steps for reconciliation of inventory (93%), and worst on the quiz reviewing how to prepare for a Health Canada visit (66%).
Conclusions
Overall, pharmacy professionals showed adequate knowledge of the CDSA and provincial/territorial regulations regarding opioids inventory management. Conversely, the study highlighted poor compliance with the reporting of losses and theft of controlled substances by pharmacy professionals. Innovative approaches are needed to influence pharmacy professionals’ behaviours to improve their compliance with best practices concerning inventory management to reduce drug diversion.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.