Introduction Pharmacovigilance (PV) detects, assesses, and prevents adverse events (AEs) and other drug-related problems by collecting, evaluating, and acting upon AEs. The volume of individual case safety reports (ICSRs) increases yearly, but it is estimated that more than 90% of AEs go unreported. In this landscape, embracing assistive technologies at scale becomes necessary to obtain a higher yield of AEs, to maintain compliance, and transform the PV professional work life. Aim The aim of this study was to identify areas across the PV value chain that can be augmented by cognitive service solutions using the methodologies of contextual analysis and cognitive load theory. It will also provide a framework of how to validate these PV cognitive services leveraging the acceptable quality limit approach. Methods The data used to train the cognitive service were an annotated corpus consisting of 20,000 ICSRS from which we developed a framework to identify and validate 40 cognitive services ranging from information extraction to complex decision making. This framework addresses the following shortcomings: (1) needing subject-matter expertise (SME) to match the artificial intelligence (AI) model predictions to the gold standard, commonly referred to as 'ground truth' in the AI space, (2) ground truth inconsistencies, (3) automated validation of prediction missing context, and (4) auto-labeling causing inaccurate test accuracy. The method consists of (1) conducting contextual analysis, (2) assessing human cognitive workload, (3) determining decision points for applying artificial intelligence (AI), (4) defining the scope of the data, or annotated corpus required for training and validation of the cognitive services, (5) identifying and standardizing PV knowledge elements, (6) developing cognitive services, and (7) reviewing and validating cognitive services. Results By applying the framework, we (1) identified 51 decision points as candidates for AI use, (2) standardized the process to make PV knowledge explicit, (3) embedded SMEs in the process to preserve PV knowledge and context, (4) standardized acceptability by using established quality inspection principles, and (5) validated a total of 126 cognitive services. Conclusion The value of using AI methodologies in PV is compelling; however, as PV is highly regulated, acceptability will require assurances of quality, consistency, and standardization. We are proposing a foundational framework that the industry can use to identify and validate services to better support the gathering of quality data and to better serve the PV professional.

The healthcare industry, and specifically the pharmacovigilance industry, recognizes the need to support the increasing amount of data received from individual case safety reports (ICSRs). To cope with this increase, more healthcare and qualified professionals are required to capture and evaluate the data. To address the evolving landscape, it will be necessary to embrace assistive technologies such as artificial intelligence (AI) at scale. AI in the field of pharmacovigilance will possibly result in the transformation of the drug safety (DS) professional's daily work life and their career development. Celgene's Global Drug Safety and Risk Management (GDSRM) function has established a series of work activities to drive innovation across the pharmacovigilance value chain (Celgene Chrysalis Fact Sheet. https://www.celgene.com/newsroom/media-library/ chrysalis-fact-sheet/, 2018). The development of AI in pharmacovigilance raises questions about the possible changes in DS professionals' lives, who may find themselves curious about their future roles in a workplace assisted by AI. We discuss the current state of pharmacovigilance and the DS professional, AI in pharmacovigilance and the potential skillsets a DS professional may require when working with AI. We also describe the results of research conducted at Celgene GDSRM. The objective of the research was to understand the thoughts of pharmacovigilance professionals about their jobs. These results are provided in the form of aggregated responses to interview questions based on a 12-part questionnaire [see the Electronic Supplementary Material (ESM)]. A sample of six DS professionals representing various areas of pharmacovigilance operations were asked a range of questions about their backgrounds, current roles and future expectations. The DS professionals interviewed were, overall, enthusiastic about their job roles potentially changing with AI enhancements. Interviewees suggested that AI would allow for pharmacovigilance resources, time, and skills to shift the work from a volume-based to a value-based focus. The results suggest that pharmacovigilance professionals wish to use their qualifications, skillsets and experience in work that provides more value for their efforts. Machine learning algorithms have the potential to enhance DS professionals' decision-making processes and support more efficient and accurate case processing. Key Points Increases in the number of individual case safety reports require assistive technologies such as artificial intelligence (AI) to support the drug safety (DS) professional with the increasing volume and complexity of work. Using AI, the DS professional's work life may potentially change as their decision making is augmented and efficiency enhanced. DS professionals may need to learn new skills and competencies to understand and work with AI.

Introduction Identification of adverse events and determination of their seriousness ensures timely detection of potential patient safety concerns. Adverse event seriousness is a key factor in defining reporting timelines and is often performed manually by pharmacovigilance experts. The dramatic increase in the volume of safety reports necessitates exploration of scalable solutions that also meet reporting timeline requirements. Objective The aim of this study was to develop an augmented intelligence methodology for automatically identifying adverse event seriousness in spontaneous, solicited, and medical literature safety reports. Deep learning models were evaluated for accuracy and/or the F1 score against a ground truth labeled by pharmacovigilance experts. Methods Using a stratified random sample of safety reports received by Celgene, we developed three neural networks for addressing identification of adverse event seriousness: (1) a binary adverse-event level seriousness classifier; (2) a classifier for determining seriousness categorization at the adverse-event level; and (3) an annotator for identifying seriousness criteria terms to provide supporting evidence at the document level. Results The seriousness classifier achieved an accuracy of 83.0% in post-marketing reports, 92.9% in solicited reports, and 86.3% in medical literature reports. F1 scores for seriousness categorization were 77.7 for death, 78.9 for hospitalization, and 75.5 for important medical events. The seriousness annotator achieved an F1 score of 89.9 in solicited reports, and 75.2 in medical literature reports. Conclusions The results of this study indicate that a neural network approach can provide an accurate and scalable solution for potentially augmenting pharmacovigilance practitioner determination of adverse event seriousness in spontaneous, solicited, and medical literature reports. Ramani Routray and Niki Tetarenko contributed equally to this work.

IntroductionRegulations are increasing the scope of activities that fall under the remit of drug safety. Currently, individual case safety report (ICSR) collection and collation is done manually, requiring pharmacovigilance professionals to perform many transactional activities before data are available for assessment and aggregated analyses. For a biopharmaceutical company to meet its responsibilities to patients and regulatory bodies regarding the safe use and distribution of its products, improved business processes must be implemented to drive the industry forward in the best interest of patients globally. Augmented intelligent capabilities have already demonstrated success in capturing adverse events from diverse data sources. It has potential to provide a scalable solution for handling the ever-increasing ICSR volumes experienced within the industry by supporting pharmacovigilance professionals’ decision-making.ObjectiveThe aim of this study was to train and evaluate a consortium of cognitive services to identify key characteristics of spontaneous ICSRs satisfying an acceptable level of accuracy determined by considering business requirements and effective use in a real-world setting. The results of this study will serve as supporting evidence for or against implementing augmented intelligence in case processing to increase operational efficiency and data quality consistency.MethodsA consortium of ten cognitive services to augment aspects of ICSR processing were identified and trained through deep-learning approaches. The input data for model training were 20,000 ICSRs received by Celgene drug safety over a 2-year period. The data were manually made machine-readable through the process of transcription, which converts images into text. The machine-readable documents were manually annotated for pharmacovigilance data elements to facilitate the training and testing of the cognitive services. Once trained by cognitive developers, the cognitive services’ output was reviewed by pharmacovigilance subject-matter experts against the accepted ground-truth for correctness and completeness. To be considered adequately trained and functional, each cognitive service was required to reach a threshold of F1 or accuracy score ≥ 75%.ResultsAll ten cognitive services under development have reached an evaluative score ≥ 75% for spontaneous ICSRs.ConclusionAll cognitive services under development have achieved the minimum evaluative threshold to be considered adequately trained, demonstrating how machine-learning and natural language processing techniques together provide accurate outputs that may augment pharmacovigilance professionals’ processing of spontaneous ICSRs quickly and accurately. The intention of augmented intelligence is not to replace the pharmacovigilance professional, but rather support them in their consistent decision-making so that they may better handle the overwhelming amount of data otherwise manually curated and monitored for ongoing drug surveillance requirements. Through this supported decision-making...

No abstract