CLIMAT: Clinically-Inspired Multi-Agent Transformers for Knee Osteoarthritis Trajectory Forecasting

Abstract

In medical applications, deep learning methods are built to automate diagnostic tasks. However, a clinically relevant question that practitioners usually face, is how to predict the future trajectory of a disease (prognosis). Current methods for such a problem often require domain knowledge, and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many forecasting problem from multimodal data. Inspired by a clinical decisionmaking process with two agents -a radiologist and a general practitioner, we model a prognosis prediction problem with two transformer-based components that share information between each other. The first block in this model aims to analyze the imaging data, and the second block leverages the internal representations of the first one as inputs, also fusing them with auxiliary patient data. We show the effectiveness of our method in predicting the development of structural knee osteoarthritis changes over time. Our results show that the proposed method outperforms the state-of-theart baselines in terms of various performance metrics. In addition, we empirically show that the existence of the multi-agent transformers with depths of 2 is sufficient to achieve good performances. Our code is publicly available at https: //github.com/MIPT-Oulu/CLIMAT.

The KNee OsteoArthritis Prediction (KNOAP2020) challenge: An image analysis challenge to predict incident symptomatic radiographic knee osteoarthritis from MRI and X-ray images

“…However, the MRI pipeline outperformed the X-ray pipeline for subjects without OA and with severe OA. Nguyen et al (2021) predicted OA structural prognosis assessed by KL grade from X-ray and clinical variables and reported BACCs from 0.27 to 0.55 37 . In general, the performance of the models was lower in this study than in previous studies.…”

Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting From Multimodal Data

“…Recent work has shown that multiple transformers are needed to for such multiple modalities [32]. Therefore, similar to our previous version [22], this study adapts the idea of using multiple transformers in our framework to perform DTF from multiple modalities.…”

“…In this paper, we present an extended version of our earlier work on automatic DTF [22], where we proposed a Clinically-Inspired Multi-Agent Transformers (CLIMAT) framework, aiming to mimic the interaction process between a general practitioner / treating physician 1 and a radiologist. In our system, a radiologist module, consisting of a feature extractor (convolutional neural network; CNN) and a transformer, analyses the input imaging data and then provides an output state of the transformer representing a radiology report to the general practitioner -corresponding module (purely transformerbased).…”

“…Compared to the conference version [22], we have enhanced our framework, such that the module corresponding to the general practitioner does not only perform prognosis, but is also encouraged to make diagnostic predictions consistent with a radiologist module. The earlier version of CLIMAT relies on a simplifying assumption in relation to the inde- pendence between the diagnostic label task and non-imaging data.…”

Transformer-based Deep Learning for COVID-19 Prediction Based on Climate Variables in Indonesia

“…Transformer is one of the deep learning models with the potential to outperform existing cutting-edge time series models [16]. Transformers have been applied to datasets with long historical information to solve various problems [17], [18], [19], [20], [21], and [22]. According to a recent study, Transformer produces the best results for the time series regression problem compared to other models [23].…”