On the Use of Optimal Transportation Theory to Recode Variables and Application to Database Merging

Abstract

Merging databases is a strategy of paramount interest especially in medical research. A common problem in this context comes from a variable which is not coded on the same scale in both databases we aim to merge. This paper considers the problem of finding a relevant way to recode the variable in order to merge these two databases. To address this issue, an algorithm, based on optimal transportation theory, is proposed. Optimal transportation theory gives us an application to map the measure associated with the variable in database A to the measure associated with the same variable in database B. To do so, a cost function has to be introduced and an allocation rule has to be defined. Such a function and such a rule is proposed involving the information contained in the covariates. In this paper, the method is compared to multiple imputation by chained equations and a statistical learning method and has demonstrated a better average accuracy in many situations. Applications on both simulated and real datasets show that the efficiency of the proposed merging algorithm depends on how the covariates are linked with the variable of interest.

Regularized Optimal Transport of Covariates and Outcomes in Data Recoding

“…It can also be useful to merge two databases collected during different studies that focus on the same information. As an illustration, a previous work on data recoding [9] is applied on a French cohort study, the ELFE study, where the variable of interest is the answer to the question :"how would you rate your overall health?". During the first baseline data collection wave (January to April 2011), the different possible answers were proposed in a five points ordinal scale: "excellent", "very well", "well", "fair", "bad" and during the second baseline data collection wave (May to December 2011), another five points ordinal scale was used: "very well", "well", "medium", "bad" and "very bad".…”

“…Recently, Garès et al [9] have proposed a method based on optimal transportation (OT). Assuming that the distributions of Y and Z are the same in the two databases, the OT theory ( [20]) provides a map that pushes the distribution of Y forward to the distribution of Z.…”

“…Assuming that the distributions of Y and Z are the same in the two databases, the OT theory ( [20]) provides a map that pushes the distribution of Y forward to the distribution of Z. This approach has shown better performance when compared to classical methods such as multiple imputation [9] and machine learning methods [21], but it still exhibits two limits. First, there are several contexts where it will not be true that Y has the same distributions in the two databases.…”

“…Our aim is to build upon the work of Garès et al [9] to develop a recoding method that requires less restrictive assumptions and directly targets the solution of the recoding problem. In particular, we consider that Y and Z may be distributed differently in A and B, but we still focus on databases where covariates explain the outcomes Y and Z similarly in the two databases.…”

“…The method proposed for the recoding is an application of optimal transportation where we search for a bijective mapping between the distributions of such variable in two databases. In this article, we build upon the work by Garès et al [9], where they transport the distributions of categorical outcomes assuming that they are distributed equally in the two databases. Here, we extend the scope of the model to treat all the situations where the covariates explain the outcomes similarly in the two databases.…”