Short Note on Comparing Stacking Modelling Versus Cannistraci-Hebb Adaptive Network Automata for Link Prediction in Complex Networks

Abstract

Link prediction is an iconic problem in complex networks because deals with the ability to predict nonobserved existing or future parts of the network structure. The impact of this prediction on real applications can be disruptive: from prediction of covert links between terrorists in their social networks to repositioning of drugs in molecular diseasome networks. Here we compare: (1) an ensemble meta-learning method (Ghasemian et al.), which uses an artificial intelligence (AI) stacking strategy to create a single meta-model from hundreds of other models; (2) a structural predictability method (SPM, Lü et al.), which relies on a theory derived from quantum mechanics and does not assume any model; (3) a modelling rule named Cannistraci-Hebb (CH, Muscoloni et al.), which relies on one brain-bioinspired model adapting to the intrinsic network structure.We conclude that brute-force stacking of algorithms by AI does not perform better than (and is often significantly outperformed by) SPM and one simple brain-bioinspired rule such as CH. This agrees with the Gödel incompleteness: stacking is optimal but incomplete, you cannot squeeze out more than what is already in your features. Hence, we should also pursue AI that resembles human-like physical ‘understanding’ of simple generalized rules associated to complexity. The future might be populated by AI that ‘steals for us the fire from Gods’, towards machine intelligence that creates new rules rather than stacking the ones already known.

Early Retrieval Problem and Link Prediction Evaluation via the Area Under the Magnified ROC

“…These methods are tested over 5500 simulations (550 networks x 10 repetitions) and the result is that: in 66% cases AUC-PR and AUC-ROC agrees that CHA performs better than SBM; in 31% cases CHA has higher AUC-PR and SBM has higher AUC-ROC; in 3% cases SBM has higher AUC-PR and CHA has higher AUC-ROC. This means that in the study of Muscoloni and Cannistraci 18,19 there is around 34% disagreement between AUC-ROC and AUC-PR that indeed is in line with Tao Zhou 17 mentioning that 1/3 of cases have disagreement. Interestingly, in a recent study, Tao Zhou 20 offers a valid theoretical explanation of the conditions under which AUC-PR and AUC-ROC evaluation agrees.…”

“…Truchon and Bayly noted that, in terms of the 'early recognition', the first case is clearly better than second one, which is also significantly better than the third one. We analysed the results of a large-scale experimental study 18,19 on link prediction, which reports AUC-ROC and AUC-PR evaluations of two landmark link prediction methods -Cannistraci-Hebb adaptive network automata (CHA) and stochastic block model (SBM) -tested over 5500 simulations (550 networks x 10 repetitions). One of the interesting finding is that in 31% cases CHA has higher AUC-PR and SBM has higher AUC-ROC.…”

“…At this stage of the study we deepen our investigation. Among all 550 real networks recently analysed in the large-scale experimental link-prediction paper of Muscoloni and Cannistraci 18,19 , in Fig. 3 we selected three representative scenarios: 1) inverse trend of AUC-PR and AUC-ROC (which is the same scenario discussed in the previous figures); 2) large AUC-PR difference and similar AUC-ROC; 3) similar trend of AUC-PR and AUC-ROC (which is similar to the scenario considered by Tao Zhou in one of his recent studies 20 ).…”

“…Because of these issues associated to the AUC-ROC evaluation in link prediction, several studies suggested to adopt only the AUC-PR 9,10,14 which indeed is becoming increasingly popular in link prediction evaluation. However, in a recent and milestone review article on link prediction, Tao Zhou 17 commented that (text is reported verbatim): "In summary, empirical comparisons and systematic analyses about evaluation metrics for link prediction are important tasks in this stage because many publications use AUC-ROC as the sole metric, while an ongoing empirical study (by Y.-L. Lee and T. Zhou, unpublished) shows that in about 1/3 pairwise comparisons, AUC-ROC and AUC-PR give different ranks of algorithms, and a recent large-scale experimental study also shows inconsistent results by AUC-ROC and AUC-PR 18,19 . Before a comprehensive and explicit picture obtained, my suggestion is that we have to at least simultaneously report both AUC-ROC and AUC-PR, and only if an algorithm can obviously beat another one in both metrics for a network, we can say the former performs better in this case".…”

“…Before a comprehensive and explicit picture obtained, my suggestion is that we have to at least simultaneously report both AUC-ROC and AUC-PR, and only if an algorithm can obviously beat another one in both metrics for a network, we can say the former performs better in this case". The most updated version of the large-scale experimental study 18,19 to which Tao Zhou refers in his review, reports AUC-ROC and AUC-PR evaluations of two landmark link prediction methods: Cannistraci-Hebb adaptive network automata (CHA) and stochastic block model (SBM). These methods are tested over 5500 simulations (550 networks x 10 repetitions) and the result is that: in 66% cases AUC-PR and AUC-ROC agrees that CHA performs better than SBM; in 31% cases CHA has higher AUC-PR and SBM has higher AUC-ROC; in 3% cases SBM has higher AUC-PR and CHA has higher AUC-ROC.…”