Counterfactual Debiasing Inference for Compositional Action Recognition

Abstract

Compositional action recognition is a novel challenge in the computer vision community and focuses on revealing the different combinations of verbs and nouns instead of treating subject-object interactions in videos as individual instances only. Existing methods tackle this challenging task by simply ignoring appearance information or fusing object appearances with dynamic instance tracklets. However, those strategies usually do not perform well for unseen action instances. For that, in this work we propose a novel learning framework called Counterfactual Debiasing Network (CDN) to improve the model generalization ability by removing the interference introduced by visual appearances of objects/subjects. It explicitly learns the appearance information in action representations and later removes the effect of such information in a causal inference manner. Specifically, we use tracklets and video content to model the factual inference by considering both appearance information and structure information. In contrast, only video content with appearance information is leveraged in the counterfactual inference. With the two inferences, we conduct a causal graph which captures and removes the bias introduced by the appearance information by subtracting the result of the counterfactual inference from that of the factual inference. By doing that, our proposed CDN method can better recognize unseen action instances by debiasing the effect of appearances. Extensive experiments on the Something-Else dataset clearly show the effectiveness of our proposed CDN over existing state-of-the-art methods. CCS CONCEPTS• Computing methodologies → Activity recognition and understanding; Causal reasoning and diagnostics.

Combinatorial action recognition based on causal segment intervention

“…To verify the validity of the results, we compare them with the I3D [1] , STIN [6] , SAFCAR [13] , and CDF [9] models. Our model can be realized based on existing benchmark models, so SAFCAR and Interactive Fusion are used as benchmark models.…”

“…But this cannot avoid the relationship between the object category of the labeled frame and the action. Sun et al [9] proposed to use the counterfactual algorithm to solve the problem of the visual information brought about by the representation role. Our present work, on the other hand, pays more attention to some irrelevant background visual information as well as the interference of action recognition caused by irrelevant actions, so that the model learns the motion patterns in the core action data.…”

Challenges of food security in the Gulf Cooperation Council countries: an empirical analysis of fixed and random effects

“…It noted that fertilizers consumption has a statistically significant correlation with all variables except food price inflation. As the evidence of highly correlated variables can affect the precise estimation of regression which might lead to bias inference [61], the study achieves a multi-collinearity protection omission or inclusion of explanatory variables within the study model. The tested values of VIF are lower than 2 and the results of mean VIF are less than 5 (1.09), beside the degree of tolerance for explanatory variables is greater than 0.5 (Table 4) suggesting that there is no multicollinearity between the explanatory variables, so the population, fertilizers consumption and food prices inflation are included in the study models as social, agri-environmental, and economic keys challenges are affected the food security indicators in GCC countries.…”