Attention-Translation-Relation Network for Scalable Scene Graph Generation

Scene Graph Generation: A Comprehensive Survey

“…Fig. 7: Two types of visual translation embedding approaches according to whether to embed the predicate into N-dimensional space [70] or not [68], beyond the subject and object embedding.…”

“…Translation Embedding among Subject, Object and Predicate: Another extension is ATR-Net (Attention-Translation-Relation Network), proposed by Gkanatsios et al [70] which projects the visual features from the subject, the object region and their union into a score space as S, O and P with multi-head language and spatial attention guided. Let A denotes the attention matrix of all predicates, Eq.…”

“…Most works have seen PhrDet as a multiclass problem and they use k = 1 to reward the correct top-1 prediction for each pair. While other works [63], [85], [105] tackle this as a multilabel problem and they use a k equal to the number of predicate classes to allow for predicate co-occurrences [70]. Some works [42], [70], [90], [129], [164] have also identified this inconsistency and interpret it as whether there is graph constraint (i.e., the k is the maximum number of edges allowed between a pair of object nodes).…”

“…While other works [63], [85], [105] tackle this as a multilabel problem and they use a k equal to the number of predicate classes to allow for predicate co-occurrences [70]. Some works [42], [70], [90], [129], [164] have also identified this inconsistency and interpret it as whether there is graph constraint (i.e., the k is the maximum number of edges allowed between a pair of object nodes). The unconstrained metric (i.e., no graph constraint) evaluates models more reliably, since it does not require a perfect triplet match to be the top-1 prediction, which is an unreasonable expectation in a dataset with plenty of synonyms and mislabeled annotations.…”

“…Obviously, the SGGen+ metric above has a similar motivation as removing the graph constraint. Gkanatsios et al [70] re-formulated the metric as Recall k @K(R k @K). k = 1 is equivalent to 'graph constraints" and a larger k to "no graph constraints", also expressed as ngR k @K. For n examined subject-object pairs in an image, Recall k @K(R k @K) keeps the top-k predictions per pair and examines the K most confident out of nk total.…”