Enhanced distance-aware self-attention and multi-level match for sentence semantic matching

Semantic Sentence Matching Based on Multiple Parallelly Organized Interaction Layers at Various Granularity Combinations With Two-Stage Aggregation Strategy

“…Wan et al put forward a model to match two sentences with multiple positional sentence representations, by the aggregation over interactions between different positional sentence representations and the employment of K-Max pooling, the semantic relationship between sentences can be well captured by its model design [30]. In order for more effectual extraction of sentence interactive features, in many recent studies on sentence matching, the cross-attention mechanism has been largely applied, intended for a more accurate sentence alignment, which is usually implemented by stacking multiple of them so that more in-depth interaction information can be possibly exploited [15], [31]- [33]. For instance, Chen et al utilized two Bi-LSTM (Bi-directional Long Short-Term Memory) in its model with the first to encode sentence semantic information, and the second to aggregate semantic information and sentence alignment information extracted by attention mechanism, so that more effective sentence representation vectors can be obtained for final semantic matching [3].…”

“…Secondly, in recent years, due to wide application of at-tention mechanism in sentence matching models, it's not uncommon to find studies trying to address this issue by stacking multiple cross-attention layers so that implicit interactive features can be more profoundly extracted [15], [16]. However, for this interaction-layer-stacking strategy, there may exist some drawbacks.…”

“…Once the interactive features retrieved at a low layer contain major errors, the error information may remain long in the network and be propagated all the way to the high layers, causing the alignments information at top layer to be less distinctive for final sentence matching. On the other hand, because the interactive representation of one sentence is obtained by dynamically attending to the information of another one, the interactive representation obtained at intermediate layers may vary a lot between layers, making it hard to really capture stable and effective semantic alignments information between sentences [9], [15]. Thirdly, given the advantages of multi-grained semantic information which has shown great potential in improving sentence matching accuracy, many researches try to utilize CNN (Convolutional Neural Network) to encode sentences at multiple granularities.…”

“…To ensure that all the convoluted representations of each word can be fit to be fed into the identical subsequent layer, we impose the constraint that all convoluted representations live in the same space and have the same dimensionality by applying the same number of filters to each convolutional network. In addition, although the above encoding process using CNN network can already encode the contextual information to a certain extent, but the contextual features captured by CNN are mostly local semantics within a relatively small word span in sentence, for the complex long-distance dependency information, CNN is still unable to capture it, hence, in our work, in order to well capture such global contextual information, we follow the method in [15] to enhance the contextual representation of each word by a distance-aware self-attention mechanism, the equations are simplified as follows:…”