Auxiliary Loss for BERT-Based Paragraph Segmentation

Abstract

Paragraph segmentation is a text segmentation task. Iikura et al. achieved excellent results on paragraph segmentation by introducing focal loss to Bidirectional Encoder Representations from Transformers.In this study, we investigated paragraph segmentation on Daily News and Novel datasets. Based on the approach proposed by Iikura et al., we used auxiliary loss to train the model to improve paragraph segmentation performance. Consequently, the average F1-score obtained by the approach of Iikura et al. was 0.6704 on the Daily News dataset, whereas that of our approach was 0.6801. Our approach thus improved the performance by approximately 1%. The performance improvement was also confirmed on the Novel dataset. Furthermore, the results of two-tailed paired t-tests indicated that there was a statistical significance between the performance of the two approaches.

Novel framework for dialogue summarization based on factual-statement fusion and dialogue segmentation

“…The input in BERT includes two special tokens, [CLS] and [SEP]: [CLS] signifies the beginning of the input text, while [SEP] marks the junction between two input segments. Considering the study on the impact of BERT input format on text segmentation results in [ 31 ], we add [SEP] tokens between each dialogue turns before inputting into BERT. As a result, the input of BERT is changed to the following format: …”