L2F/INESC-ID at SemEval-2017 Tasks 1 and 2: Lexical and semantic
            features in word and textual similarity

Fialho, Pedro; Rodrigues, Hugo; Coheur, Luísa; Quaresma, Paulo

doi:10.18653/v1/s17-2032

Cited by 7 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The overall results obtained following this setting are promising (with an accuracy of 0.835 in the test set). Comparing our results with those of the participants in the ASSIN Challenge [14], our approach obtains an overall accuracy close to the results obtained by the best performing system (0.8385 of accuracy obtained by Fialho et al [33]) and outperforms all the proposed systems in the corresponding macro F1-score metric (the best performing system [33] reported 0.71 of macro F1-score). Additionally, we observe that the performance of our approach improved with semantic-based features, albeit not significantly.…”

Section: Discussionsupporting

confidence: 78%

“…Fialho et al [33] extracted several metrics for each pair of sentences, namely edit distance, words overlap, BLEU [28] and ROUGE [34], among others. They reported several experiments considering different preprocessing steps in the NLP pipeline, namely: original sentences (baseline), removing stop-words, lower-case words and clusters of words.…”

Section: Feature-engineered Machine Learning Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Recognizing Textual Entailment: Challenges in the Portuguese Language

Rocha

Cardoso

2018

Information

View full text Add to dashboard Cite

Abstract:Recognizing textual entailment comprises the task of determining semantic entailment relations between text fragments. A text fragment entails another text fragment if, from the meaning of the former, one can infer the meaning of the latter. If such relation is bidirectional, then we are in the presence of a paraphrase. Automatically recognizing textual entailment relations captures major semantic inference needs in several natural language processing (NLP) applications. As in many NLP tasks, textual entailment corpora for English abound, while the same is not true for more resource-scarce languages such as Portuguese. Exploiting what seems to be the only Portuguese corpus for textual entailment and paraphrases (the ASSIN corpus), in this paper, we address the task of automatically recognizing textual entailment (RTE) and paraphrases from text written in the Portuguese language, by employing supervised machine learning techniques. We employ lexical, syntactic and semantic features, and analyze the impact of using semantic-based approaches in the performance of the system. We then try to take advantage of the bi-dialect nature of ASSIN to compensate its limited size. With the same aim, we explore modeling the task of recognizing textual entailment and paraphrases as a binary classification problem by considering the bidirectional nature of paraphrases as entailment relationships. Addressing the task as a multi-class classification problem, we achieve results in line with the winner of the ASSIN Challenge. In addition, we conclude that semantic-based approaches are promising in this task, and that combining data from European and Brazilian Portuguese is less straightforward than it may initially seem. The binary classification modeling of the problem does not seem to bring advantages to the original multi-class model, despite the outstanding results obtained by the binary classifier for recognizing textual entailments.

show abstract

Section: Discussionsupporting

confidence: 78%

Section: Feature-engineered Machine Learning Modelsmentioning

confidence: 99%

Recognizing Textual Entailment: Challenges in the Portuguese Language

Rocha

Cardoso

2018

Information

View full text Add to dashboard Cite

show abstract

“…15 Select methods are highlighted below. 83.02• 15.50 compiLIG 76.84 14.64 compiLIG 79.10 14.94 DT TEAM (Maharjan et al, 2017) 85.36 DT TEAM (Maharjan et al, 2017) 83.60 DT TEAM (Maharjan et al, 2017) 83.29 FCICU (Hassan et al, 2017) 82.17 ITNLPAiKF (Liu et al, 2017) 82.31 ITNLPAiKF (Liu et al, 2017) 82.31 ITNLPAiKF (Liu et al, 2017) 81.59 L2F/INESC-ID (Fialho et al, 2017 10: STS 2017 rankings ordered by average correlation across tracks 1-5. Performance is reported by convention as Pearson's r × 100.…”

Section: Methodsmentioning

confidence: 99%

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

Cer¹,

Diab²,

Agirre³

et al. 2017

Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)

1,129

775

View full text Add to dashboard Cite

Semantic Textual Similarity (STS) measures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems. The STS shared task is a venue for assessing the current state-of-the-art. The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track exploring MT quality estimation (MTQE) data. The task obtained strong participation from 31 teams, with 17 participating in all language tracks. We summarize performance and review a selection of well performing methods. Analysis highlights common errors, providing insight into the limitations of existing models. To support ongoing work on semantic representations, the STS Benchmark is introduced as a new shared training and evaluation set carefully selected from the corpus of English STS shared task data (2012)(2013)(2014)(2015)(2016)(2017).

show abstract

“…In the end we received a wide variety of participants: proposing distributional semantic models learnt directly from raw corpora, using syntactic features, exploiting knowledge from lexical resources, and hybrid approaches combining corpus-based and knowledge-based clues. Due to lack of space we cannot describe all the systems in detail, but we recommend the reader to refer to the system description papers for more information about the individual systems: HCCL (He et al, 2017), Citius (Gamallo, 2017), jmp8 (Melka and Bernard, 2017), l2f (Fialho et al, 2017), QLUT (Meng et al, 2017), RUFINO (Jimenez et al, 2017), MERALI (Mensa et al, 2017), Luminoso (Speer and Lowry-Duda, 2017), hhu (Qasem-iZadeh and Kallmeyer, 2017), Mahtab (Ranjbar et al, 2017), SEW (Delli Bovi and Raganato, 2017) and Wild Devs (Rotari et al, 2017), and OoO.…”

Section: Participating Systemsmentioning

confidence: 99%

SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity

Camacho-Collados

Pilehvar

Collier

et al. 2017

Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)

118

132

View full text Add to dashboard Cite

This paper introduces a new task on Multilingual and Cross-lingual Semantic Word Similarity which measures the semantic similarity of word pairs within and across five languages: English, Farsi, German, Italian and Spanish. High quality datasets were manually curated for the five languages with high inter-annotator agreements (consistently in the 0.9 ballpark). These were used for semi-automatic construction of ten cross-lingual datasets. 17 teams participated in the task, submitting 24 systems in subtask 1 and 14 systems in subtask 2. Results show that systems that combine statistical knowledge from text corpora, in the form of word embeddings, and external knowledge from lexical resources are best performers in both subtasks. More information can be found on the task website: http://alt.qcri. org/semeval2017/task2/ .

show abstract

L2F/INESC-ID at SemEval-2017 Tasks 1 and 2: Lexical and semantic features in word and textual similarity

Cited by 7 publications

References 13 publications

Recognizing Textual Entailment: Challenges in the Portuguese Language

Recognizing Textual Entailment: Challenges in the Portuguese Language

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity

Contact Info

Product

Resources

About