The Domain-Specific Task of CLEF - Specific Evaluation Strategies in Cross-Language Information Retrieval

Kluck, Michael; Gey, Fredric C.

doi:10.1007/3-540-44645-1_5

Cited by 31 publications

(11 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The accuracy of this translation will directly affect the relevance of the questions retrieved in the target language. Kluck and Gey [15] point out that in many cases there exists a clear difference between the domain-specific meaning and the common meaning of a word. This means that it can be difficult to use general translation of a word for domain-specific information retrieval.…”

Section: Challenges In Domain-specific Translationmentioning

confidence: 98%

“…Even more, they assess the strengths and limitations of LSI for traceability recovery and devise the need for an incremental approach. Kluck and Gey describe the domain-specific cross-language information retrieval (CLIR) task of CLEF, why and how it is important and how it differs from general cross-language retrieval problem associated with the general CLEF collections [14].…”

Section: Studies On Information Retrieval In Software Engineeringmentioning

confidence: 99%

See 1 more Smart Citation

Domain-specific cross-language relevant question retrieval

Xing

Xia

et al. 2016

Proceedings of the 13th International Conference on Mining Software Repositories

View full text Add to dashboard Cite

In software development process, developers often seek solutions to the technical problems they encounter by searching relevant questions on Q&A sites. When developers fail to find solutions on Q&A sites in their native language (e.g., Chinese), they could translate their query and search on the Q&A sites in another language (e.g., English). However, developers who are non-native English speakers often are not comfortable to ask or search questions in English, as they do not know the proper translation of the Chinese technical words into the English technical words. Furthermore, the process of manually formulating cross-language queries and determining the weight of query words is a tedious and time-consuming process.For the purpose of helping Chinese developers take advantage of the rich knowledge base of the English version of Stack Overflow and simplify the retrieval process, we propose an automated crosslanguage relevant question retrieval (CLRQR) system to retrieve relevant English questions on Stack Overflow for a given Chinese question. Our CLRQR system first extracts essential information (both Chinese and English) from the title and description of the input Chinese question, then performs domain-specific translation of the essential Chinese information into English, and formulates a query with highest-scored English words for retrieving relevant questions in a repository of 684,599 Java questions in English from Stack Overflow. To evaluate the performance of our proposed approach, we also propose four online retrieval approaches as baselines. We randomly select 80 Java questions in SegmentFault and V2EX (two Chinese Q&A websites for computer programming) as the query Chinese questions. Each approach returns top-10 most relevant questions for a given Chinese question. We invite 5 users to evaluate the relevance of the retrieved English questions. The experiment results show that CLRQR system outperforms the four baseline approaches, and the statistical tests show the improvements are significant.

show abstract

Section: Challenges In Domain-specific Translationmentioning

confidence: 98%

Section: Studies On Information Retrieval In Software Engineeringmentioning

confidence: 99%

Domain-specific cross-language relevant question retrieval

Xing

Xia

et al. 2016

Proceedings of the 13th International Conference on Mining Software Repositories

View full text Add to dashboard Cite

show abstract

“…The information which is provided by these documents is far more targeted than news stories and contains much terminology. Kluck and Gey (2001) discuss the implications of performing CLIR in a vertical domain. They claim that the users of domain-specific collections are typically interested in the completeness of results.…”

Section: Clef 2000mentioning

confidence: 99%

European research letter: Cross‐language system evaluation: The CLEF campaigns

Peters¹,

Braschler

2001

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

The goals of the CLEF (Cross-Language Evaluation Forum) series of evaluation campaigns for information retrieval systems operating on European languages are described. The difficulties of organizing an activity which aims at an objective evaluation of systems running on and over a number of different languages are examined. The discussion includes an analysis of the first results and proposals for possible developments in the future.

show abstract

“…more likely to be a person or entity) vs. medical domain documents (more likely to be a chemical). Challenges for domainspecific CLIR, in particular the problem of distinguishing domainspecific meanings, have been noted in [12]. We argue that these variations can be captured by successfully matching training resources to target corpora.…”

Section: Introductionmentioning

confidence: 98%

Resource selection for domain-specific cross-lingual IR

Rogati

Yang

2004

Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

An under-explored question in cross-language information retrieval (CLIR) is to what degree the performance of CLIR methods depends on the availability of high-quality translation resources for particular domains. To address this issue, we evaluate several competitive CLIR methods -with different training corpora -on test documents in the medical domain. Our results show severe performance degradation when using a general-purpose training corpus or a commercial machine translation system (SYSTRAN), versus a domain-specific training corpus. A related unexplored question is whether we can improve CLIR performance by systematically analyzing training resources and optimally matching them to target collections. We start exploring this problem by suggesting a simple criterion for automatically matching training resources to target corpora. By using cosine similarity between training and target corpora as resource weights we obtained an average of 5.6% improvement over using all resources with no weights. The same metric yields 99.4% of the performance obtained when an oracle chooses the optimal resource every time.

show abstract

The Domain-Specific Task of CLEF - Specific Evaluation Strategies in Cross-Language Information Retrieval

Cited by 31 publications

References 7 publications

Domain-specific cross-language relevant question retrieval

Domain-specific cross-language relevant question retrieval

European research letter: Cross‐language system evaluation: The CLEF campaigns

Resource selection for domain-specific cross-lingual IR

Contact Info

Product

Resources

About