Proceedings of the 15th Workshop on Biomedical Natural Language Processing 2016
DOI: 10.18653/v1/w16-2916
|View full text |Cite
|
Sign up to set email alerts
|

Measuring the State of the Art of Automated Pathway Curation Using Graph Algorithms - A Case Study of the mTOR Pathway

Abstract: This paper evaluates the difference between human pathway curation and current NLP systems. We propose graph analysis methods for quantifying the gap between human curated pathway maps and the output of state-of-the-art automatic NLP systems. Evaluation is performed on the popular mTOR pathway. Based on analyzing where current systems perform well and where they fail, we identify possible avenues for progress.

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...
2
1
1
1

Citation Types

0
9
0

Year Published

2017
2017
2019
2019

Publication Types

Select...
2
1
1

Relationship

1
3

Authors

Journals

citations
Cited by 4 publications
(9 citation statements)
references
References 29 publications
(34 reference statements)
0
9
0
Order By: Relevance
“…Of particular note is the DREAM network inference challenge where prior-based methods took the top two positions in an independent evaluation ( 18 ). Others have looked at the overlap between curated models and literature-derived networks ( 25 , 37 ). Our work is the first to carefully examine whether the biochemical pathways extracted by the machine can be successfully combined with human-curated models in the context of a specific analytical task.…”
Section: Extrinsic Evaluation: Discovery Of Biological Hypothesesmentioning
confidence: 99%
“…Of particular note is the DREAM network inference challenge where prior-based methods took the top two positions in an independent evaluation ( 18 ). Others have looked at the overlap between curated models and literature-derived networks ( 25 , 37 ). Our work is the first to carefully examine whether the biochemical pathways extracted by the machine can be successfully combined with human-curated models in the context of a specific analytical task.…”
Section: Extrinsic Evaluation: Discovery Of Biological Hypothesesmentioning
confidence: 99%
“…Spranger et al (2016) propose a number of graph overlap algorithms for quantifying the difference and similarity of two pathways. Here we employ the same measures.…”
Section: Discussionmentioning
confidence: 99%
“…If we analyze the classifiers from this paper in more detail, results (Figure 1, Table 4: This table compares macro F-score performance of the classifiers discussed in this paper with results reported in Spranger et al (2016) for the strictest matching strategy (nmeq, sboeq) the best classifiers reach a macro F-score of 12…”
Section: General Trends Subgraphs Overlapmentioning
confidence: 94%
See 1 more Smart Citation
“…Recently, there has been some attempt at remedying the situation and new datasets and evaluation measures have been proposed. For instance, Spranger et al (2016) use the popular human-generated mTOR pathway map (Caron et al, 2010;Efeyan and Sabatini, 2010;Katiyar et al, 2009) and quantify the performance of a particular APC system and its ability to recreate the complete pathway automatically. Results reported were mixed.…”
Section: Introductionmentioning
confidence: 99%