UZH@SMM4H: System Descriptions

Ellendorff, Tilia; Cornelius, Joseph; Gordon, Heath; Colic, Nicola; Rinaldi, Fabio

doi:10.18653/v1/w18-5916

Cited by 3 publications

(7 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Those scores were reported across different kinds of classification tasks, generally showing very good scores for straightforward tasks that include only document‐level classification. For these binary classification tasks F1, precision, and recall scores higher than 0.9 is becoming more common 51–55 . Scores decrease for harder classification tasks such as normalisation to controlled vocabularies, commonly ranging between 0.2 and 0.6 in precision, recall or F1.…”

Section: Resultsmentioning

confidence: 99%

“…Ambiguity also increases complexity, for example when drug name can have multiple synonyms, trade names, or multiple correct labels 39,56 . Noise is a concept that generally refers to data being unreliable due to their unstructured and naturally expressed form, thus causing errors both while labelling gold‐standard data and when processing and predicting on new data 31,52 …”

Section: Resultsmentioning

confidence: 99%

“…For these binary classification tasks F1, precision, and recall scores higher than 0.9 is becoming more common. [51][52][53][54][55] Scores decrease for harder classification tasks such as normalisation to controlled vocabularies, commonly ranging between 0.2 and 0.6 in precision, recall or F1. In part, this large variation between reported scores, spanning 0.2-0.6, can be seen for normalisation because the included papers used the same evaluation metrics but applied them in different ways.…”

Section: Evaluation Of Algorithmsmentioning

confidence: 99%

“…39,56 Noise is a concept that generally refers to data being unreliable due to their unstructured and naturally expressed form, thus causing errors both while labelling gold-standard data and when processing and predicting on new data. 31,52 Sparsity or imbalance within the data ML or neural networks architectures require annotated data during the learning process. When training classifiers for forum posts to identify drugs, Chee et al 65 noted that some drugs did not have enough mentions in posts in order to train and evaluate robust classifiers, while Arnold et al 57 also describe rare entities and unseen data as a problem.…”

Section: Practical Challenges and Research Gaps That Constitute Barri...mentioning

confidence: 99%

See 3 more Smart Citations

Automated data analysis of unstructured grey literature in health research: A mapping review

Schmidt,

Mohamed,

Meader

et al. 2023

Research Synthesis Methods

View full text Add to dashboard Cite

The amount of grey literature and ‘softer’ intelligence from social media or websites is vast. Given the long lead‐times of producing high‐quality peer‐reviewed health information, this is causing a demand for new ways to provide prompt input for secondary research. To our knowledge, this is the first review of automated data extraction methods or tools for health‐related grey literature and soft data, with a focus on (semi)automating horizon scans, health technology assessments (HTA), evidence maps, or other literature reviews. We searched six databases to cover both health‐ and computer‐science literature. After deduplication, 10% of the search results were screened by two reviewers, the remainder was single‐screened up to an estimated 95% sensitivity; screening was stopped early after screening an additional 1000 results with no new includes. All full texts were retrieved, screened, and extracted by a single reviewer and 10% were checked in duplicate. We included 84 papers covering automation for health‐related social media, internet fora, news, patents, government agencies and charities, or trial registers. From each paper, we extracted data about important functionalities for users of the tool or method; information about the level of support and reliability; and about practical challenges and research gaps. Poor availability of code, data, and usable tools leads to low transparency regarding performance and duplication of work. Financial implications, scalability, integration into downstream workflows, and meaningful evaluations should be carefully planned before starting to develop a tool, given the vast amounts of data and opportunities those tools offer to expedite research.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Evaluation Of Algorithmsmentioning

confidence: 99%

Section: Practical Challenges and Research Gaps That Constitute Barri...mentioning

confidence: 99%

See 2 more Smart Citations

Automated data analysis of unstructured grey literature in health research: A mapping review

Schmidt,

Mohamed,

Meader

et al. 2023

Research Synthesis Methods

View full text Add to dashboard Cite

show abstract

“…For this last merging step, we gave the system resulting from merging early stopping systems nine times the weight of the other system which resulted from merging systems trained for a fixed number of epochs. For the last run (MTL+BERT), we combined predictions from all 20 BERT systems with the first system and a second MTL configuration which uses different word embeddings (Ellendorff et al, 2018) and omits lexicon features.…”

Section: Experiments and System Descriptionsmentioning

confidence: 99%

Approaching SMM4H with Merged Models and Multi-task Learning

Ellendorff¹,

Furrer²,

Colic³

et al. 2019

Proceedings of the Fourth Social Media Mining for Health Applications (#SMM4H) Workshop &Amp; Shared Task

Self Cite

View full text Add to dashboard Cite

We describe our submissions to the 4th edition of the Social Media Mining for Health Applications (SMM4H) shared task. Our team (UZH) participated in two sub-tasks: Automatic classifications of adverse effects mentions in tweets (Task 1) and Generalizable identification of personal health experience mentions (Task 4). For our submissions, we exploited ensembles based on a pretrained language representation with a neural transformer architecture (BERT) (Tasks 1 and 4) and a CNN-BiLSTM(-CRF) network within a multi-task learning scenario (Task 1). These systems are placed on top of a carefully crafted pipeline of domain-specific preprocessing steps.

show abstract

Automated data extraction of unstructured grey literature in health research: a mapping review of the current research literature

Schmidt

Mohamed

Meader

et al. 2023

Preprint

View full text Add to dashboard Cite

The amount of grey literature and "softer" intelligence from social media or websites is vast. Given the long lead-times of producing high-quality peer-reviewed health information this is causing a demand for new ways to provide prompt input for secondary research. To our knowledge this is the first review of automated data extraction methods or tools for health-related grey literature and soft intelligence, with a focus on (semi)automating horizon scans, health technology assessments, evidence maps, or other literature reviews. We searched six databases to cover both health- and computer-science literature. After deduplication, 10% of the search results were screened by two reviewers, the remainder was single-screened up to an estimated 95% sensitivity; screening was stopped early after screening an additional 1000 results with no new includes. All full texts were retrieved, screened, and extracted by a single reviewer and 10% were checked in duplicate. We included 84 papers covering automation for health-related social media, internet fora, news, patents, government agencies and charities, or trial registers. From each paper we answered three research questions: Firstly, important functionalities for users of the tool or method; secondly, information about the level of support and reliability; and thirdly, practical challenges and research gaps. Poor availability of code, data, and usable tools leads to low transparency regarding performance and duplication of work. Financial implications, scalability, integration into downstream workflows, and meaningful evaluations should be carefully planned before starting to develop a tool, given the vast amounts of data and opportunities those tools offer to expedite research.

show abstract

UZH@SMM4H: System Descriptions

Cited by 3 publications

References 6 publications

Automated data analysis of unstructured grey literature in health research: A mapping review

Automated data analysis of unstructured grey literature in health research: A mapping review

Approaching SMM4H with Merged Models and Multi-task Learning

Automated data extraction of unstructured grey literature in health research: a mapping review of the current research literature

Contact Info

Product

Resources

About