Approximating Style by N-gram-based Annotation

Andresen, Melanie; Zinsmeister, Heike

doi:10.18653/v1/w17-4913

Cited by 6 publications

(4 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our codebook was developed based on the literature 47–49 and updated during the annotation process through regular discussions among the study team ( Supplementary Appendix SA ). To capture the ambiguities present in natural communication, annotators labeled turns on a 0–3 scale, 50 with 3 implying certain relevance to symptoms. Study team consensus determined that an average score of 2 or higher indicated symptom content (the “broad gold standard”).…”

Section: Methodsmentioning

confidence: 99%

Development of a keyword library for capturing PRO-CTCAE-focused “symptom talk” in oncology conversations

et al. 2023

View full text Add to dashboard Cite

Objectives As computational methods for detecting symptoms can help us better attend to patient suffering, the objectives of this study were to develop and evaluate the performance of a natural language processing keyword library for detecting symptom talk, and to describe symptom communication within our dataset to generate insights for future model building. Materials and Methods This was a secondary analysis of 121 transcribed outpatient oncology conversations from the Communication in Oncologist-Patient Encounters trial. Through an iterative process of identifying symptom expressions via inductive and deductive techniques, we generated a library of keywords relevant to the Patient-Reported Outcome version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) framework from 90 conversations, and tested the library on 31 additional transcripts. To contextualize symptom expressions and the nature of misclassifications, we qualitatively analyzed 450 mislabeled and properly labeled symptom-positive turns. Results The final library, comprising 1320 terms, identified symptom talk among conversation turns with an F1 of 0.82 against a PRO-CTCAE-focused gold standard, and an F1 of 0.61 against a broad gold standard. Qualitative observations suggest that physical symptoms are more easily detected than psychological symptoms (eg, anxiety), and ambiguity persists throughout symptom communication. Discussion This rudimentary keyword library captures most PRO-CTCAE-focused symptom talk, but the ambiguity of symptom speech limits the utility of rule-based methods alone, and limits to generalizability must be considered. Conclusion Our findings highlight opportunities for more advanced computational models to detect symptom expressions from transcribed clinical conversations. Future improvements in speech-to-text could enable real-time detection at scale.

show abstract

Section: Methodsmentioning

confidence: 99%

Development of a keyword library for capturing PRO-CTCAE-focused “symptom talk” in oncology conversations

et al. 2023

View full text Add to dashboard Cite

show abstract

“…As we provide the annotations of two annotators for most plays, the data can also be used to investigate annotation disagreement. One may investigate if annotation disagreements point to ambiguous and potentially crucial text passages or look into the causes of disagreements (Andresen, Vauth, & Zinsmeister, 2020;Gius & Jacke, 2017).…”

Section: Reuse Potentialmentioning

confidence: 99%

Knowledge Distribution in German Drama: An Annotated Corpus

Andresen,

Krautter,

Pagel

et al. 2024

Journal of Open Humanities Data

View full text Add to dashboard Cite

What do characters in theater plays know about character relations, and how does the distribution of knowledge evolve over a play's course? We present a dataset of 30 German plays annotated with information about the distribution of knowledge about character relations (such as "A learns from B that C is the parent of D"). All plays were manually annotated by two independent annotators in the Q:TRACK project, which aims to systematically model character knowledge. The dataset is available on GitHub and Zenodo and can be reused, for example, for systematic studies of knowledge in plays or for analyzing annotator disagreements.

show abstract

“…The purpose of including the action history is to capture additional information from human input during interactive demonstrations. An extended policy π n , which operates on the extended states π n : S t,n → a t , is useful for modeling human actions in a manner similar to n-grams text models in natural language processing (NLP) (e.g., [72], [73], [74]). Of course, the analogy with n-gram models in NLP works only if both state and action spaces are discrete.…”

Section: ) Markov Decision Process With Extended Statementioning

confidence: 99%

Winning Isn't Everything: Enhancing Game Development with Intelligent Agents

Zhao¹,

Borovikov²,

Silva³

et al. 2019

Preprint

View full text Add to dashboard Cite

Recently, there have been several high-profile achievements of agents learning to play games against humans and beat them. In this paper, we study the problem of training intelligent agents in service of game development. Unlike the agents built to "beat the game", our agents aim to produce human-like behavior to help with game evaluation and balancing. We discuss two fundamental metrics based on which we measure the human-likeness of agents, namely skill and style, which are multi-faceted concepts with practical implications outlined in this paper. We report four case studies in which the style and skill requirements inform the choice of algorithms and metrics used to train agents; ranging from A* search to state-of-the-art deep reinforcement learning. We, further, show that the learning potential of state-of-the-art deep RL models does not seamlessly transfer from the benchmark environments to target ones without heavily tuning their hyperparameters, leading to linear scaling of the engineering efforts and computational cost with the number of target domains.

show abstract

Approximating Style by N-gram-based Annotation

Cited by 6 publications

References 20 publications

Development of a keyword library for capturing PRO-CTCAE-focused “symptom talk” in oncology conversations

Development of a keyword library for capturing PRO-CTCAE-focused “symptom talk” in oncology conversations

Knowledge Distribution in German Drama: An Annotated Corpus

Winning Isn't Everything: Enhancing Game Development with Intelligent Agents

Contact Info

Product

Resources

About