Using detailed linguistic structure in language modelling

Clarkson, Philip; Robinson, Tony

doi:10.21437/eurospeech.1999-423

Cited by 11 publications

(2 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This study first evaluated prompt performance with the perplexity measure [ 6 ]. Perplexity is a standard metric for evaluating the quality of language models and quantifies how “surprised” the model is when seeing a passage of text [ 24 ]. A passage of text with a perplexity too high may contain language errors or nonsensical content [ 30 ], while a perplexity too low may signify repetitive and uninteresting text [ 34 ].…”

Section: Study I: Prompt Engineeringmentioning

confidence: 99%

Towards AI-Driven Healthcare: Systematic Optimization, Linguistic Analysis, and Clinicians’ Evaluation of Large Language Models for Smoking Cessation Interventions

Calle,

Shao,

Liu

et al. 2024

Proceedings of the CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Creating intervention messages for smoking cessation is a labor-intensive process. Advances in Large Language Models (LLMs) offer a promising alternative for automated message generation. Two critical questions remain: 1) How to optimize LLMs to mimic human expert writing, and 2) Do LLM-generated messages meet clinical standards? We systematically examined the message generation and evaluation processes through three studies investigating prompt engineering (Study 1), decoding optimization (Study 2), and expert review (Study 3). We employed computational linguistic analysis in LLM assessment and established a comprehensive evaluation framework, incorporating automated metrics, linguistic attributes, and expert evaluations. Certified tobacco treatment specialists assessed the quality, accuracy, credibility, and persuasiveness of LLM-generated messages, using expert-written messages as the benchmark. Results indicate that larger LLMs, including ChatGPT, OPT-13B, and OPT-30B, can effectively emulate expert writing to generate well-written, accurate, and persuasive messages, thereby demonstrating the capability of LLMs in augmenting clinical practices of smoking cessation interventions.

show abstract

Section: Study I: Prompt Engineeringmentioning

confidence: 99%

Towards AI-Driven Healthcare: Systematic Optimization, Linguistic Analysis, and Clinicians’ Evaluation of Large Language Models for Smoking Cessation Interventions

Calle,

Shao,

Liu

et al. 2024

Proceedings of the CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…Perplexity seems an appropriate figure of merit when we try to predict the next word given the previous one but, if we plan to use the LM as part of a recognition or of a translation system, we should consider the correlation between perplexity and the extrinsic figures of merit we would like to optimize (such as WER). This relationship has been investigated by several authors [Iyer et al 1997;; perplexity is a coarse measure Clarkson and Robinson 1999;Klakow and Peters 2002] and it is widely known that, for instance, two LMs with the same test set perplexity can lead to different WERs. 20 Indeed, since perplexity is a coarse measure of goodness (the reciprocal of a geometrical average), two models with the same perplexity may assign different probabilities to the same particular sequence.…”

Section: Segment Generation Confusabilitymentioning

confidence: 99%

Contributions to the joint segmentation and classification of sequences (My two cents on decoding and handwriting recognition)

Boquera¹

View full text Add to dashboard Cite

To my family, past, present and future, very specially to my parents and to my wife Manuela for their unconditional love and support. v 65 3.1.8 Prediction/Forecasting 65 3.1.9 Translation 65 3.1.10 Alignment 66 3.1.11 KWS, indexing, querying by example 69 3.1.12 Unsupervised term discovery 72 3.1.13 Filtering/Denoising and Smoothing 73 3.1.14 Summarization 73 3.1.15 Joint Segmentation and Classification 74 3.2 Some extensions 80 3.2.1 Several input sequences 80 3.2.2 Interactive systems 82 xv xvi Contents 3.3 Limitations, how to face them, new proposal 87 3.4 Questionnaire when faced with a new problem 91 3.5 Analysis and evaluation measures 93 3.5.1 Quality assessment 96 3.5.2 Performance 99 3.5.3 Interaction 100 3.5.4 Other measures 102 3.6 Summary and some conclusions 102 105 4.1 Some preliminary ML concepts 106 4.1.1 (Probabilistic) graphical models 115 4.2 Two stage generative model 130 4.2.1 Hierarchy for the second stage 133 4.2.2 Limitations, extensions and generalizations 137 4.3 Classical problems of two stage generative models 141 4.3.1 Probability of observation 142 4.3.2 Decoding 144 4.3.3 Model estimation 145 4.4 Some alternative models 146 4.4.1 Relationship with Dynamic Graphical Models 146 4.4.2 Fixed dimension feature segments 150 4.4.3 Estimation of frame-wise segment posteriors 155 4.4.4 Graph transformer networks 160 4.4.5 Some non-probabilistic frameworks 163 4.5 Summary and some conclusions 165 , , , 167 5.1 Introduction 167 5.2 Recognition, parsing, decoding 170 5.3 Weighted languages and semirings 172 5.4 Some formalisms 177 5.4.1 Formal/generative grammars 177 5.4.2 Finite state automata and transducers 184 5.4.3 Recurrent transition networks 187 5.5 Deriving the composition of a regular and a CF model 194 5.5.1 State-pair transducer composition 195 5.5.2 Extension to null-transitions 200 5.5.3 Extension to model reference transitions 209 5.5.4 Transformation to homogeneous epsilon form 220 5.6 Review of parsing approaches, decoders and algorithms 225 5.7 From composition to recognition/decoding 232 5.7.1 Acyclic inputs 233 5.7.2 Semiring specific optimizations 235 5.8 Summary and some conclusions 235 239 6.1 Probabilistic decompositions 242 6.1.1 Chain rule, clustering histories 243 6.1.2 Whole sentence LMs 244 6.1.3 Combining spans of the sequence 245 6

show abstract

Three Issues in Modern Language Modeling

Klakow

2004

Mathematical Foundations of Speech and Language Processing

View full text Add to dashboard Cite

Using detailed linguistic structure in language modelling

Cited by 11 publications

References 8 publications

Towards AI-Driven Healthcare: Systematic Optimization, Linguistic Analysis, and Clinicians’ Evaluation of Large Language Models for Smoking Cessation Interventions

Towards AI-Driven Healthcare: Systematic Optimization, Linguistic Analysis, and Clinicians’ Evaluation of Large Language Models for Smoking Cessation Interventions

Contributions to the joint segmentation and classification of sequences (My two cents on decoding and handwriting recognition)

Three Issues in Modern Language Modeling

Contact Info

Product

Resources

About