GEMv2: Multilingual NLG Benchmarking in a Single Line of Code

Gehrmann, Sebastian; Bhattacharjee, Abhik; Abinaya, Mahendiran,; Wang, Alex; Papangelis, Alexandros; Madaan, Aman; McMillan-Major, Angelina; Shvets, Anna; Upadhyay, Ashish Datt; Wilie, Bryan; Bhagavatula, Chandra; You, Chaobin; Thomson, Craig; Gârbacea, Cristina; Wang, Dakuo; Deutsch, Daniel; Xiong, Deyi; Jin, Dayong; Gkatzia, Dimitra; Radev, Dragomir; Clark, Elizabeth A.; Durmus, Esin; Ladhak, Faisal; Ginter, Filip; Winata, Genta Indra; Strobelt, Hendrik; Hayashi, Hiroaki; Novikova, Jekaterina; Kanerva, Jenna; Chim, Jenny; Jordan, Clive,; Maynez, Joshua; Sedoc, João; Juraska, Juraj; Dhole, Kaustubh; Chandu, Khyathi Raghavi; Perez-Beltrachini, Laura; Ribeiro, Leonardo F. R.; Tunstall, Lewis C.; Zhang, Li; Pushkarna, Mahima; Creutz, Mathias; White, Michael; Kale, Mihir; Eddine, Moussa Kamal; Nico, Daheim,; Subramani, Nishant; Dušek, Ondřej; Liang, Paul Pu; Ammanamanchi, Pawan Sasanka; Zhu, Quanxin; Puduppully, Ratish; Kriz, Reno; Shahriyar, Rifat; Cardenas, Ronald; Mahamood, Saad; Osei, Salomey; Cahyawijaya, Samuel; Štajner, Sanja; Montella, Sebastien; Shailza,; Jolly, Shailza; Mille, Simon; Hasan, Tahmid; Shen, Tao; Adewumi, Tosin; Raunak, Vikas; Raheja, Vipul; Николаев, В. А.; Tsai, Vivian; Jernite, Yacine; Xu, Ying; Sang, Yisi; Liu, Yixin; Hou, Yufang

doi:10.48550/arxiv.2206.11249

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, generation benchmarks prompt an LM to produce a free-form response to a given prompt [5,21,26,73,80,144], and it is often unclear how to assess the quality of the output. Previous studies have measured the lexical or semantic similarity between the predicted free-form response and the reference answer to quantify the quality of the output [33,34,76,89,95,139,141]. However, the critical drawback is that it fails to identify false negatives, where the output is satisfactory but different from the reference answer [18,29,40,105].…”

Section: Related Workmentioning

confidence: 99%

An Outbreak of Fungal Endophthalmitis After Cataract Surgery in South Korea

Kim

Choi

et al. 2023

JAMA Ophthalmol

View full text Add to dashboard Cite

ImportanceFungal endophthalmitis caused by contaminated medical products is extremely rare; it follows an intractable clinical course with a poor visual prognosis.ObjectiveTo report the epidemiologic and clinical features and treatment outcomes of a nationwide fungal endophthalmitis outbreak after cataract surgery as a result of contaminated viscoelastic agents in South Korea.Design, Setting, and ParticipantsThis was a retrospective case series analysis of clinical data from multiple institutions in South Korea conducted from September 1, 2020, to October 31, 2021. Data were collected through nationwide surveys in May and October 2021 from the 100 members of the Korean Retinal Society. Patients were diagnosed with fungal endophthalmitis resulting from the use of the viscoelastic material sodium hyaluronate (Unial [Unimed Pharmaceutical Inc]). Data were analyzed from November 1, 2021, to May 30, 2022.Main Outcomes and MeasuresThe clinical features and causative species were identified, and treatment outcomes were analyzed for patients who underwent 6 months of follow-up.ResultsThe fungal endophthalmitis outbreak developed between September 1, 2020, and June 30, 2021, and peaked in November 2020. An official investigation by the Korea Disease Control and Prevention Agency confirmed contamination of viscoelastic material. All 281 eyes of 265 patients (mean [SD] age, 65.4 [10.8] years; 153 female individuals [57.7%]) were diagnosed with fungal endophthalmitis, based on clinical examinations and supportive culture results. The mean (SD) time period between cataract surgery and diagnosis was 24.7 (17.3) days. Patients exhibited characteristic clinical features of fungal endophthalmitis, including vitreous opacity (212 of 281 [75.4%]), infiltration into the intraocular lens (143 of 281 [50.9%]), and ciliary infiltration (55 of 281 [19.6%]). Cultures were performed in 260 eyes, and fungal presence was confirmed in 103 eyes (39.6%). Among them, Fusarium species were identified in 89 eyes (86.4%). Among the 228 eyes included in the treatment outcome analysis, the mean (SD) best-corrected visual acuity improved from 0.78 (0.74) logMAR (Snellen equivalent, 20/120 [7.3 lines]) to 0.36 (0.49) logMAR (Snellen equivalent, 20/45 [4.9 lines]) at 6 months. Furthermore, disease remission with no signs of fungal endophthalmitis (or cells in the anterior chamber milder than grade 1) was noted in 214 eyes (93.9%).Conclusions and RelevanceThis was a retrospectively reviewed case series of a fungal endophthalmitis outbreak resulting from contaminated viscoelastic material. Findings of this case series study support the potential benefit of prompt, aggressive surgical intervention that may reduce treatment burden and improve prognosis of fungal endophthalmitis caused by contaminated medical products.

show abstract