Taxonomy of Mathematical Plagiarism

Satpute, Ankit; Greiner-Petter, André; Gießing, Noah; Beckenbach, Isabel; Schubotz, Moritz; Teschke, Olaf; Aizawa, Akiko; Gipp, Bela

doi:10.1007/978-3-031-56066-8_2

Cited by 1 publication

(1 citation statement)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given that mathematical content is prevalent in science, technology, engineering, and mathematics (STEM), it is imperative to evaluate the capacity of LLMs to engage with mathematical language. The challenges of LLMs with mathematics are manifold; mathematics, with its rigorous logic and abstract concepts, is communicated through a specialized language that mixes symbols and syntax in complex arrangements [24]. Unlike natural language, mathematical expressions rely on unmentioned rules and assumptions, demanding explicit knowledge and a level of precision.…”

Section: Introductionmentioning

confidence: 99%

TEIMMA: The First Content Reuse Annotator for Text, Images, and Math

Satpute,

Greiner-Petter,

Schubotz

et al. 2023

2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL)

View full text Add to dashboard Cite

Large Language Models (LLMs) have demonstrated exceptional capabilities in various natural language tasks, often achieving performances that surpass those of humans. Despite these advancements, the domain of mathematics presents a distinctive challenge, primarily due to its specialized structure and the precision it demands. In this study, we adopted a two-step approach for investigating the proficiency of LLMs in answering mathematical questions. First, we employ the most effective LLMs, as identified by their performance on math question-answer benchmarks, to generate answers to 78 questions from the Math Stack Exchange (MSE). Second, a case analysis is conducted on the LLM that showed the highest performance, focusing on the quality and accuracy of its answers through manual evaluation. We found that GPT-4 performs best (nDCG of 0.48 and P@10 of 0.37) amongst existing LLMs finetuned for answering mathematics questions and outperforms the current best approach on ArqMATH3 Task1, considering P@10. Our Case analysis indicates that while the GPT-4 can generate relevant responses in certain instances, it does not consistently answer all questions accurately. This paper explores the current limitations of LLMs in navigating complex mathematical problemsolving. Through case analysis, we shed light on the gaps in LLM capabilities within mathematics, thereby setting the stage for future research and advancements in AI-driven mathematical reasoning. We make our code and findings publicly available for research: https://github.com/gipplab/LLM-Investig-MathStackExchange CCS CONCEPTS• Computing methodologies → Discourse, dialogue and pragmatics; • Mathematics of computing → Mathematical software performance.

show abstract

Section: Introductionmentioning

confidence: 99%