PAWLS: PDF Annotation With Labels and Structure

Neumann, Mark E; Shen, Zejiang; Skjonsberg, Sam

doi:10.18653/v1/2021.acl-demo.31

Cited by 5 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…S2-VL is manually labeled by graduate students who frequently read scientific papers. Using the PAWLS annotation tool (Neumann et al, 2021), annotators draw rectangular text blocks directly on each PDF page, and specify the block-level semantic categories from 15 possible candidates. 7 Tokens within a group can therefore inherit the category from the parent text block.…”

Section: S2-vlmentioning

confidence: 99%

VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups

Shen

Wang

et al. 2022

Transactions of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Accurately extracting structured content from PDFs is a critical first step for NLP over scientific papers. Recent work has improved extraction accuracy by incorporating elementary layout information, for example, each token’s 2D position on the page, into language model pretraining. We introduce new methods that explicitly model VIsual LAyout (VILA) groups, that is, text lines or text blocks, to further improve performance. In our I-VILA approach, we show that simply inserting special tokens denoting layout group boundaries into model inputs can lead to a 1.9% Macro F1 improvement in token classification. In the H-VILA approach, we show that hierarchical encoding of layout-groups can result in up to 47% inference time reduction with less than 0.8% Macro F1 loss. Unlike prior layout-aware approaches, our methods do not require expensive additional pretraining, only fine-tuning, which we show can reduce training cost by up to 95%. Experiments are conducted on a newly curated evaluation suite, S2-VLUE, that unifies existing automatically labeled datasets and includes a new dataset of manual annotations covering diverse papers from 19 scientific disciplines. Pre-trained weights, benchmark datasets, and source code are available at https://github.com/allenai/VILA.

show abstract

Section: S2-vlmentioning

confidence: 99%

VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups

Shen

Wang

et al. 2022

Transactions of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…We have addressed these layout-centric pinch points in the OKN construction process in two ways. First, we created PAWLS (PDF Annotation with Labels and Structure), a new annotation tool designed for PDF documents (Neumann, Shen, and Skjonsberg 2021). PAWLS supports labeling span-based textual regions, free form visual bounding boxes, and easy authoring of n-ary relations among different visual elements (see Figure 1).…”

Section: Layout-aware Document Processingmentioning

confidence: 99%

Infrastructure for rapid open knowledge network development

et al. 2022

View full text Add to dashboard Cite

The past decade has witnessed a growth in the use of knowledge graph technologies for advanced data search, data integration, and query-answering applications. The leading example of a public, general-purpose open knowledge network (aka knowledge graph) is Wikidata, which has demonstrated remarkable advances in quality and coverage over this time. Proprietary knowledge graphs drive some of the leading applications of the day including, for example, Google Search, Alexa, Siri, and Cortana. Open Knowledge Networks are exciting: they promise the power of structured database-like queries with the potential for the wide coverage that is today only provided by the Web. With the current state of the art, building, using, and scaling large knowledge networks can still be frustratingly slow. This article describes a National Science Foundation Convergence Accelerator project to build a set of Knowledge Network Programming Infrastructure systems to address this issue.

show abstract

TEIMMA: The First Content Reuse Annotator for Text, Images, and Math

Satpute,

Greiner-Petter,

Schubotz

et al. 2023

2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL)

View full text Add to dashboard Cite

Large Language Models (LLMs) have demonstrated exceptional capabilities in various natural language tasks, often achieving performances that surpass those of humans. Despite these advancements, the domain of mathematics presents a distinctive challenge, primarily due to its specialized structure and the precision it demands. In this study, we adopted a two-step approach for investigating the proficiency of LLMs in answering mathematical questions. First, we employ the most effective LLMs, as identified by their performance on math question-answer benchmarks, to generate answers to 78 questions from the Math Stack Exchange (MSE). Second, a case analysis is conducted on the LLM that showed the highest performance, focusing on the quality and accuracy of its answers through manual evaluation. We found that GPT-4 performs best (nDCG of 0.48 and P@10 of 0.37) amongst existing LLMs finetuned for answering mathematics questions and outperforms the current best approach on ArqMATH3 Task1, considering P@10. Our Case analysis indicates that while the GPT-4 can generate relevant responses in certain instances, it does not consistently answer all questions accurately. This paper explores the current limitations of LLMs in navigating complex mathematical problemsolving. Through case analysis, we shed light on the gaps in LLM capabilities within mathematics, thereby setting the stage for future research and advancements in AI-driven mathematical reasoning. We make our code and findings publicly available for research: https://github.com/gipplab/LLM-Investig-MathStackExchange CCS CONCEPTS• Computing methodologies → Discourse, dialogue and pragmatics; • Mathematics of computing → Mathematical software performance.

show abstract

PAWLS: PDF Annotation With Labels and Structure

Cited by 5 publications

References 17 publications

VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups

VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups

Infrastructure for rapid open knowledge network development

TEIMMA: The First Content Reuse Annotator for Text, Images, and Math

Contact Info

Product

Resources

About