LineCap: Line Charts for Data Visualization Captioning Models

The State of the Art in Creating Visualization Corpora for Automated Chart Analysis

“…We went through the collected papers, unified their vocabularies about their research goals and tasks, and identified 6 categories for the goal dimension: Create a chart corpus , which aims at introducing a benchmark chart collection for certain chart types or analysis tasks. For example, the Beagle corpus [BDM * 18] consists of SVG visualizations collected from five popular charting tools on the web; the LineCap corpus [MKT22] curated line charts with figure captioning; and the MapQA corpus [CPL * 22] introduced a question answering annotations specifically for choropleth maps. Extract chart semantics , where “semantics” refers to information spanning a range of concepts, including low‐level primitives like the mark type [LWL21] and attributes [PH17], the role of a mark group (e.g., axis, glyph), and high‐level meta‐information such as chart type [JKS * 17] or the underlying dataset [MTW * 18]. The extracted semantics are useful for various downstream applications. Modify an existing chart , which transforms a chart for new contexts or needs.…”

“…One advantage of bitmap charts is that they are naturally compatible with modern convolutional neural networks (CNNs). Figure 2 shows NN is the most frequently used method on bitmap‐based corpora, which indicates that the bitmap graphics format is usually the first choice for many end‐to‐end neural‐network‐based systems [TLL * 16, CAM * 18, LWL21, MBT * 22, MKT22, LWW * 22, FWD * 19, HWWL21, MTW * 18, CPL * 22, CZK * 19, RSE * 21]. In these cases, usually some additional preprocessing steps, such as image cropping & resizing [CJP * 19] and data augmentation [KM18, ZFF20], are needed.…”

“…, and (3) relation‐related [CPL * 22], such as what are the highest and lowest values?. A similar annotation type is Question‐Caption (QC) pair, observed in chart captioning systems [CZK * 19, MKT22]. Both of them are only considered in the task of generate natural language descriptions , where answers to questions or captions to charts are outputs of the type synthesized descriptions shown in Section 3.…”

“… Crowdsourcing : an online annotation process where workers from crowdsourcing platforms such as Amazon's Mechanical Turk are recruited to annotate charts. This method usually is considered when the size of a chart corpus is large.The two considerations described for in‐house labeling also apply here: Crowdsourcing often involves a GUI for labeling and a training session to teach online workers the annotation tasks (e.g., Saleh et al [SDHL15] included three examples in the introduction session to teach the workers the purpose of the experiment as well as the meaning of the annotation type ‐ stylistic infographics similarity) and identify qualified workers [MKT22]. Besides, a formal post‐examination is usually carried out by the organizers to remove invalid annotations, because the quality of annotations from online workers varies even if a training session is included.…”

SciCapenter: Supporting Caption Composition for Scientific Figures with Machine-Generated Captions and Ratings

“…Unlike previous works that used synthetic data [4,21,22], SciCap facilitated the development of AI models capable of handling realworld scientific figures. This includes models for generating captions based on figure images [15,16,32,36,47,51] or paper content [15,19,32,51]. There are also models that evaluate the quality of given captions [18].…”