The Why and The How: A Survey on Natural Language Interaction in Visualization

Voigt, Henrik; Alaçam, Özge; Meuschke, Monique; Lawonn, Kai; Zarrieß, Sina

doi:10.18653/v1/2022.naacl-main.27

Cited by 9 publications

(13 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, the idea of using NL as a way to create visualisations has gained significant attention within the field of data visualisation [19,9]. The use of NLIs has also grown in popularity in commercial software as a means of improving the usability of visualisation systems.…”

Section: Related Workmentioning

confidence: 99%

“…Meanwhile, rule-based approaches rely on predefined rules created by experts for understanding NL. They are more accurate and reliable but, also less flexible in handling complex queries [19]. Probabilistic grammar-based approaches use formal grammar rules and probability distributions over the possible parses of a given input.…”

Section: Symbolic Nlp Approachesmentioning

confidence: 99%

“…These approaches combine the processes of language understanding, reasoning and chart generation in a single system and are closer to the proposed system in this work. These approaches aim to achieve greater robustness, flexibility and adaptability compared to traditional methods [19].…”

Section: Deep-learning Model Approachesmentioning

confidence: 99%

See 2 more Smart Citations

Chat2VIS: Generating Data Visualisations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Models

Maddigan¹,

Sušnjak²

2023

Preprint

View full text Add to dashboard Cite

The field of data visualisation has long aimed to devise solutions for generating visualisations directly from natural language text. Research in Natural Language Interfaces (NLIs) has contributed towards the development of such techniques. However, the implementation of workable NLIs has always been challenging due to the inherent ambiguity of natural language, as well as in consequence of unclear and poorly written user queries which pose problems for existing language models in discerning user intent. Instead of pursuing the usual path of developing new iterations of language models, this study uniquely proposes leveraging the advancements in pre-trained large language models (LLMs) such as ChatGPT and GPT-3 to convert free-form natural language directly into code for appropriate visualisations. This paper presents a novel system, Chat2VIS, which takes advantage of the capabilities of LLMs and demonstrates how, with effective prompt engineering, the complex problem of language understanding can be solved more efficiently, resulting in simpler and more accurate end-to-end solutions than prior approaches. Chat2VIS shows that LLMs together with the proposed prompts offer a reliable approach to rendering visualisations from natural language queries, even when queries are highly misspecified and underspecified. This solution also presents a significant reduction in costs for the development of NLI systems, while attaining greater visualisation inference abilities compared to traditional NLP approaches that use hand-crafted grammar rules and tailored models. This study also presents how LLM prompts can be constructed in a way that preserves data security and privacy while being generalisable to different datasets. This work compares the performance of GPT-3, Codex and ChatGPT across a number of case studies and contrasts the performances with prior studies.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Symbolic Nlp Approachesmentioning

confidence: 99%

See 1 more Smart Citation

Chat2VIS: Generating Data Visualisations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Models

Maddigan¹,

Sušnjak²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Moreover, in VA systems such as Emblaze [SWP22] and the tool from Heimerl et al [HG18], users are able to customize color assignments to be able to compare different embedding spaces or data co‐occurrences in the same view. There are also a few techniques augmenting the scatter plot by constructing a pattern graph and Voronoi maps for each cluster sub‐space via Delaunay triangulation [LDL*20, VMZL22]. AnchorViz chooses to use non‐orthogonal layouts to project the embedding space [SGR*20], and the VA technique from Heimerl et al allows users to define the axis of the 2D projection [HG18].…”

Section: Categorization Of Va + Embedding Approachesmentioning

confidence: 99%

“…Visualization for NLP and CL Creating and using embeddings in NLP and CL is crucial for representing and capturing the context and content of words, phrases, sentences, and documents. VA + embedding techniques in this set focus on four themes: exploring the semantics and contextualization of embedding spaces [CTL18,LBT * 18,EAKC * 20,MWZ19,SSKEA21, GHM21, BN21, BCS22,VMZL22,LWZ * 23,MM23], active learning and interpretation for language models [LCSEK19, TWB * 20, SH20, ARCL21, LXW * 21, SKB * 21, SCR * 23], data‐driven information retrieval [CWDH09,BMS17,ZSHL18,KOK * 18,DMdO19, RSBV21, PdSP * 22, JWC * 23], and annotation tools [SJB * 17, BNL * 18,PKL * 18,MWJ22].…”

Section: Categorization Of Va + Embedding Approachesmentioning

confidence: 99%

VA + Embeddings STAR: A State‐of‐the‐Art Report on the Use of Embeddings in Visual Analytics

Huang

Witschard

Kucher

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

Over the past years, an increasing number of publications in information visualization, especially within the field of visual analytics, have mentioned the term “embedding” when describing the computational approach. Within this context, embeddings are usually (relatively) low‐dimensional, distributed representations of various data types (such as texts or graphs), and since they have proven to be extremely useful for a variety of data analysis tasks across various disciplines and fields, they have become widely used. Existing visualization approaches aim to either support exploration and interpretation of the embedding space through visual representation and interaction, or aim to use embeddings as part of the computational pipeline for addressing downstream analytical tasks. To the best of our knowledge, this is the first survey that takes a detailed look at embedding methods through the lens of visual analytics, and the purpose of our survey article is to provide a systematic overview of the state of the art within the emerging field of embedding visualization. We design a categorization scheme for our approach, analyze the current research frontier based on peer‐reviewed publications, and discuss existing trends, challenges, and potential research directions for using embeddings in the context of visual analytics. Furthermore, we provide an interactive survey browser for the collected and categorized survey data, which currently includes 122 entries that appeared between 2007 and 2023.

show abstract

Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models

Maddigan

Sušnjak

2023

IEEE Access

View full text Add to dashboard Cite

The field of data visualisation has long aimed to devise solutions for generating visualisations directly from natural language text. Research in Natural Language Interfaces (NLIs) has contributed towards the development of such techniques. However, the implementation of workable NLIs has always been challenging due to the inherent ambiguity of natural language, as well as in consequence of unclear and poorly written user queries which pose problems for existing language models in discerning user intent. Instead of pursuing the usual path of developing new iterations of language models, this study uniquely proposes leveraging the advancements in pre-trained large language models (LLMs) such as ChatGPT and GPT-3 to convert free-form natural language directly into code for appropriate visualisations. This paper presents a novel system, Chat2VIS, which takes advantage of the capabilities of LLMs and demonstrates how, with effective prompt engineering, the complex problem of language understanding can be solved more efficiently, resulting in simpler and more accurate end-to-end solutions than prior approaches. Chat2VIS shows that LLMs together with the proposed prompts offer a reliable approach to rendering visualisations from natural language queries, even when queries are highly misspecified and underspecified. This solution also presents a significant reduction in costs for the development of NLI systems, while attaining greater visualisation inference abilities compared to traditional NLP approaches that use hand-crafted grammar rules and tailored models. This study also presents how LLM prompts can be constructed in a way that preserves data security and privacy while being generalisable to different datasets. This work compares the performance of GPT-3, Codex and ChatGPT across several case studies and contrasts the performances with prior studies.INDEX TERMS ChatGPT, codex, end-to-end visualisations from natural language, GPT-3, large language models, natural language interfaces, text-to-visualisation.

show abstract

The Why and The How: A Survey on Natural Language Interaction in Visualization

Cited by 9 publications

References 28 publications

Chat2VIS: Generating Data Visualisations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Models

Chat2VIS: Generating Data Visualisations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Models

VA + Embeddings STAR: A State‐of‐the‐Art Report on the Use of Embeddings in Visual Analytics

Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models

Contact Info

Product

Resources

About