Visualizing Natural Language Descriptions

Hassani, Kaveh; Lee, Won-Sook

doi:10.1145/2932710

Cited by 39 publications

(38 citation statements)

References 73 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…According to [24], in addition to the technical issues, difficulties and fundamental challenges in developing a text-to-picture system are rooted in natural language characteristics such as language being semi-structured, ambiguous, context-sensitive, and subjective. This work highlighted that designing a text-to-picture system capable of integrating a natural language interface and an interface for visualization purposes requires overcoming profound technical challenges in integrating artificial intelligence techniques, including natural language understanding (NLU), knowledge representation, planning, reasoning, and the integration of multiple knowledge sources and computer graphics techniques such as real-time rendering.…”

Section: Requirements and Challengesmentioning

confidence: 99%

“…This task presents a fundamental challenge; for more details, a brief overview of NLU issues can be found in [41]. Furthermore, language rarely mentions common-sense facts about a world that contains critically important spatial knowledge and negation [24]. Enabling a machine to understand natural language, which is variable, ambiguous, and imprecise, also involves feeding the machine grammatical structures (e.g., parts of speech), semantic relationships (e.g., emotional value and intensity), and visual descriptions (e.g., colors and motion direction) in order for it to match the language with suitable graphics [23].…”

Section: Natural Language Understandingmentioning

confidence: 99%

“…A reasoning mechanism embedded within the knowledge representation component can also help the system to derive implicit knowledge from available knowledge. Designing such a component is not a trivial task [24]. In addition, gathering the required tools and repositories, modeling, and obtaining the knowledge base (KB) is a challenge.…”

Section: Knowledge Representation Reasoning and Implicit Knowledgementioning

confidence: 99%

“…According to [24], despite all these features of text-to-picture systems, they still have many limitations in terms of performing their assigned tasks. The authors pointed out that there is a possible way to improve the visualization to be more dynamic.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Text-to-picture tools, systems, and approaches: a survey

Zakraoui

Saleh

Jaam

2019

Multimed Tools Appl

View full text Add to dashboard Cite

Text-to-picture systems attempt to facilitate high-level, user-friendly communication between humans and computers while promoting understanding of natural language. These systems interpret a natural language text and transform it into a visual format as pictures or images that are either static or dynamic. In this paper, we aim to identify current difficulties and the main problems faced by prior systems, and in particular, we seek to investigate the feasibility of automatic visualization of Arabic story text through multimedia. Hence, we analyzed a number of well-known text-to-picture systems, tools, and approaches. We showed their constituent steps, such as knowledge extraction, mapping, and image layout, as well as their performance and limitations. We also compared these systems based on a set of criteria, mainly natural language processing, natural language understanding, and input/output modalities. Our survey showed that currently emerging techniques in natural language processing tools and computer vision have made promising advances in analyzing general text and understanding images and videos. Furthermore, important remarks and findings have been deduced from these prior works, which would help in developing an effective text-to-picture system for learning and educational purposes.

show abstract

Section: Requirements and Challengesmentioning

confidence: 99%

Section: Natural Language Understandingmentioning

confidence: 99%

Section: Knowledge Representation Reasoning and Implicit Knowledgementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Text-to-picture tools, systems, and approaches: a survey

Zakraoui

Saleh

Jaam

2019

Multimed Tools Appl

View full text Add to dashboard Cite

show abstract

“…In recent years, there has been extensive work in the field of deep learning to combine natural language with other modalities like images [34,70] and videos [13,69]. A recent survey on proposed solutions to the problem of visualizing natural language descriptions is given in [26].…”

Section: Related Workmentioning

confidence: 99%

Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks

Plappert¹,

Mandery²,

Asfour³

2018

Robotics and Autonomous Systems

View full text Add to dashboard Cite

Linking human whole-body motion and natural language is of great interest for the generation of semantic representations of observed human behaviors as well as for the generation of robot behaviors based on natural language input. While there has been a large body of research in this area, most approaches that exist today require a symbolic representation of motions (e.g. in the form of motion primitives), which have to be defined a-priori or require complex segmentation algorithms. In contrast, recent advances in the field of neural networks and especially deep learning have demonstrated that sub-symbolic representations that can be learned end-to-end usually outperform more traditional approaches, for applications such as machine translation. In this paper we propose a generative model that learns a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks (RNNs) and sequence-to-sequence learning. Our approach does not require any segmentation or manual feature engineering and learns a distributed representation, which is shared for all motions and descriptions. We evaluate our approach on 2 846 human whole-body motions and 6 187 natural language descriptions thereof from the KIT Motion-Language Dataset. Our results clearly demonstrate the effectiveness of the proposed model: We show that our model generates a wide variety of realistic motions only from descriptions thereof in form of a single sentence. Conversely, our model is also capable of generating correct and detailed natural language descriptions from human motions.

show abstract

Text-to-Movie Authoring of Anatomy Lessons

Murukutla

Cattan

Palombi

et al. 2019

Artificial Intelligence in Medicine

View full text Add to dashboard Cite

With popular use of multimedia and 3D content in anatomy teaching there is a need for a simple yet comprehensive tool to create and edit pedagogical anatomy video lessons. In this paper we present an automated video authoring tool created for teachers. It takes text written in a novel domain specific language (DSL) called the Anatomy Storyboard Language (ASL) as input and translates it to real time 3D animation. Preliminary results demonstrates the ease of use and effectiveness of the tool for quickly drafting video lessons in realistic medical anatomy teaching scenarios.

show abstract

Visualizing Natural Language Descriptions

Cited by 39 publications

References 73 publications

Text-to-picture tools, systems, and approaches: a survey

Text-to-picture tools, systems, and approaches: a survey

Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks

Text-to-Movie Authoring of Anatomy Lessons

Contact Info

Product

Resources

About