Ubiq-Genie: Leveraging External Frameworks for Enhanced Social VR Experiences

Numan, Nels; Giunchi, Daniele; Congdon, Benjamin; Steed, Anthony

doi:10.1109/vrw58643.2023.00108

Cited by 7 publications

(8 citation statements)

References 22 publications

(25 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, integrating voice recognition APIs into virtual environments has become more feasible [41]. This study explored the potential of incorporating voice recognition APIs into virtual settings.…”

Section: B Application Of Artificial Intelligence Technologymentioning

confidence: 99%

Interactive Design With Gesture and Voice Recognition in Virtual Teaching Environments

Fang,

Wang

2024

IEEE Access

View full text Add to dashboard Cite

In virtual teaching scenarios, head-mounted display (HMD) interactions often employ traditional controller and UI interactions, which are not very conducive to teaching scenarios that require hand training. Existing improvements in this area have primarily focused on replacing controllers with gesture recognition. However, the exclusive use of gesture recognition may have limitations in certain scenarios, such as complex operations or multitasking environments. This study designed and tested an interaction method that combines simple gestures with voice assistance, aiming to offer a more intuitive user experience and enrich related research. A speech classification model was developed that can be activated via a fistclenching gesture and is capable of recognising specific Chinese voice commands to initiate various UI interfaces, further controlled by pointing gestures. Virtual scenarios were constructed using Unity, with hand tracking achieved through the HTC OpenXR SDK. Within Unity, hand rendering and gesture recognition were facilitated, and interaction with the UI was made possible using the Unity XR Interaction Toolkit. The interaction method was detailed and exemplified using a teacher training simulation system, including sample code provision. Following this, an empirical test involving 20 participants was conducted, comparing the gesture-plus-voice operation to the traditional controller operation, both quantitatively and qualitatively. The data suggests that while there is no significant difference in task completion time between the two methods, the combined gesture and voice method received positive feedback in terms of user experience, indicating a promising direction for such interactive methods. Future work could involve adding more gestures and expanding the model training dataset to realize additional interactive functions, meeting diverse virtual teaching needs.

show abstract

Section: B Application Of Artificial Intelligence Technologymentioning

confidence: 99%

Interactive Design With Gesture and Voice Recognition in Virtual Teaching Environments

Fang,

Wang

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…The system produces a composite output that includes story content, narration, and visual representation. Its framework, based on the Ubiq [5,9] platform, is highly modular, enabling seamless integration of various third-party models including Text-To-Speech (TTS), image, and speech generation. This flexibility allows for the creation of tailored storytelling experiences.…”

Section: Introductionmentioning

confidence: 99%

AIsop: Exploring Immersive VR Storytelling Leveraging Generative AI

Gatti,

Giunchi,

Numan

et al. 2024

2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)

Self Cite

View full text Add to dashboard Cite

We introduce AIsop, a system that autonomously generates VR storytelling experiences using generative artificial intelligence (AI). AIsop crafts unique stories by leveraging state-of-the-art Large Language Models (LLMs) and employs Text-To-Speech (TTS) technology for narration. Further enriching the experience, a visual representation of the narrative is produced through a pipeline that pairs LLM-generated prompts with diffusion models, rendering visuals for clusters of sentences in the story. Our evaluation encompasses two distinct use cases: the narration of pre-existing content and the generation of entirely new narratives. AIsop highlights the myriad research prospects spanning its technical architecture and user engagement.

show abstract

“…The interplay between Virtual Reality (VR) and Artificial Intelligence (AI) presents a unique conjuncture, providing a platform for innovative immersive experiences [59,5,62]. AI has been seamlessly woven into the fabric of VR applications, utilizing its generative ability to create captivating multimedia content, such as images, audio, text, and 3D meshes [52,61,4,57]. AI has significantly influenced VR, driving the development of visually and auditorily impressive environments and characters through algorithms that manipulate static content generation [15,28,16,36].…”

Section: Introductionmentioning

confidence: 99%

“…While several works have been proposed to change behavior at runtime in VR [64,63,75], they tend to offer limited functionality and applicability, such as replacing a single object property or shape. We introduce DreamCodeVR (Figure ), a VR system based on Unity and Ubiq [26,57] that bridges the gap between static content creation and dynamic behavior generation in VR environments. Using LLMs [14,66], our approach leverages the generation of procedural code, aligning with the principles of low-code programming [35].…”

Section: Introductionmentioning

confidence: 99%

DreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming

Giunchi,

Numan,

Gatti

et al. 2024

2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)

Self Cite

View full text Add to dashboard Cite

Virtual Reality (VR) has revolutionized how we interact with digital worlds. However, programming for VR remains a complex and challenging task, requiring specialized skills and knowledge. Powered by large language models (LLMs), DreamCodeVR is designed to assist users, irrespective of their coding skills, in crafting basic object behavior in VR environments by translating spoken language into code within an active application. This approach seeks to simplify the process of defining behaviors visual changes through speech. Our preliminary user study indicated that the system's speech interface supports elementary programming tasks, highlighting its potential to improve accessibility for users with varying technical skills. However, it also uncovered a wide range of challenges and opportunities. In an extensive discussion, we detail the system's strengths, weaknesses, and areas for future research.

show abstract

Ubiq-Genie: Leveraging External Frameworks for Enhanced Social VR Experiences

Cited by 7 publications

References 22 publications

Interactive Design With Gesture and Voice Recognition in Virtual Teaching Environments

Interactive Design With Gesture and Voice Recognition in Virtual Teaching Environments

AIsop: Exploring Immersive VR Storytelling Leveraging Generative AI

DreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming

Contact Info

Product

Resources

About