Human-Oriented IoT-Based Interfaces for Multimodal Visual Analytics Systems

Abstract

This paper describes an approach to use modern programmable microelectronics and ontology engineering to create custom hardware human-machine interfaces for solving particular visual analytics tasks. The idea of these special interfaces is to involve additional modalities like motor or haptic into the analytics process to improve its quality and speed. We propose using tangible user interfaces built upon the Internet of Things technologies to present the visual analytics system as a cyber-physical one, melting together the real and virtual environments. To automate the creation of such an alloy we suggest a unified approach of composing the firmware for the interface hardware device and corresponding drivers for the computer; of calibration the device' sensors; of testing and debugging the communication between the interface device and the application it is supposed to steer; and of solving visual analytics tasks using the device created. All the mentioned steps are supported by the highlevel built-in mechanisms of SciVi visual analytics platform we created during the previous research and improved in the current work. Data flow diagrams are used to visually describe the data preprocessing and rendering, as well as the ways hardware interface affects them. Ontology engineering is used to ensure flexibility and extensibility of the entire platform combined with the semantic power of its individual blocks: the behavior of SciVi is fully governed by underlaying ontologies, which describe supported data formats and types, filters, rendering mechanisms, supported electronic components of the hardware interfaces, the ways to program them and to communicate with them. The proposed methods and means are tested by solving real-world visual analytics task of automated identification of the relationship between native speakers' psychological characteristics and their verbal behavior.

Visualization of energy efficiency of using a material in a deformable solid

“…1. Cyber-physical multiplatform systems based on IoT technologies that provide tools for high-level adaptation to the data source from the solver (processor) [1,2]; 2. Web services that provide post-processing for the finite element method.…”