Finding Regions of Interest from Multimodal Human-Robot Interactions

Abstract

Learning new concepts, such as object models, from humanrobot interactions entails different recognition capabilities on a robotic platform. This work proposes a hierarchical approach to address the extra challenges from natural interaction scenarios by exploiting multimodal data. First, a speech-guided recognition of the type of interaction happening is presented. This first step facilitates the following segmentation of relevant visual information to learn the target object model. Our approach includes three complementary strategies to find Regions of Interest (RoI) depending on the interaction type: Point, Show or Speak. We run an exhaustive validation of the proposed strategies using the recently published Multimodal Human-Robot Interaction dataset [1]. The currently presented pipeline is built on the pipeline proposed with the dataset and provides a more complete baseline for target object segmentation on all its recordings.

Incremental Learning of Object Models From Natural Human–Robot Interactions

“…• Speak: the user describes where a certain object is in relation to other objects. This paper builds on our previous works in [1] and [2]. The new contributions here are:…”

“…[59], show that the combination of language and vision leads to substantial improvements. In our previous work [60] we demonstrated that including Speak interactions to train the models obtains better accuracy than using only Point and Show ones.…”

Implementasi Sistem Kasir Digital Berbasis Teknologi Deteksi Tangan Computer Vision dan OpenCV

“…Oleh karena itu, Dalam penelitian ini, kami merancang sebuah antarmuka virtual untuk mesin kasir digital di Café Lentera Coffee & Eatery menggunakan teknologi Computer Vision dan Convolutional Neural Network (CNN) dengan tujuan meningkatkan pelayanan konsumen menggunakan teknologi kecerdasan buatan [6]. Selain itu, tujuan utama dari penelitian ini adalah untuk mengembangkan teknologi kecerdasan buatan dan juga akan di implementasikan pada antarmuka virtual tersebut untuk mengenali wajah konsumen, mengenali permintaan yang dipesan dan memproses permintaan pelanggan dengan lebih cepat dan efisien [7]. Sistem ini akan menjadi antarmuka virtual antara pelanggan dan mesin kasir, memungkinkan pelanggan untuk memesan secara real-time dan juga bisa membayar tanpa harus P-ISSN: 2089-676X E-ISSN: 2549-0796 990 berinteraksi langsung dengan pelayan dan crew [8] .…”