Control of Smart Home Operations Using Natural Language Processing, Voice Recognition and IoT Technologies in a Multi-Tier Architecture

Abstract

The Internet of Things (IoT) is an emerging Internet-based architecture, enabling the exchange of data and services in a global network. With the advent of the Internet of Things, more and more devices are connecting to the Internet in order to help people get and share data or program actions. In this paper, we introduce an IoT Agent, a Web application for monitoring and controlling a smart home remotely. The IoT Agent integrates a chat bot that can understand text or voice commands using natural language processing (NLP). With the use of NLP, home devices are more user-friendly and controlling them is easier, since even when a command or question/command is different from the presets, the system understands the user’s wishes and responds accordingly. Our solution exploits several available Application Programming Interfaces (APIs), namely: the Dialogflow API for the efficient integration of NLP to our IoT system, the Web Speech API for enriching user experience with voice recognition and synthesis features, MQTT (Message Queuing Telemetry Transport) for the lightweight control of actuators and Firebase for dynamic data storage. This is the most significant innovation it brings: the integration of several third-party APIs and open source technologies into one mash-up, highlighting how a new IoT application can be built today using a multi-tier architecture. We believe that such a tiered architecture can be very useful for the rapid development of smart home applications.

A Smart Home System based on Internet of Things

“…However, three features that were considered advantages on other smart systems have not been included in this proposal, so they were not added to Table I. The first feature is the multi-tier architecture suggested by [25] as it is not practical to increase the number of the system servers when all the web services are developed in-house, and as one of the characteristics of the smart home system, as mentioned in the introduction section, is that data management occurs through a local server, thus the use of third-party online services provided through an API was avoided in the suggested architecture. The second feature is the hierarchal mobile software agent distributed system supported by [5], [31], and [32], because creating an agent for each sensor or actuator device is not practical economically, takes up considerable space in the home, and presents a low level of security, as a mobile agent third party application can be untrusted and it may harm the system.…”

“…The MQTT broker can provide services, such as monitoring and controlling room temperatures, suppressing fires, and controlling alarms, all of which require microcontrollers to be used as the IoT end devices that connect sensors and actuators to the smart home system through a Wi-Fi channel [29]. If the user requests data from sensors or actuators, then the smart home system sends a message to the appropriate microcontroller through the MQTT broker to perform the user request [25]. The MQTT and broker acts as a simple, common interface to which everything can connect.…”

3D Point-of-Intention Determination Using a Multimodal Fusion of Hand Pointing and Eye Gaze for a 3D Display

“…As methods available for farther distance, the speech approach seems to work similarly to natural human-to-human commands and it may be preferred by users. For instance, speech activation for the control of home’s equipment [ 11 , 12 ], speech-based control for music and mobile phone infotainment [ 13 ], etc. However, it is not robust against noise, including signal noise and the conversations of other users.…”

Spoken notifications in smart environments using Croatian language

“…As a result, there are many examples of employing speech interfaces in smart home systems, industrial facilities, smart vehicles, public services, etc. Regardless of the operational domain, spoken communication between human users and computer systems needs to address specific features related to the wider context of the target environment [1]. This includes events and conditions in the environment, current and future activities related to users, and the linguistic context when interaction is in process.…”