Integrating Behavior of Children with Profound Intellectual, Multiple, or Severe Motor Disabilities With Location and Environment Data Sensors for Independent Communication and Mobility: App Development and Pilot Testing

Abstract

Background Children with profound intellectual and multiple disabilities (PIMD) or severe motor and intellectual disabilities (SMID) only communicate through movements, vocalizations, body postures, muscle tensions, or facial expressions on a pre- or protosymbolic level. Yet, to the best of our knowledge, there are few systems developed to specifically aid in categorizing and interpreting behaviors of children with PIMD or SMID to facilitate independent communication and mobility. Further, environmental data such as weather variables were found to have associations with human affects and behaviors among typically developing children; however, studies involving children with neurological functioning impairments that affect communication or those who have physical and/or motor disabilities are unexpectedly scarce. Objective This paper describes the design and development of the ChildSIDE app, which collects and transmits data associated with children’s behaviors, and linked location and environment information collected from data sources (GPS, iBeacon device, ALPS Sensor, and OpenWeatherMap application programming interface [API]) to the database. The aims of this study were to measure and compare the server/API performance of the app in detecting and transmitting environment data from the data sources to the database, and to categorize the movements associated with each behavior data as the basis for future development and analyses. Methods This study utilized a cross-sectional observational design by performing multiple single-subject face-to-face and video-recorded sessions among purposively sampled child-caregiver dyads (children diagnosed with PIMD/SMID, or severe or profound intellectual disability and their primary caregivers) from September 2019 to February 2020. To measure the server/API performance of the app in detecting and transmitting data from data sources to the database, frequency distribution and percentages of 31 location and environment data parameters were computed and compared. To categorize which body parts or movements were involved in each behavior, the interrater agreement κ statistic was used. Results The study comprised 150 sessions involving 20 child-caregiver dyads. The app collected 371 individual behavior data, 327 of which had associated location and environment data from data collection sources. The analyses revealed that ChildSIDE had a server/API performance >93% in detecting and transmitting outdoor location (GPS) and environment data (ALPS sensors, OpenWeatherMap API), whereas the performance with iBeacon data was lower (82.3%). Behaviors were manifested mainly through hand (22.8%) and body movements (27.7%), and vocalizations (21.6%). Conclusions The ChildSIDE app is an effective tool in collecting the behavior data of children with PIMD/SMID. The app showed high server/API performance in detecting outdoor location and environment data from sensors and an online API to the database with a performance rate above 93%. The results of the analysis and categorization of behaviors suggest a need for a system that uses motion capture and trajectory analyses for developing machine- or deep-learning algorithms to predict the needs of children with PIMD/SMID in the future.

Machine learning-based classification of the movements of children with profound or severe intellectual or multiple disabilities using environment data features

“…Their severe condition presents significant limitations in communication due to the inability to comprehend social cues, spoken, verbal or symbolic language [ 1 – 4 ]. These are often manifested in atypical, minute, and refined body and hand postures and movements, vocalizations and utterances, muscle tensions, or facial expressions on a presymbolic (nonsymbolic) or protosymbolic level with no shared meaning and stereotypical non-communicative behaviors [ 1 – 3 , 7 – 9 ]. Such extensive communication ambiguities impede expressing their needs, desires, thoughts, and feelings requiring great dependence on their close caregivers who are more capable of discerning and interpreting the most idiosyncratic behaviors than other people, thus limiting their communication group [ 2 , 7 , 10 ].…”

“…This entails two major steps, the first of which is the collection, extraction, and initial categorization of the movements of the children based on the interpretations of their caregivers. This was initially performed using a previously developed mobile app called ChildSIDE that enabled the collection of 327 behavior data from 20 children-caregiver dyads [ 9 ]. Video-based retrospective inter-rater agreement Kappa analyses revealed that most of the movements of children with PIMD/IDs were manifested mainly through body (27.7%; approaching, contacting, and movement of part of the body) and hand movements (22.8%; pointing, reaching, and moving).…”

“…ML-based behavior studies have sourced and analyzed behavior data from phenotypic, electrodermal activity (EDA), or virtual reality (VR) systems, wearable sensors, standardized scales and instruments or task or performance-based measures [ 15 , 17 – 19 , 21 , 23 – 25 ]. Mobile platforms and apps have also been validated using ML classifiers as a screening method, utilized to embed ML-based scoring models, and to deliver or collect the behavioral assessment, screening tools or data for ML-based analyses [ 9 , 29 – 32 ]. Collection of fine-grained and extensive records of behavioral data and expression across situations such as daily activities, social interactions and communication, mobility patterns, bodily functions, and biological markers, location (e.g.…”

“…Collection of fine-grained and extensive records of behavioral data and expression across situations such as daily activities, social interactions and communication, mobility patterns, bodily functions, and biological markers, location (e.g. GPS map coordinates and iBeacon or BLE indoor), timestamps and logs have become possible using mobile devices [ 9 , 33 – 39 ]. Equipped with Wi-Fi, Bluetooth, near-field communication (NFC), and sensors harnessed by apps, smartphones have become a primary tool in extracting useful features with ML algorithms [ 40 – 44 ].…”

“…Our previous [ 9 , 39 ] and current work are aligned to the aim of supporting children with communication, physical or other impairments, specifically the underrepresented population of children with PIMD/IDs, to have independent communication and mobility. For such purpose, we introduce a behavior inference computer-based communication-aid AT system to interpret the movements of children with PIMD/IDs.…”