Visual Information Requirements for Remotely Supervised Autonomous Agricultural Machines

Abstract

A study to determine the visual requirements for a remote supervisor of an autonomous sprayer was conducted. Observation of a sprayer operator identified 9 distinct “look zones” that occupied his visual attention, with 39% of his time spent viewing the look zone ahead of the sprayer. While observation of the sprayer operator was being completed, additional GoPro cameras were used to record video of the sprayer in operation from 10 distinct perspectives (some look zones were visible from the operator’s seat, but other look zones were selected to display other regions of the sprayer that might be of interest to a sprayer operator). In a subsequent laboratory study, 29 experienced sprayer operators were recruited to view and comment on video clips selected from the video footage collected during the initial ride-along. Only the two views from the perspective of the operator’s seat were rated highly as providing important information even though participants were able to identify relevant information from all ten of the video clips. Generally, participants used the video clips to obtain information about the boom status, the location and movement of the sprayer within the field, the weather conditions (especially the wind), obstacles to be avoided, crop conditions, and field conditions. Sprayer operators with more than 15 years of experience provided more insightful descriptions of the video clips than their less experienced peers. Designers can influence which features the user will perceive by positioning the camera such that those specific features are prominent in the camera’s field of view. Overall, experienced sprayer operators preferred the concept of presenting visual information on an automation interface using live video rather than presenting that same information using some type of graphical display using icons or symbols.

Real-Time Sensory Information for Remote Supervision of Autonomous Agricultural Machines

“…Camera angle and position influenced what information the operators perceived from the video clips [34]. Participants tended to describe features that were more prominent within the frame of the camera in comparison to less-prominent features.…”

“…Although these studies identified the different regions of importance, they did not describe what information was gained by viewing those regions. Hence, a study was conducted to identify what visual information about the machine and its environment would assist the remote supervisor to make decisions [34]; the primary focus of this study was the high-clearance sprayer. GoPro cameras were mounted at difference locations on a sprayer to record the sprayer and its environment while in operation (Figure 1).…”

“…A study was conducted to assess which of the seven modalities (visual, auditory, tactile, audio-visual, audio-tactile, visual-tactile, visual-auditory-tactile) would be the most effective in providing feedback to the remote supervisor of an autonomous sprayer [52]. They modified an autonomous agricultural machine control interface (AAMCI) simulator that was designed by [29] to include the different warning methods.…”

“…The experimental setup is shown in Figure 9. Further details of the experimental procedure can be found in [52]. They noticed that all seven warning modalities were able to accurately warn the participants of the errors, but varied in their effectiveness (i.e., response time).…”

“…The text-based notifications are enhanced by a color-coded status indicator -green, yellow, and red -to indicate the corresponding severity. Three video feeds were included in the automation interface following the recommendations made by [63]. The videos provide visual and auditory information, although at a lower fidelity than experienced when inside a tractor.…”