Multi-Task Deep Learning for Depth-based Person Perception in Mobile Robotics

Seichter, Daniel; Lewandowski, Benjamin; Hochemer, Dominik; Wengefeld, Tim; Groß, Horst–Michael

doi:10.1109/iros45743.2020.9340870

Cited by 11 publications

(3 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Each configuration may yield disparate outcomes, underscoring the necessity to rigorously evaluate typical multi-task balancing techniques within each specific setup, encompassing the tasks involved, dataset characteristics, and model complexity ( Standley et al, 2019 ; Zamir et al, 2020 ). As highlighted by Seichter et al (2020) in their work on multi-task learning for person detection, posture classification, and orientation estimation, balancing the tasks becomes harder the more heterogeneous the tasks are, e.g., mixing both regression and classification tasks. Various works underscore the importance of physical testing, as relying solely on previous works may overlook unique aspects of the current configuration, thereby potentially leading to inaccurate or sub-optimal results ( Kendall et al, 2017 ; Liu et al, 2019 ).…”

Section: Related Workmentioning

confidence: 99%

Efficient human 3D localization and free space segmentation for human-aware mobile robots in warehouse facilities

Arapis,

Jami,

Nalpantidis

2023

Front. Robot. AI

View full text Add to dashboard Cite

Real-time prediction of human location combined with the capability to perceive obstacles is crucial for socially-aware navigation in robotics. Our work focuses on localizing humans in the world and predicting the free space around them by incorporating other static and dynamic obstacles. We propose a multi-task learning strategy to handle both tasks, achieving this goal with minimal computational demands. We use a dataset captured in a typical warehouse environment by mounting a perception module consisting of a Jetson Xavier AGX and an Intel L515 LiDAR camera on a MiR100 mobile robot. Our method, which is built upon prior works in the field of human detection and localization demonstrates improved results in difficult cases that are not tackled in other works, such as human instances at a close distance or at the limits of the field of view of the capturing sensor. We further extend this work by using a lightweight network structure and integrating a free space segmentation branch that can independently segment the floor space without any prior maps or 3D data, relying instead on the characteristics of the floor. In conclusion, our method presents a lightweight and efficient solution for predicting human 3D location and segmenting the floor space for low-energy consumption platforms, tested in an industrial environment.

show abstract

Section: Related Workmentioning

confidence: 99%

Efficient human 3D localization and free space segmentation for human-aware mobile robots in warehouse facilities

Arapis,

Jami,

Nalpantidis

2023

Front. Robot. AI

View full text Add to dashboard Cite

show abstract

“…For simplicity, in our formulation we do not consider the scene to be dynamic and we assume that targets' states are provided with accuracy to our controller by means of existing perception algorithms [16], [17]. Nevertheless, the experiments in Section V show that CineMPC can work transparently with continuous scene changes and handles robustly the noise associated to real perception achieving good results.…”

Section: B Scenementioning

confidence: 99%

CineMPC: Controlling Camera Intrinsics and Extrinsics for Autonomous Cinematography

Pueyo¹,

Montijano²,

Murillo³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present CineMPC, an algorithm to autonomously control a UAV-borne video camera in a nonlinear MPC loop. CineMPC controls both the position and orientation of the camera-the camera extrinsics-as well as the lens focal length, focal distance, and aperture-the camera intrinsics. While some existing solutions autonomously control the position and orientation of the camera, no existing solutions also control the intrinsic parameters, which are essential tools for rich cinematographic expression. The intrinsic parameters control the parts of the scene that are focused or blurred, and the viewers' perception of depth in the scene. Cinematographers commonly use the camera intrinsics to direct the viewers' attention through the use of focus, to convey suspense through telephoto views, inspire awe through wide-angle views, and generally to convey an emotionally rich viewing experience. Our algorithm can use any existing approach to detect the subjects in the scene, and tracks those subjects throughout a user-specified desired camera trajectory that includes camera intrinsics. CineMPC closes the loop from camera images to UAV trajectory in order to follow the desired relative trajectory as the subjects move through the scene. The cinematographer can use CineMPC to autonomously record scenes using the full array of cinematographic tools for artistic expression.

show abstract

“…The segmentation output enriches the robot's visual perception and facilitates further processing steps by providing individual semantic masks. For our person perception [7], computations can be restricted to image regions segmented as person, instead of processing the entire image. Furthermore, the floor class indicates free space that can be used for inpainting invalid depth pixels as well as serves as additional information for avoiding even small Fig.…”

Section: Introductionmentioning

confidence: 99%

Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis

Seichter¹,

Köhler²,

Lewandowski³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Analyzing scenes thoroughly is crucial for mobile robots acting in different environments. Semantic segmentation can enhance various subsequent tasks, such as (semantically assisted) person perception, (semantic) free space detection, (semantic) mapping, and (semantic) navigation. In this paper, we propose an efficient and robust RGB-D segmentation approach that can be optimized to a high degree using NVIDIA TensorRT and, thus, is well suited as a common initial processing step in a complex system for scene analysis on mobile robots. We show that RGB-D segmentation is superior to processing RGB images solely and that it can still be performed in real time if the network architecture is carefully designed. We evaluate our proposed Efficient Scene Analysis Network (ESANet) on the common indoor datasets NYUv2 and SUNRGB-D and show that it reaches state-of-the-art performance when considering both segmentation performance and runtime. Furthermore, our evaluation on the outdoor dataset Cityscapes shows that our approach is suitable for other areas of application as well. Finally, instead of presenting benchmark results only, we show qualitative results in one of our indoor application scenarios.

show abstract

Multi-Task Deep Learning for Depth-based Person Perception in Mobile Robotics

Cited by 11 publications

References 30 publications

Efficient human 3D localization and free space segmentation for human-aware mobile robots in warehouse facilities

Efficient human 3D localization and free space segmentation for human-aware mobile robots in warehouse facilities

CineMPC: Controlling Camera Intrinsics and Extrinsics for Autonomous Cinematography

Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis

Contact Info

Product

Resources

About