High Performance Time-of-Flight and Color Sensor Fusion with Image-Guided Depth Super Resolution

Plank, Hannes; Holweg, Gerald; Herndl, Thomas; Druml, Norbert

doi:10.3850/9783981537079_0446

Cited by 10 publications

(13 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, this is only feasible for static scenes and moving sensors. For dynamic scenes, the prevalent approach is to use the corresponding RGB image as guide for depth upscaling [30,31,32,33,34,35,36].…”

Section: Depth Upscalingmentioning

confidence: 99%

“…However, such global optimization methods are difficult to parallelize [37], and tend to have prohibitive computational costs: [30] reports 318.2 ms to upscale the depth from 160 × 120 px to 810 × 610, and [31,33] report up to 572.38 s for depth upscaling to 960 × 540 px. Plank et al [32] bypass the optimization performance issue by treating upscaling as a weighted interpolation of valid depths within a nearby pixel patch. Edges from color image are used as preemptive interrupts during the marching within the pixel patch.…”

Section: Depth Upscalingmentioning

confidence: 99%

“…Edges from color image are used as preemptive interrupts during the marching within the pixel patch. Such upscaling can be effectively parallelized, and [32] report as little as 100 ms on a mobile GPU and 8 ms on a desktop GPU. However, [32] upscales a 288 × 256 px depth image to 640 × 480 px, so the relative resolution increase is low and interpolation kernels can be small.…”

Section: Depth Upscalingmentioning

confidence: 99%

“…Such upscaling can be effectively parallelized, and [32] report as little as 100 ms on a mobile GPU and 8 ms on a desktop GPU. However, [32] upscales a 288 × 256 px depth image to 640 × 480 px, so the relative resolution increase is low and interpolation kernels can be small. Furthermore, a specific RGB-D camera is used where the depth and color sensors are less than 1 cm apart, and reprojection artefacts prior to upscaling are negligible and can be ignored.…”

Section: Depth Upscalingmentioning

confidence: 99%

See 3 more Smart Citations

Camera and Lidar-Based View Generation for Augmented Remote Operation in Mining Applications

Dima

Sjöström

2021

IEEE Access

View full text Add to dashboard Cite

Remote operation of diggers, scalers, and other tunnel-boring machines has significant benefits for worker safety in underground mining. Real-time augmentation of the presented remote views can further improve the operator effectiveness through a more complete presentation of relevant sections of the remote location. In safety-critical applications, such augmentation cannot depend on preconditioned data, nor generate plausible-looking yet inaccurate sections of the view. In this paper, we present a capture and rendering pipeline for real time view augmentation and novel view synthesis that depends only on the inbound data from lidar and camera sensors. We suggest an on-the-fly lidar filtering for reducing point oscillation at no performance cost, and a full rendering process based on lidar depth upscaling and in-view occluder removal from the presented scene. Performance assessments show that the proposed solution is feasible for real-time applications, where per-frame processing fits within the constraints set by the inbound sensor data and within framerate tolerances for enabling effective remote operation.

show abstract

Section: Depth Upscalingmentioning

confidence: 99%

Section: Depth Upscalingmentioning

confidence: 99%

Section: Depth Upscalingmentioning

confidence: 99%

Section: Depth Upscalingmentioning

confidence: 99%

See 2 more Smart Citations

Camera and Lidar-Based View Generation for Augmented Remote Operation in Mining Applications

Dima

Sjöström

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…In addition, depth sensor-equipped devices, such as Google Tango, etc. [27] are also commercially available. In accordance with this trend, we believe that such advances in technologies enable our virtual keyboard to be applied to any flat surface.…”

Section: A Feasibility Of Hand Pose Tracking Systemmentioning

confidence: 99%

Designing Hand Pose Aware Virtual Keyboard With Hand Drift Tolerance

et al. 2019

View full text Add to dashboard Cite

An unintentional hand drift adversely affects the typing performance of conventional virtual keyboards. To overcome this, we proposed to utilize the typing patterns of skilled typists. First, as most typists enter the keys in the same column with a predetermined finger only, we restricted these keys to be typed by their corresponding fingers. Second, our investigation of skilled typists demonstrated that hand poses vary when the typists touch different keys. Thus, rather than locating the touch point as in the case of existing virtual keyboards, we attempted to use unique hand poses to infer the target key. Based on these two techniques, we implemented a novel hand poses aware virtual keyboard that is tolerant of hand drift. Our experimental studies yielded the following results: 1) most of the QWERTY-familiar typists who have varying typing habits were easily adaptable to the proposed keyboard design and 2) the proposed keyboard outperformed existing virtual keyboards in terms of typing speed and several error rates, and eventually achieved a typing speed of approximately 56 WPM.INDEX TERMS Eyes-free, hand drift, hands, text entry, touch typing, virtual keyboard, virtual reality.

show abstract