SpaceDrones 2.0—Hardware-in-the-Loop Simulation and Validation for Orbital and Deep Space Computer Vision and Machine Learning Tasking Using Free-Flying Drone Platforms

Abstract

The proliferation of reusable space vehicles has fundamentally changed how assets are injected into the low earth orbit and beyond, increasing both the reliability and frequency of launches. Consequently, it has led to the rapid development and adoption of new technologies in the aerospace sector, including computer vision (CV), machine learning (ML)/artificial intelligence (AI), and distributed networking. All these technologies are necessary to enable truly autonomous “Human-out-of-the-loop” mission tasking for spaceborne applications as spacecrafts travel further into the solar system and our missions become more ambitious. This paper proposes a novel approach for space-based computer vision sensing and machine learning simulation and validation using synthetically trained models to generate the large amounts of space-based imagery needed to train computer vision models. We also introduce a method of image data augmentation known as domain randomization to enhance machine learning performance in the dynamic domain of spaceborne computer vision to tackle unique space-based challenges such as orientation and lighting variations. These synthetically trained computer vision models then apply that capability for hardware-in-the-loop testing and evaluation via free-flying robotic platforms, thus enabling sensor-based orbital vehicle control, onboard decision making, and mobile manipulation similar to air-bearing table methods. Given the current energy constraints of space vehicles using solar-based power plants, cameras provide an energy-efficient means of situational awareness when compared to active sensing instruments. When coupled with computationally efficient machine learning algorithms and methods, it can enable space systems proficient in classifying, tracking, capturing, and ultimately manipulating objects for orbital/planetary assembly and maintenance (tasks commonly referred to as In-Space Assembly and On-Orbit Servicing). Given the inherent dangers of manned spaceflight/extravehicular activities (EVAs) currently employed to perform spacecraft maintenance and the current limitation of long-duration human spaceflight outside the low earth orbit, space robotics armed with generalized sensing and control and machine learning architecture have a unique automation potential. However, the tools and methodologies required for hardware-in-the-loop simulation, testing, and validation at a large scale and at an affordable price point are in developmental stages. By leveraging a drone’s free-flight maneuvering capability, theater projection technology, synthetically generated orbital and celestial environments, and machine learning, this work strives to build a robust hardware-in-the-loop testing suite. While the focus of the specific computer vision models in this paper is narrowed down to solving visual sensing problems in orbit, this work can very well be extended to solve any problem set that requires a robust onboard computer vision, robotic manipulation, and free-flight capabilities.

Image Quality Enhancement with Applications to Unmanned Aerial Vehicle Obstacle Detection

“…The core work of this paper is to improve the brightness of low-light images. Therefore, it is necessary to first establish a mathematical model for describing scene brightness, as shown in Equation (1). For a single image, the relationship between the image irradiance I and the pixel value V of each pixel can be expressed by a nonlinear function f , which is the camera response function (CRF).…”

“…Machine vision, as an intelligent technology, has been widely researched and succeeded in the fields of space robots [1], unmanned vehicles [2], micro air vehicles [3], and fixed-wing aircraft automatic landing [4].…”

“…Although the aforementioned algorithms have achieved varying degrees of success on open datasets such as LIME [20] and ExDark [25], their limitations are as follows. (1) The above algorithms rely on iterative optimization in calculating the illumination map, which is time-consuming and therefore cannot satisfy the requirements of real-time processing for UAV obstacle avoidance. (2) Deep learning-based algorithms need to rely on a high volume of paired low/normal brightness image datasets for network training.…”

A Flexible Dynamic Reliability Simulation Approach for Predicting the Lifetime Consumption of Extravehicular Spacesuits during Uncertain Extravehicular Activities

“…Almost every part of the extravehicular spacesuit is of the highest technology and is costly to build. Once the astronauts have left the spacecraft, the only protection measure is the extravehicular spacesuit [2,3]. Spacesuits working in low-orbit environments are subject to degradation from dust, radiation, and other factors [4,5].…”