A Real-time Drone Surveillance System Using Pixel-level Short-time Fourier Transform

Wide-area Operation Monitoring of Conveyors Using a Panoramic Vibration Camera

“…Considering that each pixel of the image sensor is an optical sensor, several studies on vibration imaging based on signal processing for time-series brightness signals independently at all pixels have been recently reported. Its effectiveness was verified by structural vibration analysis 24,27) with vibration distribution measurement at the audio-frequency level, rotation measurement for motors and propellers, [28][29][30] and wing-flapping measurement of honeybees. 31,32) It is expected to be a novel wide-area vibration monitoring technique in which every pixel plays the role of a sensor for time-series signal processing in high-speed vision.…”

High-frame-rate Video-based Multicopter Tracking System Using Pixel-level Short-time Fourier Transform

“…They can identify the pixels around the propellers without considering the appearance of the drone, although they cannot measure the propellers' timevarying rotation speed, because their frequency ranges were designed based on the premise of a fixed target frequency. To track a multicopter with propellers rotating at unknown and time-varying speeds, the TFR of brightness signals at all pixels in a 500-fps-video were utilized for peak frequency detection [19]; they were computed in real time by parallel-implementing pixel-level STFTs with a small number of frames such as 16 frames on a GPU. The pixellevel accuracy and robustness in real-time drone tracking were demonstrated under brightness changes, defocus blur, and complex backgrounds, and the tracking drift during the observation of a fast-flying multicopter was discussed; the tracking drift became larger as the number of frames used in the STFT computation increased.…”

“…The pixellevel accuracy and robustness in real-time drone tracking were demonstrated under brightness changes, defocus blur, and complex backgrounds, and the tracking drift during the observation of a fast-flying multicopter was discussed; the tracking drift became larger as the number of frames used in the STFT computation increased. In [19], the small number of frames was set so as not to degrade the tracking performance; the frequency resolution in the TFRs was too low for the peak frequency detection to monitor the rotation speed of the propellers. If the rotation speed of the propellers could be precisely measured in an HFR video, we could identify different types of multicopters, whose propellers rotate at different speeds depending on the model.…”

“…In this study, we propose an HFR video technique that can monitor the rotation speeds of the propellers of a multicopter flying in a wide area, as well as mechanically track it at the center of the camera view by inspecting the brightness signals blinking at hundreds of hertz around their propellers with two types of short-time Fourier transforms (STFTs) in an HFR video. This is an extension of the drone detection algorithm in [19], where vibration-based drone tracking was performed with one type of pixel-level STFTs to localize a flying multicopter in images without precisely monitoring the rotation speeds of their propellers. One type of STFT involves a small number of frames and is used to track a multicopter through visual feedback as in [19]; the other involves a large number of frames and is used for precisely obtaining the rotation speed of the propellers, thereby monitoring the flight status of the multicopter.…”

“…This is an extension of the drone detection algorithm in [19], where vibration-based drone tracking was performed with one type of pixel-level STFTs to localize a flying multicopter in images without precisely monitoring the rotation speeds of their propellers. One type of STFT involves a small number of frames and is used to track a multicopter through visual feedback as in [19]; the other involves a large number of frames and is used for precisely obtaining the rotation speed of the propellers, thereby monitoring the flight status of the multicopter. We developed a mirror-drive pan-tilt active vision system that can track a flying multicopter by processing 500 fps video of 720×540 pixels accelerated by a graphic processing unit (GPU).…”