TogetherNet: Bridging Image Restoration and Object Detection Together via Dynamic Enhancement Learning

Abstract

Adverse weather conditions such as haze, rain, and snow often impair the quality of captured images, causing detection networks trained on normal images to generalize poorly in these scenarios. In this paper, we raise an intriguing question -if the combination of image restoration and object detection, can boost the performance of cutting-edge detectors in adverse weather conditions. To answer it, we propose an effective yet unified detection paradigm that bridges these two subtasks together via dynamic enhancement learning to discern objects in adverse weather conditions, called TogetherNet. Different from existing efforts that intuitively apply image dehazing/deraining as a pre-processing step, TogetherNet considers a multi-task joint learning problem. Following the joint learning scheme, clean features produced by the restoration network can be shared to learn better object detection in the detection network, thus helping TogetherNet enhance the detection capacity in adverse weather conditions. Besides the joint learning architecture, we design a new Dynamic Transformer Feature Enhancement module to improve the feature extraction and representation capabilities of TogetherNet. Extensive experiments on both synthetic and real-world datasets demonstrate that our TogetherNet outperforms the state-of-the-art detection approaches by a large margin both quantitatively and qualitatively. Source code is available at https://github.com/yz-wang/TogetherNet.

Multi-Task Learning for UAV Aerial Object Detection in Foggy Weather Condition

“…We also see that ODFC-YOLO surpasses all multi-task learning approaches. Although TogetherNet [51] obtains competitive results in two out of five categories, our method still outperforms it and achieves a 2% improvement in mAP. This proves that enhancing the feature consistency of different tasks is more conducive to the improvement of detection performance.…”

“…Multi-task learning methods [50][51][52] aimed to learn multiple related tasks simultaneously, benefiting other tasks through the knowledge gained from one task. One popular method is the Task Relation Network [52] (TRN), which built explicit relationship between tasks from a statistical perspective.…”

“…The comparison results of our ODFC-YOLO with YOLOXs [12], GCA-YOLOX [56], IA-YOLO [16], MS-DAYOLO [19] and TogetherNet [51] on the RainCitySpaces dataset are presented in Table 5 and our method achieves the highest mAP. The second column of Table 5 lists the detection results without retraining on the dataset.…”

IDP-YOLOV9: Improvement of Object Detection Model in Severe Weather Scenarios from Drone Perspective

“…However, these approaches often struggle to adapt to various image types, and the adjustment of enhancement parameters heavily relies on manual intervention, leading to suboptimal results. Building on these foundations, some research has integrated image enhancement with object detection [23,24], proposing the Adaptive Enhancement Model for Object Detection Network (ARODNet) [25] to improve detection performance under adverse conditions [26,27]. Nonetheless, these methods do not optimize the structure of the detection network itself, limiting their effectiveness during training.…”