Image dataset creation and networks improvement method based on CAD model and edge operator for object detection in the manufacturing industry

Tang, Pengzhou; Guo, Yu; Han, Li; Wei, Zhen; Zheng, Guanguan; Pu, Juan

doi:10.1007/s00138-021-01237-y

Cited by 9 publications

(7 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Improvement in disambiguation and REC performance can be achieved by comparing and exploring different and more sophisticated disambiguation approaches, such as attribute-guided disambiguation [48], to improve the accuracy of grounding as well as by incorporating gesture [49] and gaze [50] information. While the adapted model comprehends the natural language object descriptions with 82% accuracy in the user study, the domain gap between the synthetic and real-world application can further be reduced by incorporating more variation and randomization [29] in the RefMD dataset.…”

Section: Discussion and Future Workmentioning

confidence: 99%

“…To perform manufacturing domain adaptation of the underlying pre-trained REC model we propose a two-stage process, stage-1) domain adaptation of VB for manufacturing object classification, and stage-2) domain adaptation of the complete REC system for manufacturing object localization. The manufacturing domain often involves complexities such as clutter, spatial relationships between objects, orientations, and interactions within a confined space [29]. Therefore, before performing domain adaptation and as an important contribution to this work, we create a manufacturing domain-specific dataset.…”

Section: Methodsmentioning

confidence: 99%

“…Given the comprehensiveness of the baseline dataset encompassing simple and complex referring expressions, it becomes the base for the construction of RefMD. For our use case, we recreate object arrangements from the base dataset in Unity and replace the cubes with manufacturing objects' models [29] as shown in Figure 2. We adopted a color-based rule to replace the cubes in the base dataset images with various mechanical objects.…”

Section: Dataset For Rec Adaptation To Manufacturingmentioning

confidence: 99%

See 2 more Smart Citations

Manufacturing domain instruction comprehension using synthetic data

Johari,

Tong,

Bhardwaj

et al. 2024

Vis Comput

View full text Add to dashboard Cite

Referring Expression Comprehension (REC) system solves a task to localize objects in a given image, based on natural language expression. We propose a novel approach to adapting the pre-trained REC model for the manufacturing domain.Introduction: Despite significant advances in REC research, current REC datasets fail to recognize objects from specific yet important domains such as manufacturing due to the absence of domain-specific samples during training. Thus, we introduce a synthetic data-based domain adaptation approach for REC. Methods:To adapt a REC model to the manufacturing domain, we generated a synthetic REC dataset RefMD that consists of two sub-datasets: 1) dataset for manufacturing object classification, and 2) dataset for REC adaptation to manufacturing. Each dataset serves as one step toward the REC adaptation. Adaptation of the object classification network (visual backbone) is carried out by training ResNet50 on domain-specific labeled data, while the REC adaptation completes with the adaptation of modules in RealGIN altogether. The manufacturing domain-adapted model is further enhanced with the capability to handle ambiguous referring expressions through human-in-the-loop (HITL) interaction. Results:The experiments on 3D printed manufacturing objects demonstrate that the interactive REC model can accurately comprehend human instructions with an 82% accuracy. Conclusion:This paper introduces an approach to facilitate domain adaptation using solely synthetic data for the specific case of the manufacturing domain. However, the proposed adaptation methodology can be applied to any other domain by following the same synthetic data generation process.

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Dataset For Rec Adaptation To Manufacturingmentioning

confidence: 99%

See 1 more Smart Citation

Manufacturing domain instruction comprehension using synthetic data

Johari,

Tong,

Bhardwaj

et al. 2024

Vis Comput

View full text Add to dashboard Cite

show abstract

“…Compared with the hand-designed feature-based methods, deep neural networks can automatically learn features in images. and are more robust to background, angle of view, and lighting conditions [20]. The deep learning-based object detection network structure are mainly classified into CNN and transformer.…”

Section: ) Deep Learning Based Methods For General Object Detectionmentioning

confidence: 99%

Online Assembly Inspection Integrating Lightweight Hybrid Neural Network With Positioning Box Matching

Zhao,

Wang,

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Assembly inspection methods have been widely used in the process of mechanical product assembly for quality issues. However, some challenges remain to be solved, such as low detection efficiency, poor accuracy and sensitive to camera view. This paper proposes an online assembly inspection scheme based on hybrid neural network and positioning box matching. A hybrid multi-task learning neural network with transformer attention mechanism is proposed to simultaneously detect key points and assembly parts with high accuracy and strong robustness. Utilizing the key points detection results, the transformation relationships between the in-site assembly images and the standard templates are solved. According to the results of assembly parts detection, the detected 2D positioning bounding boxes are matched with those in the standard assembly templates, so as to evaluate whether the current step has quality problems. In addition, the proposed method is tested on an assembly dataset constructed in this paper. For key points detection, the average error is less than 1 pixel. For parts detection, the mean average precision is 97.66%. The missing and wrong assembly inspection results show that the average F1 score reaches 93.96%. This inspection method can be employed to detect the missing and wrong assembly faults of each assembly step online, improving the assembly quality of products.

show abstract

“…Object detection algorithms were widely used in agriculture [19], medical treatment [20], industry [21], transportation [22] and other fields. In [23], a multi-target matching tracking method based on YOLO is proposed, which can be used to detect the passengers flow on and off buses.…”

Section: Introductionmentioning

confidence: 99%

Object Detection Method for Grasping Robot Based on Improved YOLOv5

Song

Bai

et al. 2021

Micromachines

View full text Add to dashboard Cite

In the industrial field, the anthropomorphism of grasping robots is the trend of future development, however, the basic vision technology adopted by the grasping robot at this stage has problems such as inaccurate positioning and low recognition efficiency. Based on this practical problem, in order to achieve more accurate positioning and recognition of objects, an object detection method for grasping robot based on improved YOLOv5 was proposed in this paper. Firstly, the robot object detection platform was designed, and the wooden block image data set is being proposed. Secondly, the Eye-In-Hand calibration method was used to obtain the relative three-dimensional pose of the object. Then the network pruning method was used to optimize the YOLOv5 model from the two dimensions of network depth and network width. Finally, the hyper parameter optimization was carried out. The simulation results show that the improved YOLOv5 network proposed in this paper has better object detection performance. The specific performance is that the recognition precision, recall, mAP value and F1 score are 99.35%, 99.38%, 99.43% and 99.41% respectively. Compared with the original YOLOv5s, YOLOv5m and YOLOv5l models, the mAP of the YOLOv5_ours model has increased by 1.12%, 1.2% and 1.27%, respectively, and the scale of the model has been reduced by 10.71%, 70.93% and 86.84%, respectively. The object detection experiment has verified the feasibility of the method proposed in this paper.

show abstract

Image dataset creation and networks improvement method based on CAD model and edge operator for object detection in the manufacturing industry

Cited by 9 publications

References 40 publications

Manufacturing domain instruction comprehension using synthetic data

Manufacturing domain instruction comprehension using synthetic data

Online Assembly Inspection Integrating Lightweight Hybrid Neural Network With Positioning Box Matching

Object Detection Method for Grasping Robot Based on Improved YOLOv5

Contact Info

Product

Resources

About