Attention-Enhanced Multimodal Learning for Conceptual Design Evaluations

Abstract

Conceptual design evaluation is an indispensable component of innovation in the early stage of engineering design. Properly assessing the effectiveness of conceptual design requires a rigorous evaluation of the outputs. Traditional methods to evaluate conceptual designs are slow, expensive, and difficult to scale because they rely on human expert input. An alternative approach is using computational methods to evaluate design concepts. However, most existing methods have limited utility because they are constrained to unimodal design representations (e.g., texts or sketches). To overcome these limitations, we propose an attention-enhanced multimodal learning-based model to predict five design metrics: drawing quality, uniqueness, elegance, usefulness, and creativity. The proposed model utilizes knowledge from large external datasets through transfer learning, simultaneously processes text and sketch data from early-phase concepts, and effectively fuses the multimodal information through a mutual cross-attention mechanism. To study the efficacy of multimodal learning (MML) and attention-based information fusion, we compare (1) a baseline MML model and the unimodal models and (2) the attention-enhanced models with baseline models in terms of their explanatory power for the variability of the design metrics. The results show that MML improves model explanatory power by 5%-12% and the mutual cross-attention mechanism further increases the explanatory power by 5%-9%, leading to a highest explanatory power of 44% for drawing quality, 60% for uniqueness, 45% for elegance, 43% for usefulness, and 32% for creativity. Our findings highlight the benefit of using multimodal representations for design metric assessment.

Sketch2Prototype: rapid conceptual design exploration and prototyping with generative AI

“…Sketching provides a rapid external representation that comes at very little cognitive cost (Goldschmidt, 2014). Researchers also explored creativity and decision making with sketches (Toh and Miller, 2016), using sketching for finite element analysis (Murugappan et al, 2017), and using machine learning to predict creativity-ratings from sketches and text (Edwards et al, 2022;Song et al, 2023).…”

Drug-Target-Interaction Prediction with Contrastive and Siamese Transformers

“…To ensure precise alignment of multiple representations with analytical precision, we employ Cross Attention, a widely acknowledged technique renowned for its effectiveness in multi-representation alignment. [5,6,7,8,9]. Importantly, the inclusion of the Attention Mechanism within our framework not only serves functional purposes but also augments interpretability [10,11].…”