Gallery D.C.: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery

Chen, Chunyang; Feng, Sidong; Xing, Zhenchang; Liu, Linda; Zhao, Shengdong; Wang, Jinshui

doi:10.1145/3359282

Cited by 62 publications

(38 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Anchor-box, two stage [9,40] • Must define anchor boxes • Pipeline region detection and region classification • Gallery D.C. [9] is the only work that tests the Faster RCNN on large-scale real GUIs, but it uses default settings A CNN classifier for region classification, trained jointly with region proposal network…”

Section: Deep Learningmentioning

confidence: 99%

Object detection for graphical user interface: old fashioned or deep learning or a combination?

Chen

Xie

Xing

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

Self Cite

109

View full text Add to dashboard Cite

Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation. Existing studies for GUI element detection directly borrow the mature methods from computer vision (CV) domain, including old fashioned ones that rely on traditional image processing features (e.g., canny edge, contours), and deep learning models that learn to detect from large-scale GUI data. Unfortunately, these CV methods are not originally designed with the awareness of the unique characteristics of GUIs and GUI elements and the high localization accuracy of the GUI element detection task. We conduct the first large-scale empirical study of seven representative GUI element detection methods on over 50k GUI images to understand the capabilities, limitations and effective designs of these methods. This study not only sheds the light on the technical challenges to be addressed but also informs the design of new GUI element detection methods. We accordingly design a new GUI-specific oldfashioned method for non-text GUI element detection which adopts a novel top-down coarse-to-fine strategy, and incorporate it with the mature deep learning model for GUI text detection. Our evaluation on 25,000 GUI images shows that our method significantly advances the start-of-the-art performance in GUI element detection.

show abstract

Section: Deep Learningmentioning

confidence: 99%

Object detection for graphical user interface: old fashioned or deep learning or a combination?

Chen

Xie

Xing

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

Self Cite

109

View full text Add to dashboard Cite

show abstract

“…UI Element Kit: UI kit is the most efficient and profitable way to build a rapid GUI [6]. Therefore, we store all the detected elements in the dataset for the user to reuse.…”

Section: Web Implementationmentioning

confidence: 99%

UIED: a hybrid tool for GUI element detection

Xie

Feng

Xing

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

Self Cite

View full text Add to dashboard Cite

Graphical User Interface (GUI) elements detection is critical for many GUI automation and GUI testing tasks. Acquiring the accurate positions and classes of GUI elements is also the very first step to conduct GUI reverse engineering or perform GUI testing. In this paper, we implement a User Iterface Element Detection (UIED), a toolkit designed to provide user with a simple and easy-to-use platform to achieve accurate GUI element detection. UIED integrates multiple detection methods including old-fashioned computer vision (CV) approaches and deep learning models to handle diverse and complicated GUI images. Besides, it equips with a novel customized GUI element detection methods to produce state-of-the-art detection results. Our tool enables the user to change and edit the detection result in an interactive dashboard. Finally, it exports the detected UI elements in the GUI image to design files that can be further edited in popular UI design tools such as Sketch and Photoshop. UIED is evaluated to be capable of accurate detection and useful for downstream works.

show abstract

“…Retrieval-based methods [20,24] are also used to develop user interfaces. Reiss [66] parses developers' sketch into structured queries to search related UIs of Java-based desktop software in the database.…”

Section: Ui Designmentioning

confidence: 99%

From Lost to Found

Chen

Feng

Liu

et al. 2020

Proc. ACM Hum.-Comput. Interact.

Self Cite

View full text Add to dashboard Cite

Design sharing sites provide UI designers with a platform to share their works and also an opportunity to get inspiration from others' designs. To facilitate management and search of millions of UI design images, many design sharing sites adopt collaborative tagging systems by distributing the work of categorization to the community. However, designers often do not know how to properly tag one design image with compact textual description, resulting in unclear, incomplete, and inconsistent tags for uploaded examples which impede retrieval, according to our empirical study and interview with four professional designers. Based on a deep neural network, we introduce a novel approach for encoding both the visual and textual information to recover the missing tags for existing UI examples so that they can be more easily found by text queries. We achieve 82.72% accuracy in the tag prediction. Through a simulation test of 5 queries, our system on average returns hundreds more results than the default Dribbble search, leading to better relatedness, diversity and satisfaction. CCS Concepts: • Human-centered computing → User interface design; User interface management systems; Graphical user interfaces.

show abstract

Gallery D.C.: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery

Cited by 62 publications

References 22 publications

Object detection for graphical user interface: old fashioned or deep learning or a combination?

Object detection for graphical user interface: old fashioned or deep learning or a combination?

UIED: a hybrid tool for GUI element detection

From Lost to Found

Contact Info

Product

Resources

About