UIED: a hybrid tool for GUI element detection

Xie, Mulong; Feng, Sidong; Xing, Zhenchang; Chen, Jieshan; Chen, Chunyang

doi:10.1145/3368089.3417940

Cited by 61 publications

(21 citation statements)

References 26 publications

(22 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While abundant recent work focused on extracting UI elements, including their hierarchical design information, from design artifacts such as mockups (e.g. [3,7,8,25,30,35]), they are not directly related to the objective of providing inspirational design examples. So we omit the detailed review of this body of literature here.…”

Section: Managing Ui Design Artifactsmentioning

confidence: 99%

GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial Network

Mozaffari,

Zhang,

Cheng

et al. 2022

Preprint

View full text Add to dashboard Cite

Inspiration from design examples plays a crucial role in the creative process of user interface design. However, current tools and techniques that support inspiration usually only focus on example browsing with limited user control or similarity-based example retrieval, leading to undesirable design outcomes such as focus drift and design fixation. To address these issues, we propose the GAN-Spiration approach that suggests design examples for both targeted and serendipitous inspiration, leveraging a style-based Generative Adversarial Network. A quantitative evaluation revealed that the outputs of GANSpiration-based example suggestion approaches are relevant to the input design, and at the same time include diverse instances. A user study with professional UI/UX practitioners showed that the examples suggested by our approach serve as viable sources of inspiration for overall design concepts and specific design elements. Overall, our work paves the road of using advanced generative machine learning techniques in supporting the creative design practice. CCS CONCEPTS• Human-centered computing → User interface design.

show abstract

Section: Managing Ui Design Artifactsmentioning

confidence: 99%

GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial Network

Mozaffari,

Zhang,

Cheng

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…With more computational bandwidth (i.e. execute more models with negligible latency), we can use other work to augment GreaseTerminator functionality; UIED [73] is an example of recent work that uses CNNs to detect GUI elements on mobile interfaces, and could be used to improve element detection and inpainting to complement multi-scale multi-template matching.…”

Section: Latencymentioning

confidence: 99%

Mind-proofing Your Phone: Navigating the Digital Minefield with GreaseTerminator

Datta¹,

Kollnig²,

Shadbolt³

2021

Preprint

View full text Add to dashboard Cite

Digital harms are widespread in the mobile ecosystem. As these devices gain ever more prominence in our daily lives, so too increases the potential for malicious attacks against individuals. The last line of defense against a range of digital harms -including digital distraction, political polarisation through hate speech, and children being exposed to damaging material -is the user interface. This work introduces GreaseTerminator to enable researchers to develop, deploy, and test interventions against these harms with end-users. We demonstrate the ease of intervention development and deployment, as well as the broad range of harms potentially covered with GreaseTerminator in five in-depth case studies.CCS Concepts: • Human-centered computing → Systems and tools for interaction design; Accessibility systems and tools.

show abstract

“…Fischer et al [53] transfer the style from fine art to GUI. Chen et al [54] study different GUI element detection methods on large-scale GUI data and develop UIED [55] to handle diverse and complicated GUI images. Other supporting works such as GUI tag prediction [56] and GUI component gallery construction [57] can enhance designers' searching efficiency.…”

Section: A Gui Designmentioning

confidence: 99%

GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks

Zhao

Chen

Liu

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

Self Cite

View full text Add to dashboard Cite

Graphical User Interface (GUI) is ubiquitous in almost all modern desktop software, mobile applications, and online websites. A good GUI design is crucial to the success of the software in the market, but designing a good GUI which requires much innovation and creativity is difficult even to well-trained designers. Besides, the requirement of the rapid development of GUI design also aggravates designers' working load. So, the availability of various automated generated GUIs can help enhance the design personalization and specialization as they can cater to the taste of different designers. To assist designers, we develop a model GUIGAN to automatically generate GUI designs. Different from conventional image generation models based on image pixels, our GUIGAN is to reuse GUI components collected from existing mobile app GUIs for composing a new design that is similar to natural-language generation. Our GUIGAN is based on SeqGAN by modeling the GUI component style compatibility and GUI structure. The evaluation demonstrates that our model significantly outperforms the best of the baseline methods by 30.77% in Frechet Inception distance (FID) and 12.35% in 1-Nearest Neighbor Accuracy (1-NNA). Through a pilot user study, we provide initial evidence of the usefulness of our approach for generating acceptable brand new GUI designs.

show abstract

UIED: a hybrid tool for GUI element detection

Cited by 61 publications

References 26 publications

GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial Network

GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial Network

Mind-proofing Your Phone: Navigating the Digital Minefield with GreaseTerminator

GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks

Contact Info

Product

Resources

About