Deep-learning based web UI automatic programming

Kim, Bada; Park, Sang Min; Won, Taeyeon; Heo, Junyoung; Kim, Bongjae

doi:10.1145/3264746.3264807

Cited by 6 publications

(6 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the detection of GUI elements, most approaches use CNNs, including twostage detectors, such as Region-based CNN (R-CNN) and its variants (such as Faster R-CNN [Kim et al 2018]) as well as one-stage detectors, such as SSD, RetinaNet [Jain et al 2019][Pandian andSuleri 2020], YOLO [Yun et al 2018] and their variants. Several of the approaches that divide the processing into several stages use CNN for mapping the raw input image to a learned representation and then RNN for performing language modeling on the textual description associated with the input picture.…”

Section: Which Datasets Have Been Used For Building the Machine Learning Model?mentioning

confidence: 99%

“…Concerning the evaluation, the researches vary largely in terms of scientific rigor. While some stand out due to their systematic and wide evaluation (including, e.g., [Robinson 2019][Moran et al 2018] [Chen et al 2018), other either present a very superficial evaluation just citing some results without presenting the research design (i.e., [Jain et al 2019][Kim et al 2018] [Liu et al 2018b] [Halbe and Joshi 2015] [Ge 2019 or do not present any information on the evaluation of the presented approach.…”

Section: How Have the Approaches Been Evaluated?mentioning

confidence: 99%

“…Also, images taken at low-light conditions where pixels are darker, increase the inference time. [Kim et al 2018] • Accuracy • Recall…”

Section: Sketchesmentioning

confidence: 99%

See 2 more Smart Citations

Recent Progress in Automated Code Generation from GUI Images Using Machine Learning Techniques

Baulé¹,

Wangenheim²,

Wangenheim³

et al. 2020

jucs

View full text Add to dashboard Cite

The manual transformation of a user interface design into code is a costly and time-consuming process. A solution can be the automation of the generation of code based on sketches or GUI design images. Recently, Machine Learning approaches have shown promising results in detecting GUI elements for such automation. Thus, to provide an overview of existing approaches, we performed a systematic mapping study. As a result, we identified and compared 20 approaches, that demonstrate good performance results being considered useful. These results can be used by researchers and practitioners in order to improve the efficiency of the GUI design process as well as continue to evolve and improve approaches for its support.

show abstract

Section: Which Datasets Have Been Used For Building the Machine Learning Model?mentioning

confidence: 99%

Section: How Have the Approaches Been Evaluated?mentioning

confidence: 99%

See 1 more Smart Citation

Recent Progress in Automated Code Generation from GUI Images Using Machine Learning Techniques

Baulé¹,

Wangenheim²,

Wangenheim³

et al. 2020

jucs

View full text Add to dashboard Cite

show abstract

“…The main core of the model is based on the selfattention layers. The DL is used for generating source code from image of sketch design [14,15]. Both models use the Convention Neural Network (CNN) as the main unite of model design, but different in general model and method.…”

Section: Introductionmentioning

confidence: 99%

Source Code Generation-based on NLP and Ontology

Alokla

Gad

Aref

et al. 2022

IJICIS

View full text Add to dashboard Cite

Generating source code is necessary especially as software evolves in complexity and demand. Finding a mechanism to generate the source code according to the requirements will save time for developers at the stage of development of the software. In this paper, a mechanism is proposed to generate the source code based on the database schema and user requirements (user story). This model contains three layers: The first layer is to analyze each of the database schema, extract the relationships between the tables, determine the meanings of the fields and analyze the user's story to find the functions performed by each role of the software users. The second layer is deducing new functions based on what was mentioned in the first layer and extracting the knowledge that contains the solutions to the problems that are inferred. The knowledge bases used are WordNet and Backend Ontology built from scratch. In the third Layer, the solutions are converted to source code based on templates extracted from the knowledge and configured, that is applied to the templates. The model showed success in generating the source code, generating PHP source code for a site that is tested and generated seventy percent of what was required to be written by programmers.

show abstract

“…Similar system also built by Chen et al[28] where their work was able to generate GUI skeleton code from a mockup design. While most works in aiding GUI development start from mockup, there are works byRobinson [29] and Kim et al[30] where they made system capable in generating web UI from sketch or wireframe stage.…”

mentioning

confidence: 99%

Visual Recognition Of Graphical User Interface Components Using Deep Learning Technique

Rahmadi

Sudaryanto

2020

Jurnal Ilmu Komputer dan Informasi

View full text Add to dashboard Cite

Graphical User Interface (GUI) building in software development is a process which ideally need to go through several steps. Those steps in the process start from idea or rough sketch of the GUI, then refined into visual design, implemented in coding or prototype, and finally evaluated for its function and usability to discover design problem and to get feedback from users. Those steps repeated until the GUI considered satisfactory or acceptable by the user. Computer vision technique has been researched and developed to make the process faster and easier; for example generating code for implementation, or automatic GUI testing using component images. But among those techniques, there are still few for usability testing purpose. This preliminary research attempted to make the foundation for usability testing using computer vision technique by built minimalist dataset which has images of various GUI components and used the dataset in deep learning experiment for GUI components visual recognition. The experiment results showed deep learning technique suitable for the intended task, with accuracy of 95% for recognition of two different types of components, and accuracy of 72% for six different types of component.

show abstract

Deep-learning based web UI automatic programming

Cited by 6 publications

References 4 publications

Recent Progress in Automated Code Generation from GUI Images Using Machine Learning Techniques

Recent Progress in Automated Code Generation from GUI Images Using Machine Learning Techniques

Source Code Generation-based on NLP and Ontology

Visual Recognition Of Graphical User Interface Components Using Deep Learning Technique

Contact Info

Product

Resources

About