ParamMacros: Creating UI Automation Leveraging End-User Natural Language Parameterization

Krosnick, Rebecca; Oney, Steve

doi:10.1109/vl/hcc53370.2022.9833005

Cited by 4 publications

(4 citation statements)

References 29 publications

(27 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on HintDroid, according to the GUI information of the text input component and knowledge from popular app datasets, the HintDroid can automatically generate hint-text and complete the "hint" fields of the text input component in the app. Therefore, HintDroid can also be integrated into UI automation tools [53,69,130] to provide developers with more diverse hint-text. In addition, HintDroid generates corresponding input content based on the generated hint-text, which can be used in software testing to help testers generate diverse test cases.…”

Section: Potential Applications To End-usersmentioning

confidence: 99%

Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM

Liu,

Chen,

Wang

et al. 2024

Proceedings of the CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Mobile apps have become indispensable for accessing and participating in various environments, especially for low-vision users. Users with visual impairments can use screen readers to read the content of each screen and understand the content that needs to be operated. Screen readers need to read the hint-text attribute in the text input component to remind visually impaired users what to fill in. Unfortunately, based on our analysis of 4,501 Android apps with text inputs, over 76% of them are missing hint-text. These issues are mostly caused by developers' lack of awareness when considering visually impaired individuals. To overcome these challenges, we developed an LLM-based hint-text generation model called HintDroid, which analyzes the GUI information of input components and uses in-context learning to generate the hint-text. To ensure the quality of hint-text generation, we further designed a feedback-based inspection mechanism to further adjust hint-text. The automated experiments demonstrate the high BLEU and a user study further confirms its usefulness. HintDroid can not only help visually impaired individuals, but also help ordinary people understand the requirements of input components. HintDroid demo video: https://youtu.be/FWgfcctRbfI.

show abstract

Section: Potential Applications To End-usersmentioning

confidence: 99%

Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM

Liu,

Chen,

Wang

et al. 2024

Proceedings of the CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…Systems like PLOW [14] and PUMICE [37] allow users to express concepts (e.g., hot weather) in natural language and then learn the concepts to generalize the automation. ParamMacros [25] allows users to frst generalize a concrete natural language question with potential values to identifed parameters, and then create a demonstration of how to answer the question on the website of interest. Scout [59], Designscape [50], and Iconate [65] allow users to iteratively refne their intent by directly manipulating the AI-generated artifacts.…”

Section: Programming By Demonstrationmentioning

confidence: 99%

DiLogics: Creating Web Automation Programs with Diverse Logics

Pu¹,

Yang²,

An³

et al. 2023

Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

The workfow of DiLogics. (Step 1) To create web automation programs for data entry tasks, users frst upload an input fle, which is semantically segmented into the steps representing diferent task specifcations ○ a . Users can edit inaccurately segmented steps ○ b . (Step 2) Users then follow a carousel of current steps ○ c and demonstrate the corresponding UI actions to fulfll each specifcation. DiLogics highlights semantically relevant web page elements, guiding users to perform demonstrations ○ d . (Step 3) After two iterations of demonstrations, DiLogics learns the mappings between diferent steps and actions and automates the remaining task steps, generalizing GUI actions based on the specifcation's semantic meaning. Users can refne the program logics ○ e at any stage of execution by editing the steps or adding new demonstrations ○ f .

show abstract

“…In my recent work [4] I have designed a PBD system that enables end-users to create custom automation macros that answer formulaic questions about content on a website. Our key insight is that if a user provides a single demonstration of how to answer a specific question, then if that question and the content on the website follow a structural pattern we can infer a program for answering variations of that question.…”

Section: End-user Environments For Creating Macrosmentioning

confidence: 99%

“…First, I studied the challenges and needs of programmers writing web automation scripts [3], both in a traditional text editor and in richer environments, including a prototype IDE I built that provides context about the target UI and feedback on element selection logic. Next, I designed a programming by demonstration (PBD) environment that enables end-users to create custom macros for answering questions on the web without needing to write code [4].…”

Section: Introductionmentioning

confidence: 99%