Understanding the Challenges and Needs of Programmers Writing Web Automation Scripts

Krosnick, Rebecca; Oney, Steve

doi:10.1109/vl/hcc51201.2021.9576476

Cited by 5 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even for professional developers, creating automation programs is a non-trivial task. A study showed that experienced programmers have difculty writing web macros using common web automation frameworks [24]. Participants pointed out that a primary hurdle was the labor of checking syntactical element selectors to create their programs, causing inefciency and errors.…”

Section: Web Automationmentioning

confidence: 99%

DiLogics: Creating Web Automation Programs with Diverse Logics

Pu¹,

Yang²,

An³

et al. 2023

Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

The workfow of DiLogics. (Step 1) To create web automation programs for data entry tasks, users frst upload an input fle, which is semantically segmented into the steps representing diferent task specifcations ○ a . Users can edit inaccurately segmented steps ○ b . (Step 2) Users then follow a carousel of current steps ○ c and demonstrate the corresponding UI actions to fulfll each specifcation. DiLogics highlights semantically relevant web page elements, guiding users to perform demonstrations ○ d . (Step 3) After two iterations of demonstrations, DiLogics learns the mappings between diferent steps and actions and automates the remaining task steps, generalizing GUI actions based on the specifcation's semantic meaning. Users can refne the program logics ○ e at any stage of execution by editing the steps or adding new demonstrations ○ f .

show abstract

Section: Web Automationmentioning

confidence: 99%

DiLogics: Creating Web Automation Programs with Diverse Logics

Pu¹,

Yang²,

An³

et al. 2023

Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

show abstract

“…Popular frameworks include Selenium [13], Puppeteer [14], Cypress [15], and Beautiful Soup [16] for the web, and Shortcuts [17] and App Actions [18] for mobile. However, writing such scripts is non-trivial -for example, it can be challenging to construct UI selectors [19] that are robust across different inputs [20]. Record and replay tools like Selenium IDE [21] and Cypress Studio [22] were designed for test automation and can generate code from a single user trace, but the code will not be generalized to work across scenarios.…”

Section: B Writing Ui Automation Scriptsmentioning

confidence: 99%

ParamMacros: Creating UI Automation Leveraging End-User Natural Language Parameterization

Krosnick

Oney

2022

2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)

Self Cite

View full text Add to dashboard Cite

Prior work in programming-by-demonstration (PBD) has explored ways to enable end-users to create custom automation without needing to write code. We propose a new end-user specification model -asking the end-user to explicitly identify parts of their natural language query that can be generalized. We built a PBD system, ParamMacros, where users first generalize a concrete natural language question -identifying parameters and their possible values -and then create a demonstration of how to answer the question on the website of interest. ParamMacros then infers a generalized program by using the user-provided parameter values to identify relevant patterns in the website's structure. In a lab study we found that participants were able to meaningfully parameterize natural language queries and felt such a parameterization and demonstration process would be useful for creating custom automation.

show abstract

“…In my prior work, I studied developers as they used these environments to write automation macros [3]. First, I studied programmers using traditional text editor environments, writing in Puppeteer.…”

Section: Developer Environments For Creating Macrosmentioning

confidence: 99%

“…In my PhD work, I have studied and designed tools for developers and end-users to more intuitively create user interface (UI) automation macros. First, I studied the challenges and needs of programmers writing web automation scripts [3], both in a traditional text editor and in richer environments, including a prototype IDE I built that provides context about the target UI and feedback on element selection logic. Next, I designed a programming by demonstration (PBD) environment that enables end-users to create custom macros for answering questions on the web without needing to write code [4].…”

Section: Introductionmentioning

confidence: 99%