WebRobot: Web Robotic Process Automation using Interactive Programming-by-Demonstration

Abstract

It is imperative to democratize robotic process automation (RPA), as RPA has become a main driver of the digital transformation but is still technically very demanding to construct, especially for non-experts. In this paper, we study how to automate an important class of RPA tasks, dubbed web RPA, which are concerned with constructing software bots that automate interactions across data and a web browser. Our main contributions are twofold. First, we develop a formal foundation which allows semantically reasoning about web RPA programs and formulate its synthesis problem in a principled manner. Second, we propose a web RPA program synthesis algorithm based on a new idea called speculative rewriting. This leads to a novel speculate-and-validate methodology in the context of rewrite-based program synthesis, which has also shown to be both theoretically simple and practically efficient for synthesizing programs from demonstrations. We have built these ideas in a new interactive synthesizer called WebRobot and evaluate it on 76 web RPA benchmarks. Our results show that WebRobot automated a majority of them effectively. Furthermore, we show that WebRobot compares favorably with a conventional rewrite-based synthesis baseline implemented using egg. Finally, we conduct a small user study demonstrating WebRobot is also usable. CCS Concepts:• Software and its engineering → Automatic programming.

DiLogics: Creating Web Automation Programs with Diverse Logics

“…In this section, we provide necessary information for WebRobot [20], a program synthesizer that constructs a part of the DiLogics system. WebRobot utilizes only web actions and requires no programming expertise, which is consistent with our design goals.…”

“…Finally, WebRobot executes to automate the rest of the task, without users manually performing any actions. For more details on the program synthesis algorithm, please refer to the original WebRobot paper [20].…”

“…We implemented DiLogics as a Chrome browser extension, building upon the core program synthesis engine from WebRobot [20]. Primarily, it uses plain JavaScript for recording front-end interactions on the web page.…”

“…Through a formative analysis of web automation requests in online platforms such as StackOverfow, we found that non-experts are experiencing difculty in creating automation programs tailored to their need. To lower the barriers in the program creation process, existing programming-by-demonstration (PBD) systems, such as SemanticOn and Rousillon [17,20,53,56] allow users to manually perform a part of the task and construct an automation program based on the demonstrations.…”

“…In this work, we present DiLogics 1 , a PBD system built upon a program synthesizer [20] that assists non-experts in creating web automation programs with diverse and generalizable programming logics. The completed program can execute consistent actions for every data input; it also goes beyond symbolic inferences and dynamically executes diferent UI actions based on the semantic understanding of the task input and web content.…”