Interactive Program Synthesis by Augmented Examples

Zhang, Tianyi; Lowmanstone, London; Wang, Xinyu; Glassman, Elena L.

doi:10.1145/3379337.3415900

Cited by 32 publications

(9 citation statements)

References 58 publications

(84 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…WebRobot adopts a human-in-theloop interaction model which has shown to be an effective way to incorporate human inputs when training AI systems in the HCI community [19,45]. This model has been used in the context of program synthesis, mostly interactive PBE [23,41,43,51,60,65]. In contrast, our work incorporates human inputs in PBD and proposes a new human-in-the-loop model.…”

Section: Related Workmentioning

confidence: 99%

WebRobot: Web Robotic Process Automation using Interactive Programming-by-Demonstration

Dong¹,

Huang²,

Lam³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

It is imperative to democratize robotic process automation (RPA), as RPA has become a main driver of the digital transformation but is still technically very demanding to construct, especially for non-experts. In this paper, we study how to automate an important class of RPA tasks, dubbed web RPA, which are concerned with constructing software bots that automate interactions across data and a web browser. Our main contributions are twofold. First, we develop a formal foundation which allows semantically reasoning about web RPA programs and formulate its synthesis problem in a principled manner. Second, we propose a web RPA program synthesis algorithm based on a new idea called speculative rewriting. This leads to a novel speculate-and-validate methodology in the context of rewrite-based program synthesis, which has also shown to be both theoretically simple and practically efficient for synthesizing programs from demonstrations. We have built these ideas in a new interactive synthesizer called WebRobot and evaluate it on 76 web RPA benchmarks. Our results show that WebRobot automated a majority of them effectively. Furthermore, we show that WebRobot compares favorably with a conventional rewrite-based synthesis baseline implemented using egg. Finally, we conduct a small user study demonstrating WebRobot is also usable. CCS Concepts:• Software and its engineering → Automatic programming.

show abstract

Section: Related Workmentioning

confidence: 99%

WebRobot: Web Robotic Process Automation using Interactive Programming-by-Demonstration

Dong¹,

Huang²,

Lam³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The unification also enables Falx users to easily explore synthesis results in the visualization space as opposed to program space, which is considered challenging [27]. Besides data layout transformation, many data preparation tools also support data cleaning (e.g., handling missing data or invalid data) [48], data normalization (collecting non-relational data into relation format) [3], and string formatting [6,11,56]. Falx currently does not support directly visualizing dirty or non-relational data.…”

Section: Related Workmentioning

confidence: 99%

“…In these tools, users interact with an exploration interface to navigate and save interesting solutions. Implementation tools [6,7,11,30,46,56], instead, aim to synthesize programs to help solve a concrete task (e.g., implement a design that a user already have in mind). In these tools, the main interaction objective is to help users to disambiguate spurious programs that happen to be consistent with the user specification but are incorrect for the full task [27].…”

Section: Related Workmentioning

confidence: 99%

“…In these tools, the main interaction objective is to help users to disambiguate spurious programs that happen to be consistent with the user specification but are incorrect for the full task [27]. To solve this challenge, Wrex [6] generates readable programs for users to inspect and edit; Regae [56] and FlashProg [27] interactively ask users disambiguating questions to refine synthesis results; PUMICE [23] lets users collaborate with the agent to recursively resolve any ambiguities or vagueness through conversations and demonstrations.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Falx: Synthesis-Powered Visualization Authoring

Wang

Bodík

et al. 2021

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Modern visualization tools aim to allow data analysts to easily create exploratory visualizations. When the input data layout conforms to the visualization design, users can easily specify visualizations by mapping data columns to visual channels of the design. However, when there is a mismatch between data layout and the design, users need to spend signifcant efort on data transformation.We propose Falx, a synthesis-powered visualization tool that allows users to specify visualizations in a similarly simple way but without needing to worry about data layout. In Falx, users specify visualizations using examples of how concrete values in the input are mapped to visual channels, and Falx automatically infers the visualization specifcation and transforms the data to match the design. In a study with 33 data analysts on four visualization tasks involving data transformation, we found that users can effectively adopt Falx to create visualizations they otherwise cannot implement.

show abstract

“…In addition, code search spaces can become large, which can lead either to lengthy synthesis delays or too many suggestions. Concerns like these have led to work on interpretable synthesis [21] and interactive search space exploration [22].…”

Section: Introductionmentioning

confidence: 99%

An Integrative Human-Centered Architecture for Interactive Programming Assistants

Blinn

Moon

Griffis

et al. 2022

2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)

View full text Add to dashboard Cite

Programming has become a collaboration between human programmers, who drive intent, and interactive assistants that suggest contextually relevant editor actions. There has been considerable work on suggestion synthesis strategies-from semantic autocomplete to modern program synthesis, repair, and machine learning research. This diversity of contextually viable strategies creates a need for an integrative, human-centered perspective on the problem of programming assistant design that (1) confronts the problem of integrating a variety of synthesis strategies, fed by shared semantic analyses capable of operating on program sketches, and (2) centers the needs of the human programmer: comprehending, comparing, ranking, and filtering suggestions generated by various synthesizers, and in some cases participating in a synthesizer's search by supplying additional expressions of intent. This paper contributes a conceptual architecture and API to guide programming assistant designers as they confront these integration and human-centered design challenges. We then instantiate this architecture with two prototype end-toend assistant designs, both developed for the Hazel programming environment, that emphasize understudied design aspects, namely continuity, explainability, human-in-the-loop synthesis, and the integration of multiple analyses with multiple synthesis strategies.

show abstract

Interactive Program Synthesis by Augmented Examples

Cited by 32 publications

References 58 publications

WebRobot: Web Robotic Process Automation using Interactive Programming-by-Demonstration

WebRobot: Web Robotic Process Automation using Interactive Programming-by-Demonstration

Falx: Synthesis-Powered Visualization Authoring

An Integrative Human-Centered Architecture for Interactive Programming Assistants

Contact Info

Product

Resources

About