Detecting speech act types in developer question/answer conversations during bug repair

Abstract

is paper targets the problem of speech act detection in conversations about bug repair. We conduct a "Wizard of Oz" experiment with 30 professional programmers, in which the programmers x bugs for two hours, and use a simulated virtual assistant for help. en, we use an open coding manual annotation procedure to identify the speech act types in the conversations. Finally, we train and evaluate a supervised learning algorithm to automatically detect the speech act types in the conversations. In 30 two-hour conversations, we made 2459 annotations and uncovered 26 speech act types. Our automated detection achieved 69% precision and 50% recall. e key application of this work is to advance the state of the art for virtual assistants in so ware engineering. Virtual assistant technology is growing rapidly, though applications in so ware engineering are behind those in other areas, largely due to a lack of relevant data and experiments. is paper targets this problem in the area of developer Q/A conversations about bug repair. CCS CONCEPTS•So ware and its engineering → Maintaining so ware;

A Wizard of Oz Study Simulating API Usage Dialogues With a Virtual Assistant

“…A feature of our study, differing from Wood et al [11] but similar to Benzmüller et al [74] and Kruijff-Korbayová et al [75], is that we hired wizards as experimental participants in addition to the programmers. The decision to hire multiple wizards was intended to enable us to collect a more diverse set of dialogue strategies; by hiring wizards who were unfamiliar with the APIs and allowing them to participate in multiple sessions, we anticipated that there would be a learning effect in which wizards would adopt new strategies as they became more familiar with the API upon completing successive sessions.…”

“…Many modern natural language understanding frameworks (such as Alexa Skills [25] or Xatkit [26]) require explicit samples of user phrases that correspond to different dialogue acts; rather than attempt to intuit the phrases that real users would use, virtual assistant developers can extract real examples directly from the Wizard of Oz data. Other approaches to dialogue act classification use Wizard of Oz data to train statistical models to generalize to unseen inputs [11], [27]. That said, the key advantage to the Wizard of Oz approach described by Rieser and Lemon is that it enables researchers to efficiently design optimal dialogue strategies for tasks in domains where no prior data is available.…”

“…To achieve unbiased results in this sort of analysis, Hsieh and Shannon [89] suggest using an "auditing" process, in which experts discuss the application of label sets and resolve any ambiguities. This type of procedure has frequently been used in the social sciences [93], [94], and more recently, in software engineering [11]. This process allows for the creation of a single, higher-quality set of annotations for use in applications that are less willing to rely on imperfect data [11].…”

“…For instance, a classifier trained on this corpus would be unlikely to correctly identify an utterance of the OFFER type, as it would have been exposed to very few training instances of that type. Researchers have previously used techniques such as downsampling [95], oversampling, and SMOTE [11] to counteract the effects of imbalanced data in dialogue act modeling, but these solutions are typically poor replacements for real data. In terms of dialogue strategy training, this imbalance may reduce the fidelity of a simulated learning environment, ultimately leaving a virtual assistant under-equipped to handle certain dialogue states.…”

Predictive Models in Software Engineering: Challenges and Opportunities

“…They constructed nine measures of involvement and environment based on events recorded in an issue tracking system and used logistic regression model to predict long term contributors. Wood et al [293] conducted an empirical study with 30 professional programmers and trained a supervised learning algorithm to identify speech act types in developers' conversations in order to obtain useful information for bug repair. 6.7.2 Sotware Repository Mining.…”