Cody: An AI-Based System to Semi-Automate Coding for Qualitative Research

Rietz, Tim; Maedche, Alexander

doi:10.1145/3411764.3445591

Cited by 29 publications

(10 citation statements)

References 43 publications

(88 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ML is more generalizable and easy to apply for the end user, but the results may not be directly apparent to researchers. A combination of both methods has also been proposed via user-defined query-style code rules merged with supervised ML (Rietz & Maedche, 2021). Rietz and Maedche proposed a user-centric methodology based on Interactive Machine Learning (IML) for a more transparent learning process.…”

Section: Background and Related Workmentioning

confidence: 99%

Bridging Qualitative Data Silos: The Potential of Reusing Codings Through Machine Learning Based Cross-Study Code Linking

Wildemann,

Niederée,

Elejalde

2023

Social Science Computer Review

View full text Add to dashboard Cite

For qualitative data analysis (QDA), researchers assign codes to text segments to arrange the information into topics or concepts. These annotations facilitate information retrieval and the identification of emerging patterns in unstructured data. However, this metadata is typically not published or reused after the research. Subsequent studies with similar research questions require a new definition of codes and do not benefit from other analysts’ experience. Machine learning (ML) based classification seeded with such data remains a challenging task due to the ambiguity of code definitions and the inherent subjectivity of the exercise. Previous attempts to support QDA using ML rely on linear models and only examined individual datasets that were either smaller or coded specifically for this purpose. However, we show that modern approaches effectively capture at least part of the codes’ semantics and may generalize to multiple studies. We analyze the performance of multiple classifiers across three large real-world datasets. Furthermore, we propose an ML-based approach to identify semantic relations of codes in different studies to show thematic faceting, enhance retrieval of related content, or bootstrap the coding process. These are encouraging results that suggest how analysts might benefit from prior interpretation efforts, potentially yielding new insights into qualitative data.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Bridging Qualitative Data Silos: The Potential of Reusing Codings Through Machine Learning Based Cross-Study Code Linking

Wildemann,

Niederée,

Elejalde

2023

Social Science Computer Review

View full text Add to dashboard Cite

show abstract

“…Code generation models can improve programming education and provide personalized learning experiences based on individual needs and learning styles [1]. They can improve the efficiency and effectiveness of software development, improve software testing and quality assurance, source code vulnerability detection, and accessibility testing [2].…”

Section: Introductionmentioning

confidence: 99%

Assessing the Effectiveness and Security Implications of AI Code Generators

Taeb,

Chi,

Bernadin

2024

CISSE

View full text Add to dashboard Cite

Students, especially those outside the field of cybersecurity, are increasingly turning to Large Language Model (LLM)-based generative AI tools for coding assistance. These AI code generators provide valuable support to developers by generating code based on provided input and instructions. However, the quality and accuracy of the generated code can vary, depending on factors such as task complexity, the clarity of instructions, and the model’s familiarity with the programming language. Additionally, these generated codes may inadvertently utilize vulnerable built-in functions, potentially leading to source code vulnerabilities and exploits. This research undertakes an in-depth analysis and comparison of code generation, code completion, and security suggestions offered by prominent AI models, including OpenAI CodeX, CodeBert, and ChatGPT. The research aims to evaluate the effectiveness and security aspects of these tools in terms of their code generation, code completion capabilities, and their ability to enhance security. This analysis serves as a valuable resource for developers, enabling them to proactively avoid introducing security vulnerabilities in their projects. By doing so, developers can significantly reduce the need for extensive revisions and resource allocation, whether in the short or long term.

show abstract

“…Researchers have built AI-based tools to assist qualitative analysis [7,[10][11][12]. These tools use natural language processing (NLP) and machine learning (ML) algorithms to help researchers identify patterns and themes in qualitative data.…”

Section: Introductionmentioning

confidence: 99%

Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding

Xiao

Yuan

Liao

et al. 2023

28th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

Qualitative analysis of textual contents unpacks rich and valuable information by assigning labels to the data. However, this process is often labor-intensive, particularly when working with large datasets. While recent AI-based tools demonstrate utility, researchers may not have readily available AI resources and expertise, let alone be challenged by the limited generalizability of those task-specific models. In this study, we explored the use of large language models (LLMs) in supporting deductive coding, a major category of qualitative analysis where researchers use pre-determined codebooks to label the data into a fixed set of codes. Instead of training task-specific models, a pre-trained LLM could be used directly for various tasks without fine-tuning through prompt learning. Using a curiosity-driven questions coding task as a case study, we found, by combining GPT-3 with expert-drafted codebooks, our proposed approach achieved fair to substantial agreements with expert-coded results. We lay out challenges and opportunities in using LLMs to support qualitative coding and beyond.CCS Concepts: • Human-centered computing → HCI design and evaluation methods; • Computing methodologies → Natural language processing.

show abstract

Cody: An AI-Based System to Semi-Automate Coding for Qualitative Research

Cited by 29 publications

References 43 publications

Bridging Qualitative Data Silos: The Potential of Reusing Codings Through Machine Learning Based Cross-Study Code Linking

Bridging Qualitative Data Silos: The Potential of Reusing Codings Through Machine Learning Based Cross-Study Code Linking

Assessing the Effectiveness and Security Implications of AI Code Generators

Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding

Contact Info

Product

Resources

About