Visual and Statistical Methods to Calculate Interrater Reliability for Time-Resolved Qualitative Data: Examples from a Screen Capture Study of Engineering Writing Patterns

Malviya, Manoj; Berdanier, Catherine G.P.; Buswell, Natascha Trellinger

doi:10.18260/1-2--33541

Cited by 2 publications

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the purposes of developing time stamp and frequency data of each activity, we then assigned each code a numerical value for ease of data processing in MATLAB and MS Excel. The numerical values have no significance on importance or order (e.g., category 1 is not superior or inferior to category 3), but are useful for computational bookkeeping purposes, a method applied in other studies [28][29][30]. The behavioral data, represented by numerical values, was then be analyzed as a function of the percentage of the total time spent which is further discussed in the results section.…”

Section: Analysis Of Human Design Behaviorsmentioning

confidence: 99%

An Empirical Study linking Additive Manufacturing Design Process to success in Manufacturability

Mehta¹

2019

Preprint

Self Cite

View full text Add to dashboard Cite

This paper characterizes engineering designers’ abilities to re-design a component for additive manufacturing, employing screen capture methods. Additive Manufacturing has garnered significant interest from a wide range of industries, academia and government stakeholders due to its potential to reform and disrupt traditional manufacturing processes. The technology offers unprecedented design freedom and customization along with its ability to process novel and highstrength alloys in promising lead times. To harness the maximum potential of this technology, designers are often tasked with creating new products or re-design existing portfolios of traditionally manufactured parts to achieve lightweight designs with better performance. To date, few studies explore the correspondence between design behaviors and manufacturability of final product within an Additive Manufacturing context. This paper presents empirical data from the design processes of six graduate student engineering designers as they re-design a traditionally designed part for additive manufacturing. We compare behaviors through the design task between the study participants with a quantitative measure of the manufacturability and quality of each design. Results indicate opportunities for further research and best practices in design for Additive manufacturing and engineering education practitioners across multiple disciplines.

show abstract

Section: Analysis Of Human Design Behaviorsmentioning

confidence: 99%

An Empirical Study linking Additive Manufacturing Design Process to success in Manufacturability

Mehta¹

2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Visual and Statistical Methods to Calculate Intercoder Reliability for Time-Resolved Observational Research

Malviya

Buswell

Berdanier

2021

International Journal of Qualitative Methods

Self Cite

View full text Add to dashboard Cite

While calculating intercoder reliability (ICR) is straightforward for text-based data, such as for interview transcript excerpts, determining ICR for naturalistic observational video data is much more complex. To date, there have been few methods proposed in literature that are robust enough to handle complexities such as the occurrence of simultaneous event complexity and partial agreement by the raters. This is especially important with the emergence of high-resolution video data, which collects nearly continuous or continuous observational data in naturalistic settings. In this paper, we present three approaches to calculating ICR. First, we present the technical approach to clean and compare two coders’ results such that traditional metrics of ICR (e.g., Cohen’s κ, Krippendorff’s α, Scott’s Π) can be calculated, methods previously unarticulated in literature. However, these calculations are intensive, requiring significant data manipulation. As an alternative, this paper also proposes two novel methods to calculate ICR by algorithmically comparing visual representations of each coders’ results. To demonstrate efficacy of the approaches, we employ all three methods on data from two separate ongoing research contexts using observational data. We find that the visual methods perform as well as the traditional measures of ICR and offer significant reduction in the work required to calculate ICR, with an added advantage of allowing the researcher to set thresholds for acceptable agreement in lag time. These methods may transform the consideration of ICR in other studies across disciplines that employ observational data.

show abstract

Visual and Statistical Methods to Calculate Interrater Reliability for Time-Resolved Qualitative Data: Examples from a Screen Capture Study of Engineering Writing Patterns

Abstract: She earned her B.S. in aerospace engineering from Syracuse University and her Ph.D. in engineering education from the School of Engineering Education at Purdue University. She is particularly interested in teaching conceptions and methods and graduate level engineering education.

Cited by 2 publications

References 16 publications

An Empirical Study linking Additive Manufacturing Design Process to success in Manufacturability

An Empirical Study linking Additive Manufacturing Design Process to success in Manufacturability

Visual and Statistical Methods to Calculate Intercoder Reliability for Time-Resolved Observational Research

Contact Info

Product

Resources

About