Towards a Common Framework for Evaluating Computing Outreach Activities

McGill

2019

Education Sciences

Self Cite

There has been considerable investment in pre-college educational interventions for all areas of STEM (including computer science). The goal of many of these initiatives is to engage and interest students early in their educational career. In this study, a systematic literature review was undertaken to determine the demographic and program data collected and reported for the field of computing education and for other STEM disciplines for activities that were not designed as part of the formal in-class curriculum (e.g., outreach activities). A comparison-contrast analysis of the resulting 342 articles found similarities and key differences in the reporting of this data as well as overarching characteristics of missing or incomplete reporting across disciplines. Authors from both fields reported equally well in the four categories studied: information about evaluation, participant gender, participant race and/or ethnicity, and activity demographics. However, the computing education articles were more likely to have clearly stated research questions and comparative analysis based on demographic characteristics. They were less likely to include the number of participants in the study, participant age/grade level, socioeconomic status, disability information, location of intervention, and instructor demographics. Through this analysis, it was determined that reporting can be improved across all disciplines to improve the quantity of data needed to replicate studies and to provide complete data sets that provide for the comparison of collected data.

Section: Synthesis Of Results Study Bias Limitationsmentioning

confidence: 99%

“…Based on previous work [13,14], it was hypothesized that in the more established STEM disciplines, a greater reliance on longitudinal studies was present. Unfortunately, that was not the case in this review with only 19 articles (13%) reporting on longitudinal studies.…”

Section: Techniques For Data Collection and Reporting In Computing Vementioning

confidence: 99%

A Systematic Review Exploring the Differences in Reported Data for Pre-College Educational Activities for Computer Science, Engineering, and Other STEM Disciplines

McGill

2019

Education Sciences

Self Cite

“…Within computing education, recent literature reviews and position papers have called for increased rigor in methodology (e.g., Lishinski et al, 2016). More specifically, papers that seek to aggregate data across multiple studies have decried the lack of standardization in variables that are measured and how they are measured (Decker et al, 2016;Decker et al, 2017;Ihantola et al, 2015;McGill et al, 2018).…”

Section: Increasing Standardizationmentioning

confidence: 99%

Review of measurements used in computing education research and suggestions for increasing standardization

Margulieux

Ketenci

Computer Science Education

2019

Self Cite

The variables that researchers measure and how they measure them are central in any area of research. Which research questions can be asked and how they are answered depends on measurement. This paper describes a systematic review of the literature in computing education research to summarize the commonly used variables and measurements in 197 papers and to compare them to best practices in measurement for human-subjects research. Characteristics of the literature that are examined in the review include variables measured (including learner characteristics), measurements used, and type of data analysis. The review illuminates common practices related to each of these characteristics and their interactions with other characteristics. The paper lists standardized measurements that were used in the literature and highlights commonly used variables for which no standardized measures exist. To conclude, this review compares common practice in computing education to best practices in human-subjects research to make recommendations for increasing rigor.

“…Within research in K-12 computing education, previous research has shown that studies and interventions are not well described, missing key variables for understanding the populations studied and the intervention its [10,12,23]). Researchers have also discovered weaknesses in the way statistics are reported and used, such as under-reporting of effect size [24,27,29].…”

Section: Introductionmentioning

confidence: 99%

A Gap Analysis of Statistical Data Reporting in K-12 Computing Education Research

McGill

Proceedings of the 51st ACM Technical Symposium on Computer Science Education

2020

Self Cite

The quality of reporting of experimental results in computing education literature has been previously shown to be less than rigorous. In this study, we first examined research standards set forth by four organizations: American Psychology Association (APA), American Educational Research Association (AERA), What Works Clearinghouse (WWC), and the CONsolidated Standards of Reporting Trials (CONSORT). We selected the most important data standards based on their prominence across all four and the most typical study designs in computing education research. We then examined 76 articles designated as quantitative research studies (K-12) published in ten venues (2012-2018) to determine whether the reporting in these articles met these five standards. Findings indicate that only 48% of these articles report effect size and even fewer (11%) report confidence intervals and levels. We found that reported data did not meet the standard that data should be "reported in a way that the reader could construct effect-size estimates and confidence intervals beyond those supplied in the paper". Additionally, authors used existing instruments less than a quarter of the time (24%) and used instruments with evidence of reliability and validity less than half of the time (39%). We conclude with recommendations for those in the K-12 computing education research community to consider when reporting statistical data in future work so that we can increase the level of rigorous reporting in this growing field.