Identifying design values across countries through adaptive comparative judgment

Bartholomew, Scott R.; Ruesch, Emily Yoshikawa; Hartell, Eva; Strimel, Greg J.

doi:10.1007/s10798-019-09506-8

Cited by 12 publications

(8 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This practice of submitting design portfolios is common in CID education as a method of assisting teachers in assessing the processes and outcomes of students in design scenarios (Bartholomew et al . 2019, p. 8). The results were analysed both quantitatively and qualitatively (Gunday Gul & Afacan 2018, p. 473).…”

Section: Methodsmentioning

confidence: 99%

How Does the Design Guidelines for Traditional Cultural Artefacts Inspire Design in a Culturally Inspired Design Process? A Comparative Study with Novice Design Students

Dong¹,

Zhu²,

Li³

et al. 2023

Int J Art Design Ed

View full text Add to dashboard Cite

Design guidelines for traditional cultural artefacts play an important role in CulturallyInspired Design (CID) activities and design education. In this paper, a comparative study was performed to assess the impact of present-day design guidelines for traditional Chinese cultural artefacts on novice designers in a CID process. In all, 42 novice designers enrolled in a cultural product design course participated in this study, under two different design conditions: an unaided condition and a guideline-aided condition. Each condition includes three stages: identification stage, translation stage and implementation stage. In the unaided condition, each participant was asked to finish an investigative report on traditional cultural artefacts without any guidance and then carry out a detailed cultural product design. These results were then compared with those of the participants in the guideline-aided condition, who received a design guideline for traditional cultural artefacts. The results were assessed by expert raters against six design metrics: breadth, depth, quantity, variety, novelty and quality. They revealed that the design guideline for traditional cultural artefacts in a CID process promoted the analysis of cultural features and increased the novelty and quality of design outcomes but resulted in decreased variety. We propose that the design guideline for traditional cultural artefacts may be useful as part of the design process and as a pedagogical tool in cultural creative design,

show abstract

Section: Methodsmentioning

confidence: 99%

How Does the Design Guidelines for Traditional Cultural Artefacts Inspire Design in a Culturally Inspired Design Process? A Comparative Study with Novice Design Students

Dong¹,

Zhu²,

Li³

et al. 2023

Int J Art Design Ed

View full text Add to dashboard Cite

show abstract

“…These follow a cubic function (Kimbell et al, 2007;Kimbell, 2012) and offer insight into relative performance between portfolios. This is commonly noted as a significant benefit of ACJ (Bartholomew et al, 2020b;Buckley et al, 2020a) and its potential was demonstrated by Seery et al (2019) where parameter values were transposed into student grades. However, despite articles claiming the benefit of parameter values over the rank order which is linear and thus does not present relative differences, much of the data analysis does not utilize these values.…”

Section: Theme 3: Validity In the Determination Of Validitymentioning

confidence: 98%

“…Related concepts merit further definition, in particular "time" and "criteria." Many studies examine the efficiency of ACJ in comparison to traditional assessment practices (Rowsome et al, 2013;Bartholomew et al, 2018aBartholomew et al, , 2020bZhang, 2019) however, for ACJ time is usually considered in terms of total or average judging time. There is need to consider any set-up or training times to give a truer reflection of the impact this could have on practice, and any comparisons would need to consider the time educators put into developing rubrics and repeat usage as well.…”

Section: Theme 2: Theoretical Framing To Define the Many Elements Of ...mentioning

confidence: 99%

A Review of the Valid Methodological Use of Adaptive Comparative Judgment in Technology Education Research

2022

View full text Add to dashboard Cite

There is a continuing rise in studies examining the impact that adaptive comparative judgment (ACJ) can have on practice in technology education. This appears to stem from ACJ being seen to offer a solution to the difficulties faced in the assessment of designerly activity which is prominent in contemporary technology education internationally. Central research questions to date have focused on whether ACJ was feasible, reliable, and offered broad educational merit. With exploratory evidence indicating this to be the case, there is now a need to progress this research agenda in a more systematic fashion. To support this, a critical review of how ACJ has been used and studied in prior work was conducted. The findings are presented thematically and suggest the existence of internal validity threats in prior research, the need for a theoretical framework and the consideration of falsifiability, and the need to justify and make transparent methodological and analytical procedures. Research questions now of pertinent importance are presented, and it is envisioned that the observations made through this review will support the design of future inquiry.

show abstract

“…To provide validation to ACJ data (parameter values), researchers selectively analyzed 20 POV statements out of the 124 POV statements as was done in a previous related study ( Bartholomew et al, 2020b). Based on ACJ, we selectively analyzed the 10 POV statements with the highest parameter values and the 10 POV statements with the lowest parameter values to provide contrasting cases.…”

Section: Sample Selections Of Point-of-view Statementsmentioning

confidence: 99%

Examining the Validity of Adaptive Comparative Judgment for Peer Evaluation in a Design Thinking Course

2021

Self Cite

View full text Add to dashboard Cite

Adaptive comparative judgment (ACJ) is a holistic judgment approach used to evaluate the quality of something (e.g., student work) in which individuals are presented with pairs of work and select the better item from each pair. This approach has demonstrated high levels of reliability with less bias than other approaches, hence providing accurate values in summative and formative assessment in educational settings. Though ACJ itself has demonstrated significantly high reliability levels, relatively few studies have investigated the validity of peer-evaluated ACJ in the context of design thinking. This study explored peer-evaluation, facilitated through ACJ, in terms of construct validity and criterion validity (concurrent validity and predictive validity) in the context of a design thinking course. Using ACJ, undergraduate students (n = 597) who took a design thinking course during Spring 2019 were invited to evaluate design point-of-view (POV) statements written by their peers. As a result of this ACJ exercise, each POV statement attained a specific parameter value, which reflects the quality of POV statements. In order to examine the construct validity, researchers conducted a content analysis, comparing the contents of the 10 POV statements with highest scores (parameter values) and the 10 POV statements with the lowest scores (parameter values)—as derived from the ACJ session. For the criterion validity, we studied the relationship between peer-evaluated ACJ and grader’s rubric-based grading. To study the concurrent validity, we investigated the correlation between peer-evaluated ACJ parameter values and grades assigned by course instructors for the same POV writing task. Then, predictive validity was studied by exploring if peer-evaluated ACJ of POV statements were predictive of students’ grades on the final project. Results showed that the contents of the statements with the highest parameter values were of better quality compared to the statements with the lowest parameter values. Therefore, peer-evaluated ACJ showed construct validity. Also, though peer-evaluated ACJ did not show concurrent validity, it did show moderate predictive validity.

show abstract

Identifying design values across countries through adaptive comparative judgment

Cited by 12 publications

References 31 publications

How Does the Design Guidelines for Traditional Cultural Artefacts Inspire Design in a Culturally Inspired Design Process? A Comparative Study with Novice Design Students

How Does the Design Guidelines for Traditional Cultural Artefacts Inspire Design in a Culturally Inspired Design Process? A Comparative Study with Novice Design Students

A Review of the Valid Methodological Use of Adaptive Comparative Judgment in Technology Education Research

Examining the Validity of Adaptive Comparative Judgment for Peer Evaluation in a Design Thinking Course

Contact Info

Product

Resources

About