Task difficulty prediction of figural analogies

Blum, Diego; Holling, Heinz; Galibert, María Silvia; Forthmann, Boris

doi:10.1016/j.intell.2016.03.001

Cited by 19 publications

(31 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The properties of each of the automatically generated items resemble the ones described by Blum et al (2016) , and the purpose of such items is to measure figural analogical reasoning ( Blum et al, 2011 ). The nine rules described in the introduction section of the present research were manipulated, and they were either used alone or combined during the AIG.…”

Section: Methodsmentioning

confidence: 99%

“…For a thorough understanding of how rules take part in problem-solving strategies of figural items, see Carpenter et al (1990) , and Blum et al (2015) . In the work of Blum et al (2016) , for example, the following rules were manipulated to construct 2 × 2 figural matrix items aiming to measure analogical reasoning, where each item offers the possibility to apply a unique rule or group of rules to two solution pathways (i.e., A:B::C:D and A:C::B:D) in order to reach the same missing D element:…”

Section: Introductionmentioning

confidence: 99%

“…Throughout the paper, it will be particularly important to determine the extent to which IMak-generated figural analogy rules can predict item psychometric difficulty ( β ), since the more plausible this β prediction is, the more certain we can be about the utility of the IMak package in research psychology. Furthermore, as the aforementioned rules employed by Blum et al (2016) have been used for the AIG process of the present research as well, this gives us the chance to compare both studies and to see whether rule contributions to β show stability over time. Finally, the IMak functions for figural analogy item generation are described.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Automatic Generation of Figural Analogies With the IMak Package

Blum

Holling

2018

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

Automatic Item Generation (AIG) techniques are offering innovative ways to produce test items as they overcome many disadvantages involving standard item writing, such as time-consuming work and resource-intensive demands. Although this field is relatively new, it is progressing at a high speed, and several contributions have been accomplished. Nevertheless, a scarce amount of AIG software evidencing favorable psychometric properties of the generated items has been made accessible to the broad scientific community. This research had two goals: first, to present an empirical study of items produced with the aid of the Item Maker (IMak) package available online and, second, to present IMak itself for the automatic generation of figural analogies. We were particularly interested in assessing whether automatically created figural analogy rules could predict item psychometric difficulty. A total of 23 items were generated and administered to 307 participants, 49.51% from Germany. The mean age was 28.61 (SD = 10.19) and 57.65% of the participants were female. Results reveal adequate psychometric properties including convergent validity, that most of the manipulated rules contribute to item difficulty, and that rule-based difficulty prediction is possible to some extent. In other words, psychometric quality of the generated items is supported, which reveals the utility of the IMak package in assessment settings. Finally, the package is presented and its functions for figural analogy item generation are further described.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Automatic Generation of Figural Analogies With the IMak Package

Blum

Holling

2018

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

show abstract

“…We now apply BICA to find Bayesian D-optimal designs for the 2PL model assuming one item and different prior distributions on = [−3, 3] × [0.1, 2]. The prior distributions of interest come from Blum et al (2016) and they are:…”

Section: Bayesian D-optimal Designs For Test-item Calibrationmentioning

confidence: 99%

A Metaheuristic Adaptive Cubature Based Algorithm to Find Bayesian Optimal Designs for Nonlinear Models

Masoudi

Holling

Duarte

et al. 2019

Journal of Computational and Graphical Statistics

Self Cite

View full text Add to dashboard Cite

“…To sum up, evidence on strategy use and informativeness of distractors in figural matrix items seemingly adhere to the idea behind nested logit models. Hence, in combination with the use of rule-based distractor generation (Guttman and Schlesinger 1967;Hornke and Habon 1986;Matzen et al 2010;Blum et al 2016;Blum and Holling 2018) to construct items with discriminating distractors, this item family appears to be promising for test development based on nested logit models. In particular, this allows the construction of tests with higher measurement precision at the lower end of the ability range because differentiated information about the ability of those who did not solve the item is taken from distractor choices (Myszkowski and Storme 2018;Storme et al 2019).…”

Section: Introductionmentioning

confidence: 99%

How Much g Is in the Distractor? Re-Thinking Item-Analysis of Multiple-Choice Items

et al. 2020

Self Cite

View full text Add to dashboard Cite

Distractors might display discriminatory power with respect to the construct of interest (e.g., intelligence), which was shown in recent applications of nested logit models to the short-form of Raven’s progressive matrices and other reasoning tests. In this vein, a simulation study was carried out to examine two effect size measures (i.e., a variant of Cohen’s ω and the canonical correlation RCC) for their potential to detect distractors with ability-related discriminatory power. The simulation design was adopted to item selection scenarios relying on rather small sample sizes (e.g., N = 100 or N = 200). Both suggested effect size measures (Cohen’s ω only when based on two ability groups) yielded acceptable to conservative type-I-error rates, whereas, the canonical correlation outperformed Cohen’s ω in terms of empirical power. The simulation results further suggest that an effect size threshold of 0.30 is more appropriate as compared to more lenient (0.10) or stricter thresholds (0.50). The suggested item-analysis procedure is illustrated with an analysis of twelve Raven’s progressive matrices items in a sample of N = 499 participants. Finally, strategies for item selection for cognitive ability tests with the goal of scaling by means of nested logit models are discussed.

show abstract

Task difficulty prediction of figural analogies

Cited by 19 publications

References 24 publications

Automatic Generation of Figural Analogies With the IMak Package

Automatic Generation of Figural Analogies With the IMak Package

A Metaheuristic Adaptive Cubature Based Algorithm to Find Bayesian Optimal Designs for Nonlinear Models

How Much g Is in the Distractor? Re-Thinking Item-Analysis of Multiple-Choice Items

Contact Info

Product

Resources

About