A Review of Standardised Assessment Development Procedure and Algorithms for Computer Adaptive Testing: Applications and Relevance for Fourth Industrial Revolution

Abstract

Teaching and learning have gone online in response to the pandemic, which reveals the need for accurately tailored educational assessments to ascertain the extent to which learning outcomes or objectives are achieved. Computer Adaptive Testing (CAT) is a technology-driven form of assessment that tailors items to a candidate's ability level with empirically proven benefits over the fixed-form computer based test. A systematic review was employed which shows that item bank is a key requirement for CAT and the items must through a rigorous item development process to ensure and maintain quality in terms of content, criterion constructs and internal consistency, determining the psychometric validation of behavioural measures while leveraging on variances of Item Response Theory (IRT). Following the item development stage is the need to compile validated items into administrable forms using advanced computer software for automatic test assembly and administration, such as FastTest which allows specifying empirically tried algorithms for CAT from start to termination of the test. This helps to ensure that assessment properly leverages the advantages that CAT holds. Furthermore, the review revealed that CAT has been widely applied with large-scale testing in various fields by educational, health and psychological professionals utilising different IRT models; however only in developed countries. This brings to bear the need for adoption in other parts of the world, for improvements in educational assessments. The interjections of 4IR with AI considering emerging technology aids the CAT algorithm for achieving expert and knowledge-based systems, being a requirement for survival in today’s world.

Students' and lecturers' perceptions of computerised adaptive testing as the future of assessing students

“…This, therefore, should ensure that questions that are neither too difficult nor too easy are posed. CAT measures the accuracy of a test score in relation to the duration of the test (Oladele & Ndlovu, 2021).…”

“…Given the increased algorithm flexibility, there is increased security in the administration of the assessment. Furthermore, students engaging in CAT have reported having decreased levels of stress, tiredness, boredom, and anxiety (Oladele & Ndlovu, 2021).…”

Transition from computer-based testing of national benchmark tests to adaptive testing: Robust application of fourth industrial revolution tools

“…CAT item banks can include both existing paper and pencil items as well as new items, which will maintain psychometric properties while minimizing the cost implications associated with adding additional items (Birdsall, 2011;Thompson & Weiss, 2011). Oladele and Ndlovu (2021) citing Germain (2006), suggested that items should be developed considering content, test domains, and internal consistency, thus ensuring that behavioural measures are valid psychometrically. Most tests are composed of carefully designed items to ensure that they are valid, able to test the subjects intended, and reliable, i.e., they are consistent between administrations.…”

“…In the IRT model for CAT, only appropriate item pools are used to select items according to examinee ability levels over a series of iterative steps from item selection to stopping criteria until an accurate estimation of theta (θ) level is achieved. An ideal theta level should contain hundreds of items that are uniformly distributed in difficulty, highly discriminating, and have a low guessing parameter for the purpose of effectively assessing the test assembly (Birdsall, 2011;Oladele & Ndlovu, 2021).…”

“…Several implications arise from the 4IR in terms of skills development education assessment, such as a high-end computer terminal provides multiple connectivity options and high storage capacities for educational assessments using CAT technology. Artificial intelligence breakthroughs based on emerging technology can enhance educational testing possibilities (Oladele & Ndlovu, 2021;Schwab, 2016). Schwab (2016) argues that 4IR leverages AI to create benefits that are becoming realities, with strong evidence that the 4IR technologies will significantly influence businesses, and the educational sector is not immune.…”