On the reliability of vocational workplace-based certifications

Abstract

The assessment of vocational workplace-based qualifications in England relies on human assessors (raters). These assessors observe naturally occurring, nonstandardised evidence, unique to each learner and evaluate the learner as competent/not yet competent against content standards. Whilst these are considered difficult to measure, this study aims to provide information on the operational reliability of assessor decisions for three workplace-based vocational qualifications in Hairdressing and Electrotechnical Engineering. It is shown how existing assessment records, typically generated locally by the training providers, were collected and used in a reliability study. A suitable methodology for estimating the inter-rater reliability and internal consistency was then used with this empirical data. In general, the reliability estimates measured may be considered high. This research study shows that it may be possible to measure reliability for these qualifications using commonly used measurement methods, although several characteristics of the assessment system should be considered in their interpretation.

A Review of National Occupational Standards and the Role of Human Resource Development in their Implementation

“…For example, the United Kingdom established the Office of Qualifications and Examinations Regulation. Australia established the Australian Skills Quality Authority to control qualifications at the national level through the endorsement of qualification organizations (Harth & Hemker, 2011; Pasura, 2014).…”

The reliability of results from national tests, public examinations, and vocational qualifications in England

“…Qualifications sharing components or units face particular difficulties in producing qualification-level reliabilities, as data for shared components are difficult to collect (e.g., qualifications supported by QCF may contain shared units). It is also difficult to conduct reliability studies for some components or units (e.g., teacher assessments and competencebased assessments in vocational qualifications; see Bramley & Dahwan, 2011;Harth & Hemker, 2011;Johnson, 2011). Some qualifications have small candidate entries and could be difficult and expensive to produce reliability measures.…”

“…Another feature of this type of assessment is that candidates are entered for summative assessment only ''when ready'', and the decision to ''pass'' a candidate is taken in a sense before the assessment takes place (see Harth & Hemker, 2011). Harth and Hemker (2011) investigated the reliability of outcomes from three qualifications offered by City and Guilds in two occupational areas: Level 3 Electrotechnical Services (Electrical Installation -Buildings and Structures), Hairdressing National Vocational Qualification (NVQ) (several pathways at Levels 1, 2, and 3), and the new NVQ Certificate/Diploma in Hairdressing/Barbering/ Combined Hair Types (several pathways at Qualification and Credit Framework [QCF], Levels 1, 2, and 3) and found that:…”

Políticas De Avaliação E Reconhecimento De Saberes Na Inglaterra

“…O processo de avaliação, por sua vez, pode ser conduzido diretamente pelo avaliador designado pelo Centro de avaliação para assessorar o candidato e/ou por pessoas que acompanham o trabalhador na execução da atividade de trabalho (HARTH; HEMKER, 2013). Assim sendo, avaliadores diferentes podem fazer parte do processo de avaliação do mesmo candidato, dependendo do(s) método(s) de avaliação escolhido(s), das unidades em questão, das possibilidades de observação do candidato em trabalhos que envolvem situações de risco, dentre outros (CURCIN et al, 2014;HARTH;HEMKER, 2013).…”

“…Segundo Curcin et al (2014), normalmente, o candidato somente é avaliado quando o avaliador, o empregador ou o próprio sujeito julgam que ele está completamente apto a atender a todos os critérios de avaliação. Além disso, em diversas ocasiões, os candidatos podem repetir as atividades avaliadas até que consigam contemplar todos os critérios de avaliação, um fator que faz com que o processo de reconhecimento de saberes no QCF apresente altas taxas de aprovação (ALPHAPLUS, 2014;HARTH;HEMKER, 2013).…”