Applying Models to National Surveys of Undergraduate Science Students: What Affects Ratings of Satisfaction?

Abstract

Many countries use national-level surveys to capture student opinions about their university experiences. It is necessary to interpret survey results in an appropriate context to inform decision-making at many levels. To provide context to national survey outcomes, we describe patterns in the ratings of science and engineering subjects from the UK's National Student Survey (NSS). New, robust statistical models describe relationships between the Overall Satisfaction' rating and the preceding 21 core survey questions. Subjects exhibited consistent differences and ratings of "Teaching", "Organisation" and "Support" were thematic predictors of "Overall Satisfaction" and the best single predictor was "The course was well designed and running smoothly". General levels of satisfaction with feedback were low, but questions about feedback were ultimately the weakest predictors of "Overall Satisfaction". The UK's universities affiliated groupings revealed that more traditional "1994" and "Russell" groups over-performed in a model using the core 21 survey questions to predict "Overall Satisfaction", in contrast to the under-performing newer universities in the Million+ and Alliance groups. Findings contribute to the debate about "level playing fields" for the interpretation of survey outcomes worldwide in terms of differences between subjects, institutional types and the questionnaire items.

National Student Survey: Reliability and prediction of overall satisfaction scores with special reference to UK Dental Schools

“…Bell and Brooks have also undertaken quantitative analysis of data from the NSS (2015 data); they looked at subject and regional differences and also performed correlation between all sub‐categories, including with overall satisfaction. Again, the same trend was found with teaching and academic support, with organisation and management having the highest correlation to overall satisfaction, although interestingly they identified a higher role for teaching and even weaker role for assessment and feedback compared to Langan et al…”

“…Interestingly, they also confirmed the early finding of Richardson et al that the interaction of university and subject had a contribution to overall scores more so than individual subject or university. With regards to overall student satisfaction, as with Langan's work, teaching, and organisation and management had the highest correlation. They also highlight that teaching scores show relatively little variation across the higher education sector with all universities scoring relatively high and thus suggest that for institutions seeking to improve NSS scores, attention should be given to improving the organisation and management domain.…”

“…It is also not surprising that they concluded that overall satisfaction is complex and associated with the subject studied. In later work, the same group used a different approach (Random Forests Analysis) on data from the 2007, 2008 and 2009 NSS and again showed that ratings differed between subjects with teaching, organisation and academic support being predictors of overall satisfaction, with “the course was well designed and running smoothly” being the single best predictor. In both of these studies, assessment and feedback was the weakest predictor of overall satisfaction (again noting that scores for feedback were generally low), which correlates with our findings as well as the national trend of satisfaction in this domain in all dental schools.…”

“…Studies aiming to correlate various factors of educational provision with favourable NSS outcome have been published, and there is limited work assessing the determinants of overall satisfaction: Burgess et al have undertaken a 10‐year study which has shown a pattern of increase in satisfaction each year, the reasons for which remain unclear and largely subjective and although may reflect “gaming” by higher education institutes are more likely a true reflection of real change in quality, with universities responding to ratings from the NSS. Langan et al have looked at relationships between individual question responses and overall satisfaction, but to date such data do not exist for dentistry.…”

Coherence between text comments and the quantitative ratings in the UK’s National Student Survey

“…However, it is clearly of benefit to institutions to perform well in the well-publicised survey, as HEIs compete for student applications and institutional prestige (Gibbs 2012). It is arguable that the impact of the national survey has increased beyond its intention and capabilities (see Langan et al 2013) as it's outcomes are bring acted upon directly by HEIs in an attempt to improve their metrics as the priority, rather than a primary focus on the educational gains of its learners (Gibbs 2010;. Apart from improvement of the learning experience quality itself, it is in the interest of institutions to enhance (directly) the survey metrics per se due to their public usage (Gibbs 2010).…”