Developing a new framework for evaluating e-learning systems: integrating BSC and FAHP

Abstract

Purpose E-learning is one area of strategic application of information technology (IT) in the educational field. In the past, massive investments in e-learning systems have neither been necessarily effective nor created value for e-learners. As a result, e-learning systems’ evaluation has become critical. Although many researchers have studied e-learning’s performance evaluation, there is little research on e-learning assessment, which uses pedagogical principles and organizational issues along with information systems (ISs) assessment measures. Thus, the purpose of this study is to consolidate multi-dimensional aspects of e-learning evaluation by applying balanced scorecard (BSC) as a strategic performance measurement method. This study further seeks a solution for designing a systematic approach, in which three equally important considerations must be balanced: organization, pedagogy and technology. Design/methodology/approach This paper introduces a framework for assessing e-learning systems by using fuzzy analytic hierarchy process (FAHP) and BSC. Owing to the growing use of BSC in strategic management, this study applies its four perspectives as a base for performance evaluation. To identify measures of each perspective, a comprehensive literature review has been conducted. Modifying the list of measures is completed in two phases: first by using a focus group consisting of six e-learning experts, and second, by using a survey analysis conducted among 256 Iranian e-learners. FAHP is applied to weigh and determine the rank-order of the measures. Findings The results propose integrating pedagogical, organizational and technical measures based on the BSC framework for evaluating e-learning systems. In the proposed framework, effective BSC perspectives and their rank-orders and weights are presented as a guideline for assessing e-learning systems in Iran. Research implications In e-learning initiatives, organizational issues must be considered along with technological and pedagogical factors simultaneously. The proposed framework can assist educational institutions and organizations to identify weaknesses and strengths of e-learning projects and guide them to select appropriate strategies for progress. Originality/value A review of the e-learning literature shows that there are few substantive theoretical accounts, which adequately integrate multiple dimensions of e-learning evaluation; yet, most researchers view e-learning evaluation as the most difficult part in the e-learning implementation process. This study proposes a new conceptual framework to evaluate e-learning initiatives systematically. The main contribution of this study is to develop a new systematic approach for e-learning systems’ evaluation that integrates two well-established managerial methodologies (BSC and FAHP), and considers pedagogical, organizational and technological aspects synchronously.

Gender Differences and Learner Satisfaction: An evaluation of E-‎Learning Systems at Umm A-Qura University

“…The main key factors that exist in the model were: Course development, Learner Support, Assessment, User characteristics, Institutional factors, and Overall performance. Pour, Hosseinzadeh, Azar, and Taheri (2017).mentioned that elearning systems' evaluation has become critical. Although many researchers have studied e-learning's performance evaluation, there is little research on e-learning assessment, which uses pedagogical principles and organizational issues along with information systems (ISs) assessment measures.…”

A comprehensive framework to rank cloud-based e-learning providers using best-worst method (BWM)

“…The successful application of IT in learning processes needs to be mediated by pedagogical issues (Jami Pour et al , 2017; Granic et al , 2009: Rubens et al , 2005; Govindasamy, 2001). In the past, technological possibilities, instead of pedagogical principles, often led to providing e-learning environments, and thus a high failure rate of these initiatives.…”

“…Therefore, one of the key activities to adopt cloud-based e-learning system is choosing the most appropriate vendors by considering the multi-dimensions of these services. As many researchers claim, e-learning systems are multi-disciplinary by nature (Jami Pour et al , 2017; Ozkan and Koseler, 2009; Momeni et al , 2013), in cloud computing service delivery, organizations must consider various related challenges (Chang et al , 2016); therefore, it is necessary to develop a comprehensive framework containing the most important criteria of vendor selection in adopting cloud-based e-learning systems. Although the e-learning market is growing and academic research works on e-learning have increased extensively over the past few years (Zare et al , 2016), there has not been a systematic framework for choosing the best cloud-based e-learning vendor yet.…”

Strategic orientations, knowledge management (KM) and business performance

“…When formulating desired future of their firms, they no longer focus chiefly on monetary success indicators in the financial area (Nisha, 2017; Asosheh et al , 2010). There are some reasons lead to increasing use of BSC in many researches and various business area such as KM (Jamporazmey and Mehrafrouz, 2012), small- and medium-sized enterprise (SME) performance evaluation (Malagueño et al , 2017), e-learning system (Jami Pour et al , 2017) and SCM strategy formulation (Khanaposhtani et al , 2017). First, traditional performance measures focused solely on financial measures while the BSC is developed to guide managers’ focus toward a broader view of the company’s operations and integrate financial and non-financial (customer, internal business, and learning and growth) in a single model (Asosheh et al , 2010; Hoque, 2014; Micheli and Mura, 2017).…”