Information technology evaluation: issues and challenges

Abstract

Purpose -To justify an increase in information technology (IT) spending and to understand utilization of limited organizational resources on IT, the correlation between IT and business performance has been of great interest to business managers. However, business managers face issues and challenges in finding out how and to what extent IT is able to deliver the intended benefits. The purpose of this paper is to examine IT evaluation issues and challenges faced by information systems (IS) researchers, IS specialists, and business managers. Design/methodology/approach -This paper begins by reviewing the disparate discussions in past literature on IT evaluation issues and challenges. It then provides a synthesis of the disparate discussions by identifying eight issues and challenges in IT evaluation. Findings -The eight issues and challenges identified are: evaluation scope, evaluation timing, unit of analysis, level of analysis, different perspectives, different dimensions, different measures, and underpinning theoretical frameworks. It concludes with some suggestions on ways to improve IT evaluation practices. Originality/value -This paper posits that before a pragmatic IT evaluation approach can be developed, it is necessary to first understand the issues and challenges faced by IS researchers, IS specialists, and business managers in IT evaluation. Having identified the eight issues and challenges, this paper provides pointers on what needs to be considered when conducting IT evaluation.

Towards Systemic Evaluation of the Business Value of IT

“…Marthandan and Tang, 2010) when trying to isolate IT's contributions for the higher level business measures. Mirani & Lederer (1998), Giaglis et al (1999), Dedrick et al (2003), Gregor et al (2006), Tallon et al (2007) Nature of business system -Socio-technical system -Organisational structures and layers -Tasks & Processes, Business processes -Multilevel perspectives Hamilton & Chervany (1981), Barua et al (1995), Wegen & Hoog (1996), Palvia et al (2001), Marthandan & Tang (2010) Volume of IT impacts -Broad impacts -Multiple benefits & costs Simmons (1996), Mirani & Lederer (1998), Kanungo et al (1999), Irani et al (2006) Complementarity -Contextual interaction -Conversion contingencies, complementary assets -Complementary organisational resources & capital Davern & Kauffman (2000), Lee (2001), Dedrick et al (2003), Melville et al (2004) Traceability for causes and effects -Indirectness -Attributability, accountability -Locus of value vs. locus of analysis Hamilton & Chervany (1981), Ragowsky et al (1996), Wegen & Hoog (1996), Peppard et al (2007) Traceability for causes and effects The volume of impacts together with complementarity complicate the traceability of IT impacts within the business system. The indirectness of relations between IT's first order impacts to business performance grow when the hierarchies and the length of cause-and-effect structures grow (Melville et al 2004).…”

“…The indirectness is related to the attributability and accountability issues (e.g. Marthandan and Tang, 2010) when trying to isolate IT's contributions for the higher level business measures.…”

“…Giaglis et al (1999),Delone & McLean (2003),Melville et al (2004),Petter et al (2008),Davern & Wilkin (2010),Marthandan & Tang (2010) Time & dynamics -Payback delays -Evolving effects, dynamic objectives -Locus of impact vs. measuring delays -Potential vs. realised benefitsHamilton & Chervany (1981),Giaglis et al (1999),Chan (2000),Peppard et al (2007),Davern & Wilkin (2010) easy measures -Unavailability of data for ex ante -ex post comparison…”

Approaching information system evaluation methodology and techniques: a comprehensive review

“…CIATs are incapable of investigating the intangible benefits and indirect costs associated with an IS implementation (Hochstrasser and Griffiths, 1991;Irani and Love, 2002;Irani et al, 2005b). In neglecting intangible benefits, financial capital techniques are unable to offer a comprehensive evaluation of IT/IS investment (Marthandan and Tang, 2010). Kumar (2004) identifies three main areas of concern: usage, financial assets and the flexibility of IT infrastructure.…”

Proposta de modelo para seleção de ferramentas de tecnologia da informação baseado nas dimensões da gestão do portfólio

“…No trabalho de Gunasekaran, Ngai e Mcgaughey (2006), foram detectadas algumas de ciências na literatura, que se mantiveram ao decorrer dos anos (TALLON, KRAEMER;GURBAXANI, 2000;MARTHANDAN;TANG, 2010). Dentre elas, a problemática e negligenciada identi cação e cálculo das métricas intangíveis e não nanceiras de desempenho com relevância para a justi cativa de TI.…”