Web Evaluation at the US National Institutes of Health: Use of the American Customer Satisfaction Index Online Customer Survey

Abstract

Background The National Institutes of Health (NIH), US Department of Health and Human Services (HHS), realized the need to better understand its Web users in order to help assure that websites are user friendly and well designed for effective information dissemination. A trans-NIH group proposed a trans-NIH project to implement an online customer survey, known as the American Customer Satisfaction Index (ACSI) survey, on a large number of NIH websites—the first “enterprise-wide” ACSI application, and probably the largest enterprise Web evaluation of any kind, in the US government. The proposal was funded by the NIH Evaluation Set-Aside Program for two years at a cost of US $1.5 million (US $1.275 million for survey licenses for 60 websites at US $18,000 per website; US $225,000 for a project evaluation contractor).Objective The overall project objectives were to assess the value added to the participating NIH websites of using the ACSI online survey, identify any NIH-wide benefits (and limitations) of the ACSI, ascertain any new understanding about the NIH Web presence based on ACSI survey results, and evaluate the effectiveness of a trans-NIH approach to Web evaluation. This was not an experimental study and was not intended to evaluate the ACSI survey methodology, per se, or the impacts of its use on customer satisfaction with NIH websites.Methods The evaluation methodology included baseline pre-project websites profiles; before and after email surveys of participating website teams; interviews with a representative cross-section of website staff; observations of debriefing meetings with website teams; observations at quarterly trans-NIH Web staff meetings and biweekly trans-NIH leadership team meetings; and review and analysis of secondary data.Results Of the original 60 NIH websites signed up, 55 implemented the ACSI survey, 42 generated sufficient data for formal reporting of survey results for their sites, and 51 completed the final project survey. A broad cross-section of websites participated, and a majority reported significant benefits and new knowledge gained from the ACSI survey results. NIH websites as a group scored consistently higher on overall customer satisfaction relative to US government-wide and private sector benchmarks.Conclusions Overall, the enterprise-wide experiment was successful. On the level of individual websites, the project confirmed the value of online customer surveys as a Web evaluation method. The evaluation results indicated that successful use of the ACSI, whether site-by-site or enterprise-wide, depends in large part on strong staff and management support and adequate funding and time for the use of such evaluative methods. In the age of Web-based e-government, a broad commitment to Web evaluation may well be needed. This commitment would help assure that the potential of the Web and other information technologies to improve customer and citizen satisfaction is fully realized.

Veterans' voices: use of the American Customer Satisfaction Index (ACSI) Survey to identify My HealtheVet personal health record users' characteristics, needs, and preferences

“…32 Deployment of the ACSI across multiple federal websites of the National Institutes of Health from September 2004e2006 demonstrated the usefulness of the ACSI as a standardized web evaluation tool for government websites with public audiences. 33 An important advantage of using the ACSI as a tool for program evaluation is that it provides a direct measurement of customer satisfaction which incorporates preapproved Office of Management and Budget (OMB) clearance. OMB approval for public surveys is a requirement of all federal agencies under the Paperwork Reduction Act.…”

A Decade of Veteran Voices: Examining Patient Portal Enhancements Through the Lens of User-Centered Design

“…Since 2007, VA has used the ForeSee customer experience analytics (CXA) survey tool for the direct measurement of customer satisfaction and prioritization of enhancements. The CXA survey is a standardized method of measuring and monitoring customer satisfaction based on the American Customer Satisfaction Index [ 39 ]. The survey methodology uses a psychometric “voice-of-the-customer” technique to assess consumer drivers of satisfaction (look and feel, navigation, site information, site performance, and task processes) and prioritize areas of improvement.…”

Transfer of Information from Personal Health Records: A Survey of Veterans Using My HealtheVet

“…Deployment of the ACSI across multiple federal Web sites of the National Institutes of Health from September 2004 to 2006 demonstrated the usefulness of ACSI as a standardized Web evaluation tool for government Web sites with public audiences. 10 The ACSI survey is available on the My HealtheVet portal through a Web-based pop-up browser window. Using a sampling percentage of 4%, a random sample of veterans using My HealtheVet who had viewed at least four pages within the site was invited to participate in the survey.…”