Benefits of Complementing Eye-Tracking Analysis with Think-Aloud Protocol in a Multilingual Country with High Power Distance

A Comparison of Usability Testing Methods for an E-Commerce Website: A Case Study on a Malaysia Online Gift Shop

“…In the traditional think aloud method, also known as concurrent think aloud (CTA) [18,17], a participant would verbalize each action that they are performing at the point of time. However, it was found that cultural barriers make CTA method less suitable in countries with high power distances per the Hofstede's model [19] such as Malaysia. Sivaji and Ahmad [19] found that although the subjects have been encouraged to think aloud, they are reluctant as they are afraid that a failure of completion of a given task would reflect poorly on their performance.…”

“…However, it was found that cultural barriers make CTA method less suitable in countries with high power distances per the Hofstede's model [19] such as Malaysia. Sivaji and Ahmad [19] found that although the subjects have been encouraged to think aloud, they are reluctant as they are afraid that a failure of completion of a given task would reflect poorly on their performance. This is despite the moderator briefing the user at the beginning of the task that the purpose of the usability testing is to assess the web interface and not the user themselves.…”

“…This method is suitable, as it has been found that Malaysian users prefer to type their thoughts instead of verbalizing them due to the language and cultural barrier of the users [19]. The URANUS system [16] used in this study facilitates the end-to-end usability testing and FCAT.…”

“…Since the website was in English, language barrier also exist [19].Thus, the traditional think aloud method will not provide us with the necessary feedback, which is useful for us to capture usability issues. Hence, [19] has proposed to complement retrospective think aloud (RTA) methods with eye tracking analysis as it is able to reveal key biometric information regardless of the cultural and language barriers. In this situation, a combination of 3 methods has been applied to capture defects or difficulties faced by the subjects to complete the tasks mentioned.…”

“…Using this method, users are prompted by the moderator to talk about what he or she has just done to complete the tasks, which will include any additional comments the users might have. This method is particularly useful for us as it has proven to detect usability defects [17,19] and it enables the users and moderator to observe and verbalize the activities that they have performed. Similar to FCAT, during RTA users have a tendency to forget their train of thought during performing the tasks, which may lead to fewer, incomplete, or reconstructed verbalizations of their thoughts [21].…”