The user in experimental computer systems research

Abstract

Experimental computer systems research typically ignores the end-user, modeling him, if at all, in overly simple ways. We argue that this (1) results in inadequate performance evaluation of the systems, and (2) ignores opportunities. We summarize our experiences with (a) directly evaluating user satisfaction and (b) incorporating user feedback in four different areas of client/server computing, and use our experiences to motivate principles for that domain. We then generalize (a) and (b) as recommendations for incorporating the user into experimental computer systems research.

Workload Modeling for Computer Systems Performance Evaluation

“…An intriguing possibility made available by user-based modeling is to incorporate feedback: we can create a "smart" model in which users are not oblivious, but rather create additional work in a way that reflects satisfaction with previous results (assuming we can get a handle on the factors leading to satisfaction [181]). In fact, feedback can affect all three models: at the population level, dissatisfied users may elect to leave and cease using the system; at the session level, a session may be truncated short when the system is not responsive, or else it may be extended if the work flows well; and at the activity level, jobs may be delayed until previous ones terminate and their results are examined.…”

“…As a result the response time metric should be replaced by the throughput metric, which further explains why "equivalent conditions" can no longer be interpreted as running exactly the same jobs, but rather as facing the same workload-generation process. We also obtain a more direct (albeit still debatable [181]) assessment of user satisfaction.…”

“…On the contrary, human users are very variable in terms of temperament and abilities, which is reflected in their computer-related behavior [726,181]. Moreover, they may sometimes behave in unpredictable ways [211].…”

Upgrading Mice to Elephants: Effects and End-Point Solutions

“…First, while the common wisdom is that 50 − 100 ms is the lower bound that clients care about; this is still an open research question. Recent results indicate that there is a tremendous variation in user satisfaction in interactive applications [9]. Second, there are scenarios, such as multiplayer games over the Internet, where users would care about improving their performance relative to other players, hence reducing their response time.…”

EmNet: Satisfying The Individual User Through Empathic Home Networks

“…Extensive user studies (e.g. [4,7,3]) have demonstrated in contexts outside of networks that measured user satisfaction with a given control operating point exhibits considerable variation across users. Measuring individual user satisfaction online and then using such measurements directly in the control process makes it possible to exploit this variation to the mutual benefit of users and systems (e.g.…”