Yardstick

Abstract

Online gaming applications entertain hundreds of millions of daily active players and often feature vastly complex architecture. Among online games, Minecraft-like games simulate unique (e.g., modifiable) environments, are virally popular, and are increasingly provided as a service. However, the performance of Minecraft-like services, and in particular their scalability, is not well understood. Moreover, currently no benchmark exists for Minecraft-like games. Addressing this knowledge gap, in this work we design and use the Yardstick benchmark to analyze the performance of Minecraftlike services. Yardstick is based on an operational model that captures salient characteristics of Minecraft-like services. As input workload, Yardstick captures important features, such as the mostpopular maps used within the Minecraft community. Yardstick captures system-and application-level metrics, and derives from them service-level metrics such as frequency of game-updates under scalable workload. We implement Yardstick, and, through real-world experiments in our clusters, we explore the performance and scalability of popular Minecraft-like servers, including the official vanilla server, and the community-developed servers Spigot and Glowstone. Our findings indicate the scalability limits of these servers, that Minecraft-like services are poorly parallelized, and that Glowstone is the least viable option among those tested.

Meterstick: Benchmarking Performance Variability in Cloud and Self-hosted Minecraft-like Games

“…We leverage in this work a common reference architecture for MLGs [47]. As Figure 2 Additionally, many smaller companies o er MLGs as a service; an extensive list appears in our technical report [22].…”

“…Second, players can construct dynamic elements such as simulated constructs, which increase the complexity of the terrain simulation and are discussed in §2.2.2. The impact of player workloads has been previously examined in both the context of traditional video games architectures and MLGs speci cally [35,47].…”

“…These workers emulate players by connecting the MLG and automatically sending player actions based on programmed behavior. Meterstick implements this by using the player emulation from Yardstick [47], an existing MLG benchmark which we compare to Meterstick in detail in §7.…”

“…The canonical example of an MLG is Minecraft, which is already the best-selling game of all time [31]. All MLGs, including Minecraft, present an important challenge to the performance engineering community: although their user-bases can exceed 100 million active users per month, their scalability is limited to only 200-300 players even under very favorable conditions [47]. (MLGs support high concurrency by creating separate replicas of their virtual worlds, essentially sharding state and not allowing crossinstance interaction.)…”

“…Prior work in understanding the performance of MLGs [26,47] and on improving their scalability [15,17,23] forms a valuable contribution to the eld, but does not currently consider explicitly performance variability. Addressing this important gap, we make a four-fold contribution:…”