Automated QA Testing at EA: Driven by Events

Abstract

To millions of game geeks, the position of QA (quality assurance) tester at Electronic Arts must seem like a dream job. But from the company's perspective, the overhead associated with QA can look downright frightening, particularly in an era of massively multiplayer games. Hence the appeal of automated QA testing, which has the potential to be faster, more cost-effective, more efficient, and more scalable than manual testing. While automation cannot mimic everything human testers can do, it can be very useful for many types of basic testing. Still, it turns out the transition to automated testing is not nearly as straightforward as it might at first appear. Some of the thorniest challenges are considered here. At EA, Michael Donat is an advocate of automation. His current focus is process improvement on the Player and Business Analysis team. He was previously the manager of QA at Silicon Chalk and ActiveState Corp., and has worked at Microsoft as a software design engineer. Joining the discussion is Jafar Husain, a lead software developer for Netflix. Previously he worked at Microsoft, where one of his tasks involved creating the test environment for the Silverlight development platform. There he was introduced to MVVM (Model View ViewModel); he's a convert, he says, and now likes to spread the gospel of MVVM where applicable. Terry Coatta, a member of the ACM Queue board, brought this group together to discuss the potential for automated QA testing. He and Donat once worked together at Silicon Chalk, where creating a sophisticated test environment was among their challenges. Coatta is now the CTO of Marine Learning Systems, developing a learning management system for marine workers. TERRY COATTA In terms of your efforts so far to apply automated QA testing at EA, I gather you've found the going a little bumpy. MICHAEL DONAT We started the journey thinking automation was a good idea, but then we tried it, and it failed. Still, we figured out what was wrong, and we fixed it. But, while we made it to a nice plateau, we realized there was still a long way to go. Our solution clearly wasn't going to get us everything we wanted-which was a way to broadly apply automated testing. To get there, and for some other reasons, several of us have concluded that what we really need is a new architecture along the lines of MVVM. JAFAR HUSAIN What exactly was your driver for automating in the first place? MD Our primary motivation had to do purely with the cost of manual testing, which has become quite significant given the complexity of our games. Basically, code changes that require us to retest everything manually can be incredibly expensive. By reducing those costs, we felt we would have an opportunity to redirect our testers away from what I call "stability testing"-which is something automation is capable of handling-so they can start focusing more on the authenticity and fun of our game experience. TC In terms of stability testing, what did you see as your first opportunities for automation? MD We started looking at this...

Differentia: Visualizing Incremental Game Design Changes

“…In an ACM Queue article (Coatta, Donat, and Husain 2014), the developers of FIFA Soccer 1 (released in 2009) describe experiments with automation of the QA testing process. They created scripts that would programmatically provide inputs to the game over time to navigate between different screens of the game, verifying that they were still reachable.…”