Graphically Speaking

Editor: Miguel Encarnação

Game Analytics for Game User Research, Part 1: A Workshop Review and Case Study Magy Seif El-Nasr Northeastern University Heather Desurvire User Behavioristics Research Bardia Aghabeigi and Anders Drachen Northeastern University

T

he emerging field of game user research (GUR) focuses on understanding players and how they interact with games. This is an important area of investment for the game (and interactive entertainment) industry because game companies need to produce games that engage players. Substantial adaptation of user testing methods from outside the game industry has been necessary. Although games are software applications that focus on user experience, they’re often designed not to be as effective and efficient as possible but to be challenging and engaging. The GUR community has thus focused on using, adapting, and extending human-computer interaction (HCI) methods to measure player experience, engagement, and behavior. Researchers have proposed and discussed many methods for this purpose. It has been the subject of a few books, notably Game Usability: Advancing the Player Experience.1 In addition to the standard methods from HCI, the social sciences, and psychology featured in Game Usability, the book Game Analytics: Maximizing the Value of Player Data focused on game analytics, an area that overlaps with GUR.2 Game analytics is gaining powerful momentum in research and industry, notably for persistent, social online games such as World of Warcraft, FarmVille, and Bubble Island. In collaboration with eminent figures in GUR, such as Katherine Isbister, Regina Bernhaupt, Lennart Nacke, and Alessandro Canossa, we held GUR workshops in conjunction with the 2012 ACM SIGCHI Conference on Human Factors in

6

March/April 2013

Computing Systems and the 2011 and 2012 Foundations of Digital Games conferences. We aimed to stimulate discussions between industry and academia to promote the field’s development, consider methods, and outline open problems. In part 1 of this article, we introduce game analytics in GUR and examine the open problems discussed at the workshops. In addition, a case study of a current collaboration between researchers and a game company demonstrates game analytics’ use and benefits.

Analytics in Game User Research Academia and industry have a cache of methods for identifying and understanding the player experience. These methods employ both qualitative and quantitative analysis. The intent is to help designers improve game design and thus the game experience, thereby improving the learning or training outcome, revenue, and so on. The most common GUR methods used by the game industry are think-aloud, RITE (Rapid Iterative Testing and Evaluation), heuristic evaluation, interviews, playtesting, and A/B testing. However, game analytics is emerging as a lucrative area of study and method for supplementing and augmenting existing GUR methods. Figure 1 shows the increased use of “game analytics” as a search term on Google from 2006 to 2012, indicating the increased interest. Game analytics can deliver quantifiable numbers that are easily interpreted, reduce highdimensionality behavioral data to patterns, and

Published by the IEEE Computer Society

0272-1716/13/$31.00 © 2013 IEEE

Figure 1. Google Trends analysis for the search terms “game analytics,” “game metric,” “game data mining,” and “player behavior.” “Game analytics” and “game metrics” are somewhat interchangeable, but “game analytics” is emerging as the term the game industry uses. “Game data mining,” another term the industry uses for applying data mining to analytics data, emerged around 2009.

graphically visualize player behavior. This last benefit can provide direct insight about how players use game assets—for example, when the data is overlaid on game environments (as we’ll show later). Online games often ship unfinished and are continuously refined and redeveloped by employing user behavior data collected online. Such telemetry data provide a precise quantitative measure of behavior that no other GUR method can easily provide. Developers use these data to find software bugs and problems and to further develop and tune the game’s design and system mechanics throughout its lifetime.2 Game user researchers also use behavioral telemetry data during game production to gauge what players are doing, and they supplement that information with other playtesting methods.2 A common approach to analyze player behavior telemetry is to convert the data into a more meaningful set of aggregated metrics, called game metrics.3,4 For example, user login and logoff times provide raw data you can convert to metrics representing the total time a user interacted with the game. Game metrics can then be visualized using a variety of methods, depending on the data and analysis goal. A common visualization is the heatmap. Heatmaps are density maps of specific variables in 2D or 3D space and are thus useful for displaying the spatiotemporal data extracted from avatar movement. Game developers frequently use heatmaps to spot problem areas in level design, notably in multiplayer and massively multiplayer games.5–7 To depict social metrics, developers use other methods, such as network and graph visualizations. In the social domain, one of the most interesting topics is graph layout. Researchers are also developing more dynamic and interactive visual-analytics systems to answer game user researchers’ questions. These systems

often combine standard visualization methods into a toolset, letting researchers (and other stakeholders such as designers) play around with data, investigate hypotheses, and answer questions. For example, Georg Zoeller developed a visualization system for Dragon Age, a major commercial game. To visualize game metric data, he used several standard temporal methods, such as plot-based diagrams, index charts, and stacked graphs. Ben Medler used similar temporal visualization methods in the Data Cracker system for the game Dead Space. Anders Drachen and Alessandro Canossa employed a geographical information system to analyze player behavior in games such as Kane & Lynch and Tomb Raider: Underworld. Magy Seif El-Nasr and her colleagues demonstrated the use of visual-analytics systems for real-time strategy and role-playing games, including Dragon Age. For discussions of these methods, see Game Analytics: Maximizing the Value of Player Data. Game user researchers also often use data-mining algorithms to show user behavior patterns, especially temporal patterns. Game data mining has become central to online business models; companies use the discovered patterns to inform design and make business decisions.

Challenges and Open Problems A conclusion from one of our workshops was that analytics are important because they supplement the results from other GUR methods, such as heuristics, playtesting, and think-aloud. For example, Microsoft, an early adopter of analytics in GUR, developed TRUE (Tracking Real-Time User Experience), a visual-analytics system that uses spatial visualizations of player activity correlated with qualitative data from RITE testing.8 Such a mixed method effectively pinpoints issues with engagement and difficulty and combines the “what” IEEE Computer Graphics and Applications

7

Graphically Speaking

gained from behavioral telemetry with the “why” gained through qualitative feedback. However, one challenge is that researchers don’t yet fully understand what behaviors to collect, how to usefully visualize them for designers, and how to present information quickly enough to enable decisionmaking. Additionally, each method for assessing and evaluating games has pros and cons. Telemetry and analytics, for example, are great resources that can tell researchers what players are doing, but

Our goal was to develop a system that helps designers understand players’ behaviors over time and can follow their trajectories.

8

other data sources. We aim to evaluate and understand how to develop more-effective GUR tools and methods. Since 2011, we’ve been closely collaborating with Blackbird Interactive, which is releasing Hardware, a social real-time strategy game for Facebook. We focused on developing a game analytics system composed of telemetry collection, pattern abstraction, and a visual-analytics system, to help designers understand what players like or dislike, what they have problems with, and so on.

Preliminaries The collaboration started by developing a set of metrics—that is, aggregate measures of player behavior—relevant to the company and designers. In several weekly meetings, the designers told us what questions they want answered regarding evaluation of the effectiveness and appeal of the game’s design. Behaviors of interest included how players progress in the game and how they collect and consume resources. We defined metrics to answer those questions and broke down each metric into telemetry data. Then, we programmed the telemetry component to collect the data. To allow designers to make sense of the data, we needed to use visualization methods and develop pattern or data abstraction methods to scale visualizations from hundreds or thousands of data points, across up to dozens of variables, to something more manageable. That is, we needed to reduce the dataset’s dimensionality and focus on important patterns of player behavior. However, we first had to determine what methods were needed and would be of value, given the game and the designers’ needs. To do this, we compared visualization and reporting methods through prototypes and interviews with Blackbird game designers. Our goal was to develop a system that helps designers understand players’ behaviors over time and can follow their trajectories.

they can’t directly identify the reasons for players’ actions. So, mixing methods is important, as outlined in Game Analytics: Maximizing the Value of Player Data. Therefore, an open problem is how to develop mixed methods that leverage the individual methods’ advantages to let development teams assess a game in a rapid, financially viable manner that provides actionable insights. Another challenge concerns developing ways to let game user researchers use the collected data to make sense of players’ behaviors and emotions and construct a coherent, clear narrative about them. This challenge presents opportunities in areas such as visual-analytics tools for displaying and querying quantitative data. The idea of developing tools that let designers and other stakeholders query and visualize data from players is gaining momentum. This was one of the major models that Zynga pushed forward9 and led to the success of its analytics system and business strategy and model, since copied by an entirely new segment of the game industry. Standardized methods and tools for such a system have yet to be published. However, such a solution could come from any of the plethora of game-analytics-focused middleware providers that have emerged in the past two to three years.

■■

Meeting the Challenges: A Case Study

■■

We at Northeastern University’s Game User Experience and Design Research Lab and Simon Fraser University’s School of Interactive Arts and Technology have been collaborating with the game industry to develop mixed methods to abstract and visualize telemetry data and correlate them with

All the methods we experimented with and implemented used Jaspersoft’s reporting engine. Our point here is to evaluate the visualization’s contri-

March/April 2013

The System To track and visualize player behavior, the system employs temporal graphical reporting tools, such as index charts, stacked graphs, and horizon graphs; and spatial visualization tools using bubble charts, flow maps, choropleth maps, graduated symbol maps, cartograms, and static and dynamic heatmaps.

(a)

(b)

(c)

(d)

Figure 2. Diagrams showing user progress over time: (a) experience points (XP), (b) game level, (c) money in the game’s bank, and (d) army units. These visualizations can be applied to a specific player in the beta-test group or to the entire group.

bution to the design team. So, we don’t discuss the specific software contribution in detail but rather the visualizations best suited for the design team. We integrated the tools into one rich client program and a webserver version (for generating reports for designers, producers, and the marketing team). The system comprises the following subsystems. Data storage. Game telemetry is logged into text files (compressed JSON—JavaScript Object Notation) using the game client. We store the text files in Amazon S3 (Simple Storage Service) cloud servers, then process those files into an Infobright database using an in-house ETL (extract, transform, and load) system. This data includes time stamps for when players start and end a session, the units they purchase, the checkpoints they visit, camera interactions, combat events, social-interaction events, and so on. Because the game is in beta mode, the number of users is limited. However, as it gains users, we’ll need to apply MapReduce techniques, such as Amazon EMR (Elastic MapReduce), Hadoop, and Hive, so that we can easily and efficiently prepare data for metric calculation. We expect to have more than 50,000 users by the project’s end. Data abstraction and metric calculation. To represent the data in a form usable for visualization, we use simple aggregation methods. The methods we use include sum and counting, averages, variance, and groupings based on temporal windows and spatial constraints. Even though these methods are basic, they serve as strong indicators of players’ interac

tions over time and space, which must be defined accurately to answer designers’ concerns about specific user interactions. For example, grid-based matrices can be a good representation of how often players navigate different areas. We associate each area of a map with a grid cell and count the number of times players navigate through those places. We then create a good data model that’s used for visualization methods, such as heatmaps, to answer designers’ questions about how players interact with different game locations. Visualizing and reporting abstract data. We use Jaspersoft’s reporting engine to experiment with ways to visualize data. We connect the selected system with Infobright data middleware, which includes the aggregated data and computed data structures.

Visualizing Player Progress over Time and Space Blackbird’s game design team specifically wanted to visualize how fast beta testers progress through the game, reach new levels, and discover new game spaces. So, we defined metrics that track a specific user’s experience points over time and the corresponding playing levels. We also collected data such as players’ assets (in this game, army units) and their available money (in the game’s bank) at different points of gameplay, especially when their experience points change. Then we made a report module that uses simple time series diagrams to depict these changes for a specific user. Figure 2 shows four metrics, each represented as an index chart, using small multiples. IEEE Computer Graphics and Applications

9

Graphically Speaking

(a)

Figure 2a shows the player experience points over time; Figure 2b shows the level number, which is closely related to experience points. Figure 2c shows the player’s money over time, and Figure 2d shows the number of army units over time. These visualizations can be applied to a specific player in the beta-test group or to the entire group. As the game analysts observed, the number of army units has a high positive correlation with player experience points and level number. Decreases in bank values represent spending money on units and increased army size. So, this set of diagrams shows an important relationship displaying how resource consumption (decreased money) and acquisition (increased army size) can impact a player’s progress. We also developed an animated flow map showing how a beta tester progresses over space and time (see Figure 3). It has these features: ■■

■■ ■■

■■

(b)

(c) Figure 3. The flow map for a user. (a) Time point 90. (b) Time point 239. (c) A zoomed-in section. Green arrows show the direction of the army’s movement. Red circles indicate areas that have been discovered but whose resources haven’t been consumed; gray circles indicate areas whose resources have been consumed. The background map images aren’t actual images of the game, which is still under development, but are visualizations. 10

March/April 2013

A time line represents gameplay minutes; designers can adjust it to depict any moment from the beginning of play. A resolution box represents the time scale. Designers can alter a zoomable spatial map by dragging the mouse to pan and using the mouse wheel to zoom in or out. An animated button simulates the flow map from the beginning and can be stopped at any location.

As the timelines in Figure 3 show, designers or analysts can see a specific user’s progress. The green arrows in the screenshots show the direction of the army’s movement from one game zone to the next. In this case, the player first explored the map’s initial areas and then followed a counterclockwise path. However, the designer wanted players to follow a clockwise path. The user’s behavior was interesting for the designer to see, thus allowing the designer to make decisions about the design. The designers also wanted to understand player behavior in terms of the game’s economic variables. Visualizing this is important, and the flow maps are important for gauging these variables. In Figure 3, red circles indicate sections that have been discovered but whose resources haven’t been consumed; gray circles indicate areas whose resources have been consumed. Figure 3c shows a zoomed-in area of the map, to better investigate how resources have been discovered, missed, or consumed. The flow maps also let the designers focus on specific players with different experience levels and capture their progress. After we showed these visualizations to the designers, they understood better how the beta tes-

ters explored different areas during the first two days of gameplay. As we just described, they also gained insight about whether the exploration of game zones matched the intended narrative and direction. One consequence was that they relocated areas that they had expected players to reach early but that the players never reached because of geographical remoteness.

T

he workshops and our research have led to two main findings. First, visual-analytics software can enable designers to experiment with data and the resulting visualizations to obtain actionable insights. Second, a way to abstract behavioral data and show patterns and anomalies in that data is important to assist designers. Pattern finding requires data-mining techniques supplemented with visualization methods allowing stakeholders to flexibly query and understand the data, thus making insightful conclusions. One lesson echoed at the GUR workshops is that telemetry data alone isn’t sufficient. It must be supplemented and mixed with other data—specifically, qualitative data—to enable a deeper understanding of the context in which the players are playing. So, as we mentioned earlier, an open problem is how to develop mixed-method approaches that correlate telemetry with other types of data across multiple datasets, especially when the game is constantly being revised. Developing such approaches will enrich our understanding of player behavior to enhance game design. In part 2 of this article, we’ll supplement this discussion with one on the qualitative methods used in GUR, thus giving the reader a complete picture of the qualitative and quantitative methods used in the field. We’ll also present a case study of the use of such methods in practice.

Acknowledgments We thank the design and development team at BlackBird Interactive for their collaboration and funding. We also thank the Natural Sciences and Engineering Research Council of Canada and Mitacs for funding the research presented in this article.

References 1. K. Isbister and N. Shaffer, Game Usability: Advancing the Player Experience, CRC Press, 2008. 2. M. Seif El-Nasr, A. Drachen, and A. Canossa, eds., Game Analytics: Maximizing the Value of Player Data, Springer, 2013.

3. L. Mellon, “Applying Metrics Driven Development to MMO Costs and Risks,” white paper, Versant, 2009; www.versant.com/pdf/VSNT_WP_metrics. pdf. 4. L. Mellon et al., “Tuning the Money Funnel: Customer and Process Metrics in Online Games,” panel presentation at ION Conf., 2008; www.slideserve.com/ step/tuning-the-money-funnel. 5. A. Drachen, A. Canossa, and J.R. Sorensen, “Gameplay Metrics in Game User Research: Examples from the Trenches,” Game Analytics: Maximizing the Value of Player Data, M. Seif El-Nasr, A. Drachen, and A. Canossa, eds., Springer, 2013. 6. M. Seif El-Nasr et al., “Visual Analytics Tools: A Lens into Player’s Temporal Progression and Behavior,” Game Analytics: Maximizing the Value of Player Data, M. Seif El-Nasr, A. Drachen, and A. Canossa, eds., Springer, 2013. 7. A. Drachen and M. Schubert, “Spatial Game Analytics,” Game Analytics: Maximizing the Value of Player Data, M. Seif El-Nasr, A. Drachen, and A. Canossa, eds., Springer, 2013. 8. S. Kim et al., “Tracking Real-Time User Experience (TRUE): A Comprehensive Instrumentation Solution for Complex Systems,” Proc. 2008 ACM SIGCHI Conf. Human Factors in Computing Systems (CHI 08), ACM, 2008, pp. 443–452. 9. M. Seif El-Nasr, “Interview with Jim Baer and Dan McCaffrey from Zynga,” Game Analytics: Maximizing the Value of Player Data, M. Seif El-Nasr, A. Drachen, and A. Canossa, eds., Springer, 2013. Magy Seif El-Nasr is an associate professor at Northeastern University’s College of Arts, Media, and Design and College of Computer and Information Sciences, the Director of Game Educational Programs and Research at Northeastern, and the Director of Game Design in the College of Arts, Media, and Design. Contact her at [email protected]. Heather Desurvire is the principal at User Behavioristics Research and an adjunct faculty member in the University of Southern California’s Interactive Media Department. Contact her at [email protected]. Bardia Aghabeigi is a PhD student at Northeastern University’s College of Computer and Information Sciences. Contact him at [email protected]. Anders Drachen is an associate professor at Northeastern University’s College of Arts, Media, and Design and the lead game analyst at Game Analytics. Contact him at [email protected]. Contact department editor Miguel Encarnação at lme@ computer.org. IEEE Computer Graphics and Applications

11

Game analytics for game user research, part 1: a workshop review and case study.

The emerging field of game user research (GUR) investigates interaction between players and games and the surrounding context of play. Game user resea...
907KB Sizes 1 Downloads 3 Views