Highlights

[00:01:16] So what exactly is “games user research?”

[00:03:15] What do you mean by “playtesting?”

[00:08:02] It’s not very hard to find people excited to participate in games user research.

[00:16:51] How do you navigate something as broad as “making a game fun?”

[00:27:48] Steve talks about what it’s like to be on the bleeding edge of games research.

[00:38:17] How the field has changed since Steve first started out.

About our guest

Steve Bromley specializes in establishing new user research teams and in user research for video games. He wrote Building User Research Teams and How to Be A Game User Researcher. He works with mobile studios, indie teams, AAA teams and VR studios to help them run playtests and integrate user research into the production process. Prior to this, he was a lead user researcher for the PlayStation VR headset.

Transcript

[00:00:00] Steve: When we are working with people who are very sensitive to, ‘are you trying to ruin my creative vision? Are you trying to change what I make?’ I try and very clearly explain ‘no, we’re not market research, that’s a different thing. And don’t worry, we won’t try and change your vision, we love your vision! We just want to help you make a great version of it.’

[00:00:34] Erin: Hello everybody. And welcome back to Awkward Silences! Today we're here with Steve Bromley. He's the author of How To Be A Games User Researcher, and fittingly, an expert on games user research. Steve, we're so happy to have you here today!

[00:00:48] Steve: Thank you for having me. I'm really excited to be here today.

[00:00:51] Erin: We've got JH here, too.

[00:00:52] JH: Yeah, Steve, we didn't tell you, but we're actually going to game-ify this episode, Erin mentioned that the scoring system and every answer you get -

[00:00:58] Erin: I'll be Zelda!

[00:00:59] JH: Just messin' around.

[00:01:01] Steve: I went for the high score.

[00:01:04] Erin: I'm going to shoot the moon. Yeah. Awesome. Well, Steve, thanks for being here. And I think let's just start with the beginning on this one. Games user research. What is that? And how did you get into this field?

[00:01:16] Steve: Yes. So, games user research has been, at least I've been a games user researcher for about 10 years now. As you can imagine, when people are making games there's a whole bunch of design decisions that are made at every level, but there's fundamental ones about what are the mechanics of the game, or how is the game going to work to those, those detailed implementation details?

How do I design this level? What should the menu look like and what character abilities should exist? All of those design decisions are made by level designers, artists, producers, all these other disciplines, but can be supported by user research, by running play tests and user research and usability sessions.

We can help understand what are the impact of these decisions on our users and help tweak them so that they're matching the design intent of the game behind it. What's really nice about the field of games user research is that playtesting is very common in games development. It's a thing that's been part of how games have been made for a long time.

And because of that, there is a history of recognizing that you are making design decisions and putting them in front of people. As a user researcher, you can bring some structure to that and some UX best practice. And so it's a really nice environment for practicing UX and user research.

[00:02:32] JH: And just to clarify on my side, when you say games, do you mean all games, like board games, video games, like what, how do you define games? I guess that would be my first question.

[00:02:41] Steve: Yeah. Great point. So my own experience is with video games, both console games, VR games, mobile games, that type of thing. And I think because of that tech background, that's where it's more common, but looking broader at playtesting. Playtesting is very common in board games.

Although, I think I haven't encountered a formal UX or user research process for board games yet, perhaps that's a growth area for the future.

[00:03:05] JH: And when we say playtesting, that is literally just like, Hey, I have a version of this game. These are the rules that I think are the rules. Let's play a couple of rounds. And then you realize like, okay, some of these rules got to change.

Is that basically what that means?

[00:03:15] Steve: Yes. So in the development of games, that is very common, that idea of let's just put it in front of people and see what they like or see what they can do, and can't do. Obviously coming from a user research background, we recognize that that's very dangerous. There's a whole bunch of nuances about understanding if people like things what's difficult and challenging that you want to apply appropriate methods, appropriate caveats and appropriate understanding too. So it's nice that designers are doing this anyway. But you as a user researcher have to bring in a degree of expertise and formalize that process beyond what teams are just going to do by themselves.

But ultimately, playtesting is: put some people in front of it and see what they can understand, see what they can do. See what lands for them.

[00:03:59] Erin: Yeah. And that's common in user research, right, where the idea of researching this thing, it's bringing—bringing that history into a more formalized, mixed method, kind of discipline for the modern context.

I'm thinking of the example you mentioned of just like play testing where you imagine, I don't know, like if anyone read Calvin and Hobbes, but like Calvin ball or. I know my kids will make up games all the time. And there is that playtesting process of like, can we actually play this game? Can people understand the rules?

It makes a lot of sense that that act of understanding if this game is any good is something that has always happened with games, whether they be board games or digital games or otherwise.

[00:04:40] Steve: Yeah. There's such a close link in games between the player experience and the success of the game.

That it's just really important to playtestings. And as you said even people making games at home recognize that that's a pretty valuable part of what you're doing. Yeah.

[00:04:55] JH: And when you're doing user research on games, is there an element that's almost a, I guess my mind goes to being similar to like physical products where, if you're designing a shoe, at some point, you have to say, the shoe is done, we're going to manufacture thousands and thousands of these things and we're going to sell them. And if it turns out there's a problem with it, we're kind of stuck. Is games kind of that sort of dynamic of like, okay, the game is done, we're going to ship it out now to a bunch of people. Whereas, I think in user research and most like technical products, web-based apps, you can kind of iterate and be like, oh, we messed that up. I'll just ship a change. Like where do games kind of fall on that spectrum?

[00:05:27] Steve: Yeah. That is changing over time, but it's very much part of how games user research used to be. And to some extent, one of the challenges that you have against you as a researcher is that the launch date is extremely important for the success of a game. Modern games these days, they're spending as much on their marketing budget as, as they are on, on that development budget. And so hitting that launch window is really important for their success because they've spent a whole bunch to make sure that everyone's gonna buy it that Christmas holiday.

[00:05:58] Erin: Yes. Like a blockbuster movie.

I mean, how much does a top marquee game earn in terms of like comparison to a top movie? It's gotta be ballpark, same, same ball game, right?

[00:06:09] Steve: Yeah. I don't have the specifics, but you do regularly see stats that say video games industry is as big as Hollywood or bigger than Hollywood. So it's just as important.

And like with movies. Yeah. That launch window is critical for the success of it. There are games that have come back from a disappointing launch and over a number of years, iterated on it, evolve to get that experience that they promised at the beginning, but that's not regular. That's not the standard way that many games work.

[00:06:43] Erin: So that's pretty different than what you hear in lots of SAS or other forms of digital user research, where it's all about iterative launches and just put it out there AB tests and we'll keep getting better, very different.

[00:06:55] Steve: Yeah, and it does introduce quite a few methodological challenges as well for us as user researchers, because of that importance to the launch and the marketing spend it means that secrecy is really important for doing it.

And a lot of the methods that you would apply as a user researcher and other tech where we'll launch it. We'll do some remote testing. We'll do some unmoderated testing. The industry just isn't comfortable with that. It's not comfortable with people having copies of the game at home where it could potentially be leaked and then disrupt that launch.

One thing that you see in games user research that you don't see a huge amount in other types of user research is huge multi seat usability labs as a consequence. So you can do Quanta studies, but you're bringing 40 players into your environment where you can control what they see and what they're exposed to, and crucially, what they can take away in a way that you couldn't do if you were letting them play at home yet, still answer some of those quantum questions that our teams would have.

[00:07:53] Erin: I'm thinking there are a lot of people who would probably like to be participants in that research. Is it hard to recruit for games user research?

[00:08:02] Steve: No, it's really one of the positives as well is just how excited everyone is.

Well, definitely the staff who work there, a lot of people who work in games could potentially be earning more if they were working in other forms of tech, but are there because they love games and are really enthusiastic about designing and making games. And so the people working there are really enthusiastic, but you also see that reflected in the research participants.

Whereas if you're working for a bank or an insurance company, bringing in a participant, it's, they're there for the money, it's not the most exciting meeting they've had that day, perhaps. In contrast, when you're testing a game from a studio that you love and it's unannounced, you've no idea what they're working on.

That's probably the best thing that participant has done that month or that year, it's very exciting for them. Which is obviously great for them, but that does create those risks of, we've gotta be very careful that they don't go and tell everyone about what they saw and also has a significant influence on if you're trying to measure things like enjoyment or some of these emotional things that your teams might be interested in capturing. That's very difficult to separate the experience of the actual game itself, from the experience of coming to the studio and having a great day playing games that you didn't - isn't a typical day for you.

[00:09:25] JH: So when we talk about participants, we're always, are they the right participants?

And I'd imagine for games, it seems like maybe a big kind of dichotomy that you need to make sure you're covering your bases on are, like you have gamers, people who really love games and know lots about them. But then I also imagine you want people who are less into gaming to be open to some of these things, right?

Like I think when you see some of the most successful games is because they've had wide adoption from the population at large, it became like a phenomenon. So, how do you make sure that you're getting people who can like, give you both ends in the sense of, okay. As an experienced gamer, we were familiar with some of these patterns and this game's really interesting versus like I'm a novice and I was able to approach this and, are people trying to balance both of those things or does it depend on the game?

[00:10:05] Steve: A really good point. Yes. I think it is very important to get both those experiences. Obviously what we're most interested in recruiting based on how they're going to purchase the game. So you can do a lot of segmentation based on what other games have they bought before and what have they played before and how far have they got in those games?

Which is a good indication of, are they likely to be the type of person who plays this game? And they will actually say that doesn't mean that we should exclude people who are new to the genre or would potentially buy that game, but haven't played similar games before. Because games are exciting and people want to be involved.

You do get quite a few people who are participants, misrepresenting their experience. I guess you get that in any type of user research who just want to be there to take part of it. They aren't necessarily the type of player who would really buy this for real. And so we do have to be very careful with screening based on things they've played before, their purchase behavior, to make sure that this is a realistic player and we are getting both the experienced players' behavior, but also when relevant for our research objectives, the experience of new players who this is the first time they played a game in the genre for example.

[00:11:20] JH: Gotcha.

The other quick thing that comes to mind with participants for games research is that kids like games. So I'd imagine that you're also often trying to talk to people who are under 18 and I'm curious how you navigate that aspect of it.

[00:11:32] Steve: Yeah. And I don't know, you've talked about in previous podcasts that is challenging.

So we always find it easier to work with the parents and we'll recruit the parents to at least physically be there while the child is play-testing. That helps to some extent, because obviously the safeguarding issues are lessened because their parents are there. But also does create a challenging environment for you as the moderator to make sure that the parent doesn't disrupt the playtester artificially and that they're entertained and not bored, or don't fall asleep just because they have to watch their kid playing a game. Kids in general, as I know you know, are challenging playtest participants because of that high degree of honesty. They will tell you if they don't like it, they will refuse to play anymore. If that's realistic behavior and that's probably valid feedback, but also because their behavior can be quite unpredictable, as you can imagine.

[00:12:32] Erin: Yeah, that honesty does seem like an asset though.

[00:12:35] Steve: Yes. I've had kids who have come in to play games and they've been super excited because the idea of coming into play games is great. So, they're super excited for the first 10 minutes. And then they realize, oh, this isn't a game I play at home.

I like Mario Kart. I'd much rather be playing Mario Kart. Why do I have to play this game I've never heard of? Sulking or wanting to leave, all these things, again, really valid feedback, but difficult to moderate.

[00:13:05] JH: Yeah. Yeah. It seems telling now that maybe something needs to change in that game, man.

All right. Quick awkward interruption here. It's fun to talk about user research, but what's really fun is doing user research and we wanna help.

[00:13:18] Erin: We want to help you so much that we have created a special place. It's called userinterviews.com/awkward for you to get your first three participants free.

[00:13:29] JH: We all know we should be talking to users more. So we've went ahead and removed as many barriers as possible. It's going to be easy. It's going to be quick. You're going to love it. So get over there and check it out.

[00:13:37] Erin: And then when you're done with that, go on over to your favorite podcasting app and leave us a review.

So you're specialized in games user research. So how is it different than other kinds of user research? Because as we talk about this, I'm thinking about, we talked about the kind of movie analogy and games are entertainment, right. But they're also interactive in a way that movies are perhaps not.

What does it look like to do games user research? What are the similarities and differences between say, a consumer digital product or business to business, digital products, some of the other sectors where you see a lot of user research happening.

[00:14:20] Steve: One of the first challenges that I think engineers, researchers encounter when they come from other industries is the mindset of the people we're working with.

Because people who work in games often do it for a passion project, like with movies. They have the mindset that games are creativity and that they're artwork. And how can artwork be compatible with this idea of user research or usability testing? Because you wouldn't usability test the Mona Lisa, if you were painting that. That is obviously a challenge versus usual researchers and does have quite a big impact on how we work. So it's simple enough for usability testing for us to change how we frame the value of usability testing, rather than saying, we're going to change your creative vision or improve your creative vision, we can describe usability testing as you have a creative vision, but some of that's been lost in interpretation between what you are creating and how the player's experiencing it. And so we're trying to remove those barriers between what you think you're putting into the world and actually, what a player is experiencing from it.

And through that way, you can make the case for usability testing, some of the other types of user research that you would find in other types of texts, such as understanding user needs or that discovery work. It can be more challenging to make the case for, because it does seem like it conflicts with the idea of creativity and putting artwork into the world.

I do find though that differs based on the type of games. So an emerging genre that has been bigger over the last five or 10 years is mobile games and free to play games. And often those teams come from a background of working in software development and have been exposed to these ideas from other industries.

And so we'll have a different mindset to someone who is working on a small indie game. That is their creative vision that they're putting into the world.

[00:16:15] JH: That makes a lot of sense. Even when you get people to buy in, though. And you're like, let's do some of this playtesting and other stuff. How do you know which issues are real issues?

Cause I guess the thing that comes to mind for me is kind of like that concept of flow state, where if something is too easy, people get bored and give up. If something's too challenging, people get frustrated and give up. And so some of these things in the games, right? Like as you're going through and you're discovering what's possible, or how to beat some challenge or level, you kind of want it in that middle zone.

Right. Like, it can't be too easy, but it can't be too hard. So if somebody does get stuck, how do you unpack, are you motivated to go figure it out? Are you going to give up or like, how do you work through that dynamic?

[00:16:51] Steve: Yeah, that's a great point. So one of the things that. I've learned in my own career about what's important for having those discussions is encouraging our design teams to articulate their design decisions.

Without that pushing often people can just say, I want my game to be fun, or I want it to be the right difficulty. I don't want it to be too easy or too hard. And so getting them to articulate what is too easy and what is too hard to turn that into something that you can measure. And then you can design a study around how many times they fail on a level, how, what are the challenges in the game are things that you can then measure, or that actually, you also said you can measure that based on the players experience, you can ask players to rate the ease or difficulty of each level or each encounter using things like Likert scales. You can, you can capture that and then see by correlating that to what the design vision was, is just the thing you're trying to make.

[00:17:45] Erin: And is this something that designers, usually, do they know the answers to these things? Like on a scale of one to 10, I want it to be easier or harder or why might they want that?

[00:17:57] Steve: I've found that traditionally, no, it's not a thing that designers have a thought about. You get the view that I'll know it when I see it.

So I'll just put it into the world and then I'll tell based on the player reactions how it does. I find at least with a relationship, some of the game teams I've worked with, that's a starting point that you have to build from, if the teams aren't familiar with articulating difficulty or their design decision, or the intent behind it by going through some early play tests and starting to expose them to this, the type of data you get from back from play tests, and this is the value of defining what you're expecting from players beforehand, you can build that relationship over time, which I think is important.

[00:18:44] JH: Does some of that too come from, like from the vision of the game, you mentioned kind of like some people are more indie, kind of very artistic makers versus other people probably going for mass market appeal, like the phone games and things like that.

Like if some of it that you can kind of unpack it from there, like, Hey, like what kind of game do you want this to be? Like, you want everybody playing it? Or do you want the people who are playing it to be obsessed with it and devote followers or I'm not sure if there's anything to that kind of framing.

[00:19:08] Steve: Yeah. I think that's a really nice idea. And I think teams do that kind of work at the beginning to say, what are the core pillars of this experience? What are we trying to create here and what do we want it to be like? And so as a researcher, that's a thing that you can take and adapt to try and formalize it and make it measurable.

Although some of these things are like, is the game fun? Is the game the right difficulty? Although we can try and measure it. And it's a very common research objective. There's always a gray area between how we've measured something, but have we really measured that the core part of experience that is relevant to the game design is a continual challenge I think for us.

[00:19:46] Erin: Yeah. And I imagine you know, sort of specializing in this and working with lots of different clients, you know, have that experience to bring to every situation. Are there certain benchmarks or heuristics that apply sort of game to game that can make it a little bit easier to think through what is enough friction or not enough, or how to quantify fun and things like that?

[00:20:15] Steve: Yeah. I think that definitely does exist. I know big studios, big publishers, like Microsoft for example, they have all these benchmarks from other games that they've tested before. So you can compare the scores or the difficulty to other games to try and draw a conclusion that can be quite challenging because not only are they, what makes her into trying to be different, but also via the scores, you've got to be radically different at different development stages.

So even the same game later in development is a lot more complete and a lot more representative than it is earlier on. And so it's hard to benchmark a game against itself or a game against other, other things, but that ethos and it's something the industry is trying to do definitely exists.

[00:21:03] JH: During this testing phase, is there also an element of trying to figure out if players can find exploits or ways to take advantage of the game? I feel like that's something you typically see in the gaming world, people who find glitches or different rules or incentives that can be manipulated in some way that was unintended.

Like, are the developers trying to stop that stuff out at this stage as well? Or is that a different kind of approach altogether?

[00:21:27] Steve: I think that's definitely a thing that designers are interested in. Often from the type of play tests that I personally run, which are small scale qualitative studies or small quant studies of 10 to 50 users, you might not see a huge amount of that behavior.

It's not until you hit a thousand or the millions of people playing that that really becomes clear of what, because they've crowdsourced that knowledge. What are those exploits are all those things that ruined the game by making you super wealthy and being able to buy everything else in the game, for example. They often would pick that up very late in development when they run a mass beta test, which is part of the process where they will do a soft launch for some games and have a thousand people play it, or 10,000 people play it mainly to highlight bugs. But also it does reveal these exploits if you're measuring the right type of things.

But usually that's a bit further down the line with them when our play tests. And gave to you to research knowledge, to apply applied on the design of the core mechanics, and that side of it.

[00:22:30] Erin: So, what are some of the phases for user research in a gaming context? Is it the same that you would expect in other applications?

Or is it a little bit different, for example, what does evaluative research look like or discovery research, or is there a different lingo?

[00:22:45] Steve: Yeah, the lingo isn't the same. Games has its own lingo it uses for development, I think. Firstly, they decide to prototype something and their prototypes are trying to find that core part of the fun of the game.

What is the fun bit, even in just five minutes, that if you replicated this core loop again and again, you could build a fun game around? And that's one of the most challenging times I find for a user researcher because it is the least formalized. Teams aren't that clear in what they're looking for, because what you're trying to do is just work, find a fun thing to do.

And it's often more of a toy at that point to see this is a fun thing. And that is where that informal play testing becomes part of how they currently work getting peer feedback and just trying to find out what is the core loop of this game.

[00:23:36] Erin: So in research that I'm more used to, right, with discovery, you're trying to understand, what is the problem we're trying to solve, right? With a game it seems like, well, the problem is I want something fun to do. And so when you're talking about the prototype stage, again, in other kinds of research we're looking at, okay, have we kind of sort of started to solve this problem, but here we're looking for where is what I'm hearing you say, where is there - where is the fun centered and what we've shown someone and where can we like dig into and try to create more of that fun stuff? But how did you get to that prototype in the first place?

Is it just the creative genius with the idea, or how do you get to that point of having that prototype?

[00:24:20] Steve: That's a really good point. And exactly as you've described a lot of that discovery, understanding user needs and setting up problems and using that to identify opportunities, I haven't found that apply to games in the same way.

Because the user needs it — ultimately the person wants to be entertained. They want to have a fun time. There probably is some contextual user needs about what's the environment in which they're playing in, for example. But a lot of that fundamental research isn't isn't relevant in this case, I find currently the game industry does rely a lot on the idea of a creative genius, someone coming in with ideas and trying a number of different things and seeing what lands and being inspired.

Seeing players play it. But ultimately just coming up with ideas, I wonder if in the future there's a lot more potential for user research to go further back into that space. We talked earlier about one of the challenges for games user researchers is: games are an artistic medium. And how do you, how do you apply that kind of discovery research on an artistic medium?

But I don't think it's a hopeless case. I think with further work and further thought you could think about how to apply discovery research to video games, largely through a lot of the user research that does exist for video games is in that second diamond of the double diamond. We know what we're trying to make.

Let's explore a whole bunch of different ways to make it, and let's iterate on those ways to come to the correct solution.

[00:25:50] JH: On that front half, is that an area where the entertainment industry, like TV and movies, is maybe a little bit ahead or there's things to learn from? Cause I feel like you kind of see both things happening there where some movie will come out and it's purely just someone's passion project.

And it's incredible when you're like, I would've never thought that this story would be interesting, but this person made it and it's great. But I think you also start to see studios and companies do stuff around this. Like, hey, there haven't been a lot of stories about this type of profession or this type of background.

Like we should figure out how to bring content to life in that area, because there's gaps in the market. Like, is there stuff in that area that like the games research side could learn from? Or I don't know that world well, so I'm guessing a ton here. Just curious to hear your thoughts.

[00:26:34] Steve: Yeah. I think that that probably does happen that kind of market research of working out what are some popular genres, what are some popular themes for games and which genres and themes are underserved. And do we think if we made a game of this type, it would be successful? I think that definitely does exist.

I don't know your perspective as a researcher, but I often try to distinguish what we do as user researchers from that type of inspirational market research, particularly when we are working with people who are very sensitive to, are you trying to ruin my creative vision? Are you trying to change what I make?

I try to very clearly explain. I know we're not market research. That's a different thing and don't worry. We won't try to change your vision. We love your vision. We just want to help you make a great version of it.

[00:27:19] JH: Yeah, I think you're right. I think I inadvertently described market research as I was working through that point, but that makes sense.

[00:27:24] Erin: Let's talk about some applications of user research for games. You got any cool stories, some interesting research that you've done? Research I think can be so theoretical and academic and when we did the research, we learned this and then built a better product is kind of where the magic is.

So yeah. Any, any fun stories of games user research applied?

[00:27:48] Steve: Yeah, some of the work that I've worked on in the past and found particularly interesting was around virtual reality. So, before I did this by myself, I was with the team at PlayStation, I was a user researcher in PlayStation's internal team. And about five years ago, they were working on their first virtual reality headset.

The - at the time, there were no commercially available virtual reality headsets. So it's before Oculus Rift and it's before the Valve one. And, that meant it was, there's a huge amount of really interesting research challenges that were just completely unexplored. We were working for example, very closely with game teams about how do you, how do people navigate a game world in virtual reality? And how would you tell them to pick things up? And how would you describe these fundamental interactions in a way that's understandable and usable? And because it was an entirely untapped ground, that's such a broad research question.

That's really exciting. You do a whole bunch of prototyping looking at different ways of moving, looking at different ways of interacting with things that hadn't been done before. It also led to putting on old methods. And again, answering, describing new methods that hadn't been used before, either, for example.

So one of the big deals around virtual reality about five years ago was fears around motion sickness. And I know big studios like Ubisoft and Activision and another one too, were looking at virtual reality. We're doing quite a lot of work to work out, how do we avoid triggering motion sickness? How do we measure motion sickness? All these types of questions and how I looked at it myself were, and with the team I was working with, we were aware that work had been done by the army back in the 1990s about training helicopter pilots in virtual reality environments. And so we had to go back to that type of research, look at their methods, pull them out, try and adapt them.

Trying to apply, how were the army doing it in the nineties to a commercial setting. Again, just new methods that hadn't been used before, designing new interactions, a whole bunch of things that as a researcher are really interesting because no one has done this work before.

[00:30:02] JH: When you're doing that, is there a way to prototype those interactions or explore them in like cheaper ways or is it some of it, like you have to build some of the technology to see if people can pick things up in VR using this type of pattern or like what, what are some of those ways that you actually get into that learning?

[00:30:19] Steve: Yeah. So a lot of it does require software development. At least how we approached it was just software development, but the software development doesn't have to be that onerous. There's a lot of game engines that make, actually creating ugly versions of how it would work, but testable versions, not particularly difficult.

So you can work very closely with game teams. And we did work very closely with game teams to prototype a wide range of different interaction methods and then put them in front of players. So yeah, it's at least from a software side, it's not impossible. Hardware, obviously a lot more expensive and a lot more difficult. But, yeah, a lot of the challenges we were looking at are in the actual software implementation of these things.

[00:30:59] Erin: Yeah, it feels like researchers talk about mixed methods all the time and kind of having this toolkit of methods you can use for different jobs. You're trying to do your research and this idea of kind of going into the military archives to find method inspiration is so interesting.

Like how would you even think of that? Yeah, just sort of expanding your method arsenal to include — arsenal's a military term — I guess, to find these adjacent methods that wouldn't come to mind obviously, but in fact are so appropriate for this new kind of thing you're trying to learn.

How do you go about finding that stuff?

[00:31:40] Steve: Yeah, I think as with any research, we probably started with Google. Typing it into Google and seeing what exists. I also think, perhaps we're lucky in virtual reality, that is an area that has had academic interests. And so there's a lot of theses people have published.

And so if you are looking for journals or academic publications, both those direct publications, but also their references are a trail that you can follow to see what work has been done in this space before, and then build from that, to do your own work in the future. But yeah, I imagine a lot of it is luck based on what you stumbled upon it and what you've missed.

[00:32:17] JH: On the methodology side, when you were talking about the multi-seat playtesting I realized that I don't actually know that I fully know what that means. Is it purely just that there are multiple people playing the game simultaneously? Are they playing the game together or does that vary?

Like what does that actually look like when, when you get people together to do that?

[00:32:34] Steve: Yeah. Good question. So, as you can imagine, a lot of the type of questions you want to answer are quant questions. We want to measure something, or we want to get some ratings, things that you would do through a survey or a quant method normally, but because of the secrecy, that's not available to us in games user research sometimes.

So, a setup they have in a lot of big publishers, like PlayStation, like Microsoft, like Ubisoft and Activision, is they will have a big user research lab, which has 40 or 50 gaming pods, which are set up with a TV, a console, probably a computer where you can fill out a survey and then they just bring people in to play the game.

That can be short sections of the game. If you're testing the usability of a very specific interaction or very specific thing, or for those longer play sessions, you might have people come in over a week so that you can track how that experience changes over time. Can the players complete the later levels, which obviously they've had to see the early levels to have learned the things to be able to do until you can be sat with 40 players for a week or more looking at those types of things through surveys.

When they stick up their hand and say they're stuck going and doing some direct observation, but I guess you, as an individual researcher, can't watch 40 people intently. So you have to rely on them flagging when there are usability problems or issues with the game.

[00:34:03] Erin: What about matching different methods to different devices or modalities?

You're doing a game that's meant for mobile versus something played on a TV or a computer or most games now multi-platform. I know there are a lot of, there's a robust mobile game environment, but how would you think about testing for those different things?

[00:34:23] Steve: That's a really good point.

And again, I think as researchers, there's definitely more we can be doing in this space. So, I want one usability lab, as in any type of user research environment is a very traditional method that you might be applying in games user research, and you can bring people in and get them to play a mobile phone game, for example, or a console game.

And these rooms are often set up to look like a typical living room. So it's trying to create that console or PC gaming experience. One of the downsides, I think, is that in real life people, especially with mobile games, people's contexts, isn't that quiet usability lab. It isn't even their living room.

People play mobile games on the train when being distracted by other things or in an environment with a higher level of background noise. And I don't know if we're great at recreating that. Obviously you can do things like letting players go into the wild, doing diary studies and doing something remote and unmoderated so that they can go and play in a more realistic environment, but sometimes those non-disclosure agreements and secrecy points stop you from doing that.

But yeah, there's definitely some caveats that we need to be thinking more about, about how the context and the actual device they're using to play it on affect their experience?

[00:35:44] Erin: I wonder, given the challenges with remote testing with the secrecy and the NDAs and all that. With COVID right, that's been something folks have relied on quite a bit, is remote testing. Is that something the gaming industry was able to adapt to, or?

[00:36:01] Steve: It's been a massive challenge. I think a lot more than in typical technology because of the secrecy. I know some people have tried to look at opening their labs in a COVID safe way.

So very isolated pods and making sure they didn't come in contact with anyone else during their play session. There's a good blog post by the user research agency called Player Research where they were adapting their lab to be more COVID secure, but at a certain point during the pandemic, even that hasn't been possible, just leaving your house has not been recommended.

So coming to any sort of lab isn't suitable.

[00:36:34] Erin: So what happened? Did the games ship without research or has game usage ticked up with a pandemic, or, what's that...?

[00:36:43] Steve: I think gaming just has ticked up and I've spoken to a number of researchers about different methods, they tried to overcome this remote challenge.

There is some technology out there that allows you to stream a game. So it's feasible that even if the game is on a server somewhere in a game environment, in the game studio, you can play at home. So that solves some of the technical issues, but still those secrecy and those NDA issues means that it's just not possible for some games or some setups to do that.

Other studios I've talked to have just relied a lot more heavily on expert reviews or internal play tests which are nowhere near as good, even, even the best user researcher, they're not going to find half the issues that you would in front of a real player. And so I think it has suffered over the pandemic, our ability to do this type of user research.

[00:37:35] JH: Erin, when you asked the original question about different types of devices and this and that, I don't know why my mind just went to like the Nintendo Wii and just thinking about user research in that, just watching people throw the remote into the TV screen were like, "we should put a leash on these things."

[00:37:47] Steve: Yeah. I think all those safety warnings they have at the beginning are probably a reaction to that.

[00:37:51] JH: Yeah. Nintendo must get up to some cool stuff. They're always messing with new interaction options and stuff like that. Yeah.

[00:37:57] Erin: Bowling on the Wii is so fun.

[00:37:59] JH: Yes. It was very fun when that came out. Yeah.

[00:38:02] Erin: Steve, how has the field changed?

You've been doing this for a minute and games have certainly changed over the years, or have they? Maybe they've always, the core tenant of what makes a good game, maybe they haven't changed as much, but the games themselves have, but where do you see this field going?

[00:38:17] Steve: I think there have been big differences in what's been the average game created over the last decade, so that pushes towards free to play. That idea of: the game itself is free, but you can either pay for upgrades or pay not to have to wait for things has been extremely popular. It's a way that game designers have learned how to monetize their games beyond just getting 40 quid from someone when you sell the game in a store to perhaps over a year or two, we can make a hundred pounds from that same person.

And because of that the type of research objectives have changed significantly as well. Not just "is this a fun game," but also "how much retention? Are people going to still be playing this in six months or a year?" That has had some impact on the type of, obviously research objectives we get brought, but also how we want to tackle those research questions.

And it can be very difficult to measure retention or to work out, is this game going to stick compared to the methods that we were using a decade ago, where we're just putting someone in front of it for an hour. And from that, we'll learn a whole bunch about their behavior and their opinions and whether this game is working or not.

I can see that changing further. So one of the impacts of that has been the maturity of data science and games, and the idea of "let's do some analytics and let's do some machine learning to predict the people's future behavior." That's personally not my background or my expertise, but I can see both working a lot closer with people with that background and expertise and doing joint work together, becoming much more important. A significant change I can see coming in the future for games as well is Xbox, and I don't know if PlayStation has announced something similar, but they're moving towards the idea of a Game Pass, which is very similar to the idea of what Netflix does. So rather than buying a specific game, you pay $10 a month and you have access to a library of games that they have picked.

Again, that's going to have an impact on how those studios who make the games get paid. Presumably they're going to be paid by how long people stick around playing their game versus someone else's game, and from that, that's going to have an impact on the actual game design and what they are making.

They're going to start making games that aim to be played for long periods of time, rather than creating a short experience in the way that they might have previously been able to, and again--

[00:40:45] Erin: Maybe dialing up that friction to its upper limit.

[00:40:50] Steve: Yeah, exactly. All those things about balancing, working out is this the right kind of difficulty?

Are we creating an experience that plays — we'll stay in that flow state — becomes really important in a way that perhaps it hasn't been in the past.

[00:41:08] JH: It feels like some of the stuff you just mentioned, the monetization where you can buy upgrades and do these different things, or, "Hey, you're buying a game pass, so we really need you to be playing the game a lot so that you don't churn."

Does it feel like it's pushing it a little bit more on like a business slash commercial direction? Whereas I know earlier in the conversation you were mentioning just how many of the original game creators were really artistic, visionary people? Is that, is that kind of changing or is there still a space for the vision, the art of this is how I want the game play to go, to have to fit into that model?

[00:41:38] Steve: I think they definitely do exist. There are lots of teams out there who are independent developers doing it because of their love of the craft and, and trying to create specific experiences. I know indie developers find it very challenging though, because a lot of the environment is changing.

So that is encouraging these types of non-creative decisions about how do we just make an experience people will keep playing for a long time? And so I can see in the industry, that's very challenging for developers and scientists to work in. As a researcher, I see currently we support both of these teams and these audiences, both those teams who are creating free to play games and trying to optimize their monetization and have their type of retention questions, that definitely exists. And also is often where a lot of the, the money to do these types of big research studies is, but that doesn't mean that we aren't also helping, and can't also help those smaller indie developers, those people who they don't have a huge budget, but they get huge value from a simple usability test or just seeing people play their game and that kind of qualitative feedback.

The thing I'm really interested in the future in my career is working out: how can we support those indie teams? Those who don't have that money to run these big studies, but how do we give them some of the tools and things that they can do to take advantage of games, user research, and best practice?

[00:43:08] JH: I feel like we have got to ask, what are some of your favorite games?

[00:43:12] Steve: My favorite games. One of my favorite games is Spelunky. I don't know if either of you are familiar with it. It's an indie game. I've started playing the sequel recently. You are someone who is in a cave—you're going into a cave and you're looking for treasure in the cave and it's a 2D platformer.

And it's extremely difficult. You die in the first five seconds, but you learn something. And the next time you die after six seconds and you learn something more, and the next time you're in your next run, you'll last 10 seconds. And I really liked that loop of: "every time I die it's because I made a silly mistake.

I learned why I made that silly mistake, and I'm not going to do it next time." And that you see your own individual growth. Which is really nice, but it's also very challenging. So I love that.

[00:44:00] Erin: How do you play it? What do you play it on?

[00:44:01] Steve: Oh, yes, I play it on the Switch and in general, I love the Switch currently.

It's because you can pick it up and put it down. You don't need to hog the TV. And it does instantly resume from where you were. It allows me to fit in little five minute bits of gaming, whereas on a proper console, you have to sit down for an hour, perhaps you won't have that type of free time in your life.

So I really love the Switch cause it does fit with my life and, and just picking up and putting it down. Are either of you playing anything currently?

[00:44:33] JH: I've had periods of my life, where I played tons of games. But I've been on a bit of a hiatus as I've gotten older and started a family and everything, but growing up my brother and I would play all sorts of different games, which was, which was a lot of fun.

[00:44:45] Steve: You should give the Switch a go. It does  work for parents, because you can just put it down when necessary.

[00:44:51] JH: Yeah, my wife and I were talking about that, getting one and trying to go through Zelda together or something. Cause we both liked that game. So yeah.

[00:44:56] Erin: Yeah. And Zelda's — Zelda's my husband's favorite for sure. Breath of the Wild. And then we've got, our eight year old is starting — what's the one everyone's playing with the animals and they barter and buy things.

[00:45:08] JH: Animal Crossing?

[00:45:09] Erin: Yes. That one, that's been popular. And then the little one does like Osmo, which is kind of like learning educational games. A little bit adjacent to what we've been talking about, but I imagine trying to follow some of the same tenants, keeping it engaging.

[00:45:25] Steve: Yeah. I think that's a really interesting space as well. Not one I know a huge amount about, but combining only making a fun experience with "are we achieving our learning objectives?" Again, interesting research questions for us to tackle.

[00:45:38] Erin: Steve, thanks so much for joining us. Great to have you on.

[00:45:41] Steve: Thank you for having me. It's been a lovely chat.

[00:45:43] JH: Yes. It was a ton of fun. Thank you.

[00:45:45] Steve: Thank you!