How do you make decisions when you get feedback like that, about whether there's a greater purpose in terms of the decision you're making for the game design versus "is that what the community wants?"
CC: You have to all input from the community in the lens of "Is it better for the game?" We probably are too close to it at times to appreciate how radically it's changed because, with such things, like Brian was saying, like movement speed and jump height, we actually tweak a bunch and iterate on it and tweak it some more; so we don't get that perspective of "Hey! I just played a game of Halo 3 and now I'm playing a game of Reach beta, and it feels totally different!"
When all of these people start saying, "Hey, this does not feel like Halo to me!" then we talk about it and see other ways that we can adjust it but also still fit that in the framework of what we are trying to do with the game and then play that.
What we've found with movement speed is that it actually helped us solve a few other issues that we were struggling with. So you just have to tweak and figure out what's appropriate.
BJ: But really, at the end of the day, the team here is ultimately going to make the final call, whether it's Sage [Merrill] on the weapon and sandbox side or Chris and his team for the game types and the core multiplayer stuff; these generally are people that are building on the same foundation that was laid down ten years ago.
It's their jobs to make these gut decisions and gut calls, and I think ultimately this boils down to trying to wade through all the noise.
Realize that the people that you're probably hearing the most from really represent a very small subset of your overall population but are probably the most hardcore people that you actually care a lot about; so how do you balance their needs against the needs of the broader population who maybe doesn't care or might even feel the opposite?
In this example, talking about run speed, one of the easy things for us is that we offer a whole host of customizable game settings in the final game. So if you and your friends would prefer to play something that feels more like Halo 3, you'll have the ability to do that through custom game settings. So some of those things we can actually mitigate by, again, empowering players to custom-tailor their own game experience to their own liking.
Do you guys use a lot of the data you get from gameplay? I've talked to people who sift through server data and get reports, and it's my understanding that sometimes things are counter-intuitive in terms of seeing how players actually are reacting compared to what they're saying.
CC: Yeah, the big numbers that we were using from the beta were things like: Invasion Slayer, Elites are winning 60 percent of the time, and they're winning by actually a pretty substantial margin -- the final score of those games. So that's an example of something where we weren't sure if Elites were dominating so much in Invasion Slayer because of their sandbox, their access to the weapons, their spawn being closer to some of the territories than Spartans.
So, the second week of the beta, we actually came out with a 4v4 (4 Spartans versus 4 Elites) game type on Sword Base and Powerhouse that was just purely for science, is the best way to think about that, to see if those numbers were really consistent. Sure enough, after about 60,000 games, we saw that, again, Elites were winning about 60 percent of the time by a really high margin, so that's something we completely react to.
BJ: An example I can think of off the top of my head of something where the anecdotal feedback conflicted with the data that we had was that, at the outset, a lot of people were kind of vocally spread. They thought that the magnum, the pistol in the Reach beta, was fairly useless; it was inaccurate; "This thing is a piece of garbage. Why would I ever want to use it?"
We didn't really believe that. We heard people starting to say that -- yet, at the end of the beta, of all the new weapons, the pistol ended up being the weapon that generated the most kills globally, throughout the entire population.
So I think by week 2 most people ended up understanding that this is not the same as the Halo 3 pistol; you actually have to play with it differently. It requires a little different, nuanced approach to it in the rhythm of firing and how you waited how you choose to use it.
I think that, as a result, we tried to educate our player base on how they might want to think about it differently, and by the end of the beta it ended up being the weapon they got more kills with than anything else.
If we had reacted simply to forum posts, who knows? Maybe we would have done something bad and actually changed, but I think the team stayed true to their gut. We thought the pistol was exactly what it needed to be -- for the most part; we might give a minor tweak here or there -- but the data backed it up and showed us that the majority of people were in fact using the pistol to great effect and it was working as we had intended.