Interview with BioWare’s Manager of Analytics – Stats and Game Design Decisions

[Olibb]

From time to time we’ve heard representatives from BioWare quote player engagement stats to support particular game design decisions but we've never had much insight into their usage. As players, it sometimes feels that factors weren't being considered, but then I figured that if we could think of it, BioWare should have already considered it too, and thus, rather than write a post speculating on the situation, I decided to ask.

 

Alex Tremblay, Manager of Analytics at BioWare Austin, was kind enough to answer all my questions. The interview (or perhaps more accurately Q&A) is a little long to post here, but here's a snippet:

 

Olib: Can you tell us a little about yourself?

 

Alex: Hi, my name is Alex Tremblay. I’m the Manager of Analytics here at BioWare Austin. I’ve been working on SWTOR since well before the launch of the game. This is the first time I’ve been asked to talk about the game and I have to say, I’m excited. Through data, I feel very, very connected to both our game and players, but rarely do I get to directly interact.

 

Olib: Would you explain a little about the process, how often stats are analysed and who reviews them?

 

Alex: Of course. Our analytics team is comprised of several analysts and a business intelligence (BI) developer. Together, we’re supported by data warehouse (DW), server, and gameplay engineers. Telemetry is captured on a daily basis, transformed and loaded into our primary data warehouse. From this system, we produce dozens of reports and dashboards that act as the BI side of our service. This data is provided to designers, engineers, analysts, and leadership for their own consumption and usage.

 

On top of this BI operation, our analysts, who are embedded directly with the teams they support, provide ad-hoc support in answer to questions that aren’t answered by our static reporting as well as conducting deeper investigations into gameplay features, content, Cartel Market pricing, game exploits, possible game and payment fraud, marketing initiatives, and many others. As a group, we’ll often conduct larger scale analyses where we tear down major aspects of the game to provide guidance and build intuition with our producers and senior leaders.

 

If you're interested, you can see read the full thing here: http://www.swtornetwork.com/news/interview-with-bioware-stats-and-game-design-decisions/

[Master-Nala]

I'm surprised this hasn't gotten more attention.

[Olibb]

I'm surprised this hasn't gotten more attention.

 

Some pretty interesting answers in there, and its great to have the new insight. I think its pretty reassuring to know there are guys that know there stuff in charge of this, whether you agree with everything that gets said / done or not.

[Diviciacus]

I feel like this makes all the pvp balance whiners all the more petty.

[Zoom_VI]

I feel like this makes all the armchair devs all the more petty.

 

Fixed it for you.

Although there are legitimate concerns, but that is a topic for another thread.

 

I also find it amusing how few posts this thread has. Guess the OP doesn't have enough sensationalism and absurdly extreme viewpoints to warrant attention.

[Callaron]

Nice interview! Thanks for sharing it!

[Bobs_YourUncle]

Fixed it for you.

Although there are legitimate concerns, but that is a topic for another thread.

 

I also find it amusing how few posts this thread has. Guess the OP doesn't have enough sensationalism and absurdly extreme viewpoints to warrant attention.

 

its the 3rd time, second thread, I've seen this posted, so many have said their piece elsewhere in thinking.

[Derfburger]

Nice read thanks for posting

[Bobs_YourUncle]

its the 3rd time, second thread, I've seen this posted, so many have said their piece elsewhere in thinking.

 

wait, 2 threads, the other one is zions post about redoing the side quests based on that interview. The third, I got confused with another interview on a podcast about pvp revamp I'm really looking forward to. Its in pvp section. They need to sticky these interviews so everyone gets wind.

[annabethchase]

wait, 2 threads, the other one is zions post about redoing the side quests based on that interview. The third, I got confused with another interview on a podcast about pvp revamp I'm really looking forward to. Its in pvp section. They need to sticky these interviews so everyone gets wind.

 

Agreed 1000 percent. This is more bullet points communicated in a week than we've had all year, other than basic server maintenance type stuff.

[Hudlar]

Great interview! Thanks, Olibb and Alex!

[Tahana]

This looks like it might be an interesting read as i used to be a business analyst. Posting for a later read.

[Drockter]

Fantastic read... thank you... and thank you Alex.

[DarthDymond]

Reposting from the other thread on this:

Just saw that on MassivelyOP and popped over to see if anyone had posted it here. Definitely an interesting read with some neat little tidbits (like the % of players who seem to be using 12x XP to make max-level alts vs those who are using it to see the stories, or the amount of off-the-clock playtime BW employees have put in).

 

There are parts I find enlightening and parts I can't help but question, even though I know their actual job is to consider those same questions. ("We've had an increase of 8% of players doing [A], is that due to [X], [Y] or [Z]? Or to some combination?")

 

For example the F2P Warzone question that came up, where only a tiny number of players were hitting their cap but some questioned how much of that is due to people not even bothering at all because of the cap - I know Alex said they "consider that type of information" but I am still skeptical on how informed a conclusion they could actually reach off of the existing numbers and anecdotal feedback on the forums.

 

I would think they should have test-ballooned the concept a bit, do a "Grand Melee!" Event with a week or two of removing the cap for F2P players - it would have fit well as part of last summer's "we'll roll out all the Events and some Double XP" approach to trying to keep interest up during the normal summer dip (and an abnormal lull in content at that time). It wouldn't have been a perfect representation of how things would look if it was permanently changed, (limited time events are always going to funnel more people to an activity than would normally participate if it was active all the time,) but it would have given useful data to enter into their overall analysis.

Edited May 29, 2015 by DarthDymond

[Draqsko]

I would think they should have test-ballooned the concept a bit, do a "Grand Melee!" Event with a week or two of removing the cap for F2P players - it would have fit well as part of last summer's "we'll roll out all the Events and some Double XP" approach to trying to keep interest up during the normal summer dip (and an abnormal lull in content at that time). It wouldn't have been a perfect representation of how things would look if it was permanently changed, (limited time events are always going to funnel more people to an activity than would normally participate if it was active all the time,) but it would have given useful data to enter into their overall analysis.

 

I would say longer than that, maybe a month long thing. Any type of event would see a spike in activity for a week or two, but longer than that and it'll quickly taper off to close to the real long term effect of it.

[EarnestBunbury]

I feel like this makes all the pvp balance whiners all the more petty.

 

It makes me wonder with all this at their disposal they can't fracking fix Underlurker in less than TWO MONTHS! Petty is seeing a prob and not fixing it because the cartel market is more important!

[DarthDymond]

I would say longer than that, maybe a month long thing. Any type of event would see a spike in activity for a week or two, but longer than that and it'll quickly taper off to close to the real long term effect of it.

Agreed, I was just also thinking of the countervailing issue that if you leave something active for long enough (even if it is advertised as a limited-time deal) then people have a tendency of getting pissed when it gets 'taken away' at the end (credit costs for Skill Training for example).

 

I suspect the end of 12x XP will be running into that issue come this Fall.

Edited May 29, 2015 by DarthDymond

[ZionHalcyon]

Agreed, I was just also thinking of the countervailing issue that if you leave something active for long enough (even if it is advertised as a limited-time deal) then people have a tendency of getting pissed when it gets 'taken away' at the end (credit costs for Skill Training for example).

 

I suspect the end of 12x XP will be running into that issue come this Fall.

 

Indeed, because 12x XP makes a previously unenjoyable experience (If you've done it all before) into an enjoyable one.

 

12x XP is a fix to a longstanding issue - making it temporary is a mistake.

 

Then again, I bet this is just a test run to see if they can make it permanent. They want to gage game balance and community interest with it.

[TheSeventhJedi]

I want to know who the employee who has logged 6000 personal hours is, and why he isn't in charge of more decisions.

[Costello]

The problem with the metrics that have been used is the devs ignore those that aren't convenient and misread those that they want to misrepresent.

 

Take the metric that most players enjoy the class story the most. What was done in response to this. Makeb had no class story while Shadow of Revan and Ziost had one small class story and the over all arc was the same for both factions. So the metric would suggest the continuation of the class story the devs didn't.

 

The player base tell the devs that the sniper class is under performing in PvP. The metrics don't seem to support this so the metrics are seen as exact and nothing will be done to improve this.

 

End game is very lacking with only 10% of player participating so therefore people don't want end game so few resources are devoted to this and we go over a year between raids. Rather than seeing that the reason only 10% of people seem to get there is preferred customers can't play without passes and its not enough fun for people to keep trying to overcome stupid mechanics.

 

While metric will show that events are really popular. Not cause they are well done just that there is so little content even pushing a button 1,000,000 in the nightlife event for a mount thats not that great is at least something to do. So the metric will say more slot machines is good.

 

So while metrics should be good they seem to be so easy to misue and that unfairly gets attributed to the person that generates them rather than the people that misue them.

[ImmortalLowlife]

A VERY great read.... The part that really caught my attention was about the 12XP usage....

 

Less than 10% of our player base seems to be primarily motivated to “get to the end”; the rest choosing to experience the story that exists.

 

I honestly expected this to be a little higher. I mean....I play for story, and generally stop playing characters after finishing Chapt 3. But I kinda figured I'd be in the minority with that play style.

[Bobs_YourUncle]

Personally, I wonder how many of those that have leveled with the 12x only the class stories did not go further because they already had mains that have been through it. Since the story is almost entirely the same. I know I have only bothered to go past the main story on two toons, an imp and a pub and only because the story was from different angles.

 

idk if it has so much to do with end game raiding content, as the lack of variety in story post correllia. Afterall, how many alts do you need to run endgame raids?

 

I really hope they look at this and consider taking the time and resources to make future expansions stories more varied among the classes.

[TUXs]

There are parts I find enlightening and parts I can't help but question, even though I know their actual job is to consider those same questions. ("We've had an increase of 8% of players doing [A], is that due to [X], [Y] or [Z]? Or to some combination?")

 

For example the F2P Warzone question that came up, where only a tiny number of players were hitting their cap but some questioned how much of that is due to people not even bothering at all because of the cap - I know Alex said they "consider that type of information" but I am still skeptical on how informed a conclusion they could actually reach off of the existing numbers and anecdotal feedback on the forums.

 

I agree DarthD. I think a lot of their numbers can be easily misread...for example, here's something he stated that caught my eye:

 

Our regular monitoring of PvE content turned up a substantial discrepancy in usage between the Czerka and other tactical Flashpoints. Working with design and QA, we developed a hypothesis as to the cause, tested and verified it’s veracity, and ultimately increased the reward per time spent to align them more closely with similar content.

Just because they proved a possible reason (rewards is what I believe he's stating), doesn't mean they proved the true reasons.

 

I think they discounted some pretty important factors for Czerka, specifically, the age of the FP. Not running it had nothing to do with the rewards, it had everything to do with newer content being released and FRESH (better) places to see. Czerka has always been terribly easy, boring and uninteresting to me...going somewhere like the Jedi Academy or Koriban though...that screams "STAR WARS" to me.

[TUXs]

A VERY great read.... The part that really caught my attention was about the 12XP usage....

 

I honestly expected this to be a little higher. I mean....I play for story, and generally stop playing characters after finishing Chapt 3. But I kinda figured I'd be in the minority with that play style.

I think some of you may be reading more into that than was intended...I don't believe that means only 10% ever get to the end and get played, I think he means 10% RUSH through the leveling (spacebar spacebar spacebar), most enjoy the story and take their time by watching cut-scenes.

[Vallerine]

This all just makes us a business; a business that needs to remain successful and profitable for us to continue to build new experiences for those playing the games.

 

Maybe if they build new experiences for those playing the games, they will become more successful and profitable?