So about that CLOSED beta testing????

[Anzel]

^^ Exactly.

 

The above comments dove-tail very well with my experience managing large software projects over the years.

 

UAT, as a practice, is poorly understood by the consumer.

 

Honest question. Have you clicked on the Dev Tracker? Have you looked in the BUG section? Have you looked in the Customer Service section? Have you actually logged in and seen these issues? Not sure what the SLAs are where you work but a deployment like this would get a PM terminated in almost any other situation.

[Shwarzchild]

I have no doubt that they tested the game, and it's irrelevant if it's public or closed. What I do find shocking is that the end boss is heavily bugged, and the rishi quest is bugged. I would figure those to be major gaffes. I remember last month I received a mouse pad from Prudential, cause those companies love sending out crazy swag all the time, and the mouse pad had the Ravens and Redskins schedules on them. The Ravens schedule was fine, all the dates were matched to their respective games. But, the Redskins schedule showed every single game being played on August 18th. That's a gigantic error for their marketing team, and I'm sure that mouse pad went through the proper editing channels. Things get missed, but that doesn't mean they are any less excusable. To have the end boss so heavily bugged is a big gaffe, and isn't excusable regardless of the type of testing done. It shows a hole in their testing methods that needs to be addressed for the future. With all that being said, at least the creative team who thought up SoR was right on the money, because I have loved this expansion so far...just wish I could finish it.

[LukeSaberRattler]

I don't the mouse pad issue was Prudential's fault most likely the printers fault with a batch of them. You can't ask Prudential to check every single one.

 

That said testing software is hard when you are working with dev environments or stages servers instead of live. Many things on the live server can be different depending on setup. Even the PTS is different than live. The Revan bugs easily may not have been present until they released on the actual live servers.

[Shwarzchild]

I don't the mouse pad issue was Prudential's fault most likely the printers fault with a batch of them. You can't ask Prudential to check every single one.

 

That said testing software is hard when you are working with dev environments or stages servers instead of live. Many things on the live server can be different depending on setup. Even the PTS is different than live. The Revan bugs easily may not have been present until they released on the actual live servers.

 

Valid points. And there's a good chance that this Revan bug was the "printer's error" which is something that Bioware can't control. I think I can accept that. That's a rational explanation. It's hard to believe that they'd have somehow missed the bug when testing. It seems insane to miss that. I feel like it's akin to delivering a car with only 3 of the 4 wheels attached. Any business wouldn't let something off the floor if they didn't think it was complete, and ready to go.

[LordArtemis]

I have to say, as in the past, it is likely they will take care of the current list of bugs in short order. I feel that early access, in this case could be looked at like an open beta....perhaps it gives them a chance to iron out the kinks.

[Anzel]

I have to say, as in the past, it is likely they will take care of the current list of bugs in short order. I feel that early access, in this case could be looked at like an open beta....perhaps it gives them a chance to iron out the kinks.

 

I would tend to agree with you but I really wish that was the expectation that they set. Lots of folks ran to their PCs and pre-ordered SOR thinking they would get one thing but ended up with well... this. Either way, I would have still pre-ordered.

[LordArtemis]

I would tend to agree with you but I really wish that was the expectation that they set. Lots of folks ran to their PCs and pre-ordered SOR thinking they would get one thing but ended up with well... this. Either way, I would have still pre-ordered.

 

Fair enough, and I certainly understand your frustration over bugs.

 

The most frustrating thing to me in the past (I dont know if this happened this time) is when a bug is reported in PTS and still makes it to live.

 

But I would point out that some times they do not have time to repair a bug that has been reported. There are times when it takes quite a while to figure out what is wrong. There are a few bugs in the recent months I can think of that were like this....took them a bit to take care of them.

[Andryah]

Honest question. Have you clicked on the Dev Tracker? Have you looked in the BUG section? Have you looked in the Customer Service section? Have you actually logged in and seen these issues? Not sure what the SLAs are where you work but a deployment like this would get a PM terminated in almost any other situation.

 

Yes I have.

 

But you are deflecting... my comment was specific to UAT and IAT and the fact that most players have no clue what is involved. They see a bug and they conclude testing failed, design failed, yada yada. In point of fact they have no clue.

 

As for SLAs... I don't know where you work.. but I bet it is not a consumer game company. My point? SLAs are not universal across the world of products and services offered by companies, nor should they be. Me personally, I worked in both hardware and software for CAD/CAE as well as Enterprise Computing, as well as consumer video products during my career. The SLAs are radically different, as are the commercially accepted level of bugs, and bug fixes. Stop trying to compare apples to oranges.

Edited December 5, 2014 by Andryah

[Kalfear]

I would invite you to look at the Dev Tracker and/or the Bug section and/or the Customer Service section. Or... go visit Yavin. Have fun

 

just finished Yavin for 2nd time and only bug I encountered (and did a work around so had success anyways despite bug) was in the final fight

 

What exactly am I suppose to be looking for on Yavin (as you suggested to other poster) that somehow shows this expansion as unplayable?

[Andryah]

Fair enough, and I certainly understand your frustration over bugs.

 

The most frustrating thing to me in the past (I dont know if this happened this time) is when a bug is reported in PTS and still makes it to live.

 

But I would point out that some times they do not have time to repair a bug that has been reported. There are times when it takes quite a while to figure out what is wrong. There are a few bugs in the recent months I can think of that were like this....took them a bit to take care of them.

 

Player expectations do not match reality here.

 

If they waited until every bug reported in UAT is fixed before releasing... we would never get any patches or expacs. Not just for this MMO but for all MMOs.

 

The reality is that bugs are a "continuum" in products like MMOs. Heck.. it's a continuum for all consumer software. [i present monthly update patching by Microsoft as exhibit A, and the chain proliferation of graphics driver udates as exhibit B.] Companies and the project teams inside them must balance bugs/bug-fixes/releases in the context of their product and the customer base.

 

The core issue here is that each player has different personal expectations, tolerance, and willingness to adapt (or find work arounds). Multiply this by 1 million.. and you have a dogs breakfast of player sentiment. An MMO company cannot work to a dogs breakfast set of expectations..... so they must work to their own internal design and test gates. And.. we are not flying cruise missles here... we are playing an MMO.

Edited December 5, 2014 by Andryah

[Anzel]

Player expectations do not match reality here.

 

If they waited until every bug reported in UAT is fixed before releasing... we would never get any patches or expacs. Not just for this MMO but for all MMOs.

 

The reality is that bugs are a "continuum" in products like MMOs. Heck.. it's a continuum for all consumer software. [i present monthly update patching by Microsoft as exhibit A, and the chain proliferation of graphics driver udates as exhibit B.] Companies and the project teams inside them must balance bugs/bug-fixes/releases in the context of their product and the customer base.

 

The core issue here is that each player has different personal expectations, tolerance, and willingness to adapt (or find work arounds). Multiply this by 1 million.. and you have a dogs breakfast of player sentiment. An MMO company cannot work to a dogs breakfast set of expectations..... so they must work to their own internal design and test gates. And.. we are not flying cruise missles here... we are playing an MMO.

 

I agree with everything you are saying Andryah but that's just not what we are seeing here. This was a terrible job of testing. I won't Link-Fu this thread to death but Eric and Tait today even acknowledged these problems in todays live stream. Heck... the end mission is completely bugged. I mean come on. Nobody noticed that?

 

In SWG I ran around with the "Master Bounty Hunter" title for like 18 months before it was fixed (I wasn't a Master Bounty Hunter). Did I care? No. Was it a big deal? No. Again, that's not what is happening here. I know that you will disagree and that's OK. But I would still invite you to look at the Deve Tracker and the Bug Tracker and the forums in general. This was a terrible job.

[AmericanNate]

I agree with everything you are saying Andryah but that's just not what we are seeing here. This was a terrible job of testing. I won't Link-Fu this thread to death but Eric and Tait today even acknowledged these problems in todays live stream. Heck... the end mission is completely bugged. I mean come on. Nobody noticed that?

 

In SWG I ran around with the "Master Bounty Hunter" title for like 18 months before it was fixed (I wasn't a Master Bounty Hunter). Did I care? No. Was it a big deal? No. Again, that's not what is happening here. I know that you will disagree and that's OK. But I would still invite you to look at the Deve Tracker and the Bug Tracker and the forums in general. This was a terrible job.

 

No, it wasn't noticed. It worked perfectly on test. That's why. People have already said this.

[Slurmez]

A expansion without a open beta phase is lunacy, pure and simple, fortunately this was not a real expansion and was nothing more than 2x oricon, whats bad about it is that there are still glaring issues and people paid for the privilege of it.

[Slurmez]

No, it wasn't noticed. It worked perfectly on test. That's why. People have already said this.

 

Thats why there is an open beta phase to stress test the whole thing, the mickey mouse process Biofail ran is a joke.

[Anzel]

A expansion without a open beta phase is lunacy, pure and simple, fortunately this was not a real expansion and was nothing more than 2x oricon, whats bad about it is that there are still glaring issues and people paid for the privilege of it.

 

It stinks too because there were numerous requests to help test and they insisted on doing it this way. Whatever the reason, it was a resounding failure.

[MillionsKNives]

Thats why there is an open beta phase to stress test the whole thing, the mickey mouse process Biofail ran is a joke.

 

Have you ever been on the PTS? No amount of traffic it has ever seen could be considered a "stress test".

[JimG]

Yes but see, then they might have had to go to their bosses and say something like "The players didn't like your ideas, Sir."

 

But give the testing to a few "privileged elite" and make them feel smug and superior about it, and you can almost guarantee that most of them won't complain too hard. Then you can go back to your boss and say "Gee Sir, we can't understand why we lost all those subscriptions after stubbornly forcing through some badly thought-out changes proposed by executives who don't actually play the game. After all, the guys in the closed beta loved it!"

 

Bugs have nothing to do with liking BW's ideas. Whoever was doing closed testing should have caught some obvious bugs like the final mission on Yavin or the constantly bugged Jungle Bounty tree. There is absolutely zero reason a closed-beta tester would be fearful or hesitant to inform BW of bugs. However, most beta-testers are usually more interested in early-access than they are in catching bugs.

[theSCARAYone]

Grace Hopper is rolling in her grave somewhere.

 

Ha ha, I get it.

[Ghisallo]

Closed betas miss things because A. smaller population B. When things get shipped to live it is in a different environment so things can work perfectly in closed beta and have issues when pushed live. Though not a Dev I have participated in many closed betas and seen an issue on live that I KNOW did not exist in the beta. Example I don't know how SWTOR is configured but I have played games where the configuration would result in what we see on the last Yavin fight. In Closed beta of another game no one noticed a similar issue because of the small numbers of people doing it at any one time. When it was pushed to Live though there was an issue. Obviously they all share the same processes. These processes were also shared by other encounters. When the larger live population tried to use these processes at the same time it created issues (at times complete server crashes in the case I am thinking off.)

 

So the devs (and/or bean counters) decided what to or next. Some games have open betas. This has two possible downsides. First the open beta (usually) requires a separate Client/game be installed. This can be an issue for many players and this means you still fall short or numbers that in and if themselves can create issues. 2nd the purpose of a beta is to give good, well informed feedback. Open betas really are about a little polish and free advertising. It takes A LOT of effort to weed through the "waaahhhh I hate what you did to my class" and "I told you the story is dumb" to find the feedback that actually helps.

 

That said...guess what..."early release" is essentially open beta that they make money off of. While a good number of players bought by the deadline, more won't. They got money up front in return for those playing early to be their open beta testers.

 

For those BTW that wonder "why did I not get a closed beta invite?!?!? I am just as loyal a player as that guy.!!!!". It is NOT about loyalty. It's about showing the devs you are the guy who falls into the geometry and rather than just typing /stuck first does /loc and then send the coordinates in a bug report. The person who runs the parser and after they fight can send feedback that says " is it intended that Mob Y's Z attack to randomly do 10 times what appears to be normal damage. That person who reads all the dialogue and bug reports spelling errors etc. Even with these people doing closed testing however, see paragraph 1.

Edited December 8, 2014 by Ghisallo

[Pagy]

with any software/application development, defects have severity level.

 

the entire revan fight alone either proves they were willing to launch with high severity defects, or it wasn't even tested period.

 

add in the fact that the game has unplayable ability lag and i question whether or not bw cares at all.

[MillionsKNives]

with any software/application development, defects have severity level.

 

the entire revan fight alone either proves they were willing to launch with high severity defects, or it wasn't even tested period.

 

add in the fact that the game has unplayable ability lag and i question whether or not bw cares at all.

 

Or, as others have stated repeatedly, there wasn't a problem on the test server. But don't let facts ruin your little false dichotomy.

[Pagy]

Or, as others have stated repeatedly, there wasn't a problem on the test server. But don't let facts ruin your little false dichotomy.

so they changed something AFTER testing ended?

 

which is it? or are you suggesting the code started evolving and changing on its own?

[darthscotty]

There are several people in my guild who were in the closed beta and I asked them about the issues that we are seeing. They all said that the issues never came up durring the closed beta. Stuff happens when you put it on a live server with tons more people than were in the closed beta. There is really no way to predict how things will react when it is on a live full servers.

[MillionsKNives]

so they changed something AFTER testing ended?

 

Possibly, or possibly not. It's likely that they still made changes after the end of testing. They had a deadline to meet and bugs to fix, some of which wouldn't have been completed before the end of testing. So it's certainly possible that some post-testing changes could have caused the issue and weren't caught.

 

You're also ignoring the fact that PTS is not live. Any time that you test in a different environment than release it's possible for bugs to not appear until live.

[GadgetDon]

I think the biggest problem was that Sales and Marketing (usually abbreviated to the quite appropriate S&M) absolutely committed the company to a December 2nd launch with no wiggle room whatsoever. So even if all these bugs were being experienced on the test servers (or if some changes had come so late that sufficient testing wasn't able to be done), QA and the devs couldn't have said "wait! stop! you can't launch it's not ready!"

 

Remember, every department at the company except for QA has high incentives to launch. Sales wants the money from people buying the new stuff. Publicity wants all the great reviews to come out, high profile people to look at it. Developers in part want their baby out their and frankly may be sick of looking at the same code so want a break. Community people are tired of "when will it be out". Even we customers push towards early launch. Can you imagine what would've happened on the forums if December 2nd came around and they said "It's not ready yet, no launch this week, we'll let you know when it's ready".

 

Release dates on something as complex as MMO expansions need to be fluid. Decisions on when to flip the switch need to be in the hands of QA, not a calendar. Any company that leaves the decision in the hands of a calendar will have problems - maybe far, far, far worse than we got with this launch.

 

All that said - it won't be zero bugs. Some bugs QA will decide can be lived with (yes, that's part of their job, deciding which bugs are critical and highest priority, which are annoyances that can be dealt with). And you won't hit every bug. I have no trouble believing that the Jungle Bounty bug was a surprise to them, that none of the testers got killed by the beast. But the Revan fight, the only way it could've gone live was making the calendar the authority on when to launch.