Fiveskin Posted April 13, 2012 Share Posted April 13, 2012 As paying customers of a product that is currently broken, many of you are mad at EA/Bioware. You should be. They let us down today. However, as an online software developer myself, I'd have you consider the following: 1. Software ain't easy. 2. Online software is even harder. 3. To-date EA has been VERY quick to address our concerns. 4. Bad things happen to good people (see Murphy's Law). If you feel a tl;dr coming on, it's ok to stop reading now. The rest is just more detail to support my assertions above. I Know What I'm Saying I have written, upgraded, and deployed many online enterprise applications whose complexity pales in comparison to that of an MMO, so please try to understand that my assertions come from a place of long and painful experience. The More Moving Parts, The Greater The Chance For Failure SW:TOR is a complex system of systems. I know that sounds odd, but it's true. What we think of as SW:TOR is composed of *at least* the following: 1. A software client (the game you execute on your desktop and play).2. A server (not the physical machine(s) but the software at EA that coordinates the interactions of all of us with the game and with each other).3. A database (what the "server" mentioned above uses to remember what has gone on in the game such as items in your bag, your stats, your toon name, your lockouts, etc...)4. Networking "stuff" such as routers, load balancers, firewalls, switches, fiber routers for database storage. Each of the above can be viewed as a system of it's own. Together they combine to form Voltron...er....SW:TOR as you know it. Each of them are comprised of even more "parts" or sub systems. Each sub system can break. Systems Are Made By People Each of the aforementioned systems undoubtedly has an "owner" at EA/Bioware. By "owner" I mean they are responsible for developing and/or maintaining each sub system in some way. The more people involved in the development or deployment of an online "system" such as SW:TOR (or any online system for that matter), the greater the risk for miss-communication. It may come down to a single person mistyping a single character. You never know. People are fallible and will mess up from time-to-time. The margin for error in such a complex system is very very slim. Judging by previous patches, EA has a solid dev/deployment team. If human error is to blame, it's simply a mistake. Developing An Update Is A Monumental Undertaking Midnight oil was burned. Spouses patience was tested. Microsoft Project was ridden hard and put away wet. I have no doubt that there were all-nighters pulled in order to bring us patch 1.2. There were fights between project managers and developers. Hair was lost. Some people probably had to re-evaluate their decision to go into software development. Considering the time from original release to now, I'd say EA performed a minor miracle with patch 1.2. My point here is that we may want to take the time to appreciate the personal sacrifice of EA devs and their supporting business staff. These guys just ran a 6 month gauntlet that would make a grown man cry. Development Is Not Deployment I've been both a developer and an application hosting engineer. The guy who writes the software isn't usually the guy who puts it on the live server. To deploy is to place the software on the live server. This usually means running some upgrade routine on the "server" software or replacing it. Maybe adding some new physical hardware. Finally, running routines against the database to bring it in line with the new changes installed on the "server" software. The database part is usually the most tricky. Don't ask why. It just is. One slip of the keyboard can ruin the whole thing and cause you to have to do it all over again (I recently did this to a development database and I'll now lose an entire day restoring the backup). Murphy Is Our Co-Pilot Murphy's Law states: Anything that can go wrong will go wrong. Let's assume just for the sake of argument that the Bioware team is infallible. Even a perfect man cannot account for a bad line of code in a piece of supporting software. If your software depends on that software and that software suddenly misbehaves, it could ruin your software without warning. Nobody is to blame. Bad things just happen. What I Think Happened I'm going to go out on a limb here and say that Murphy caused the rollback which is resulting in 8 hours of sudden downtime. Consider if you will: 1. The patch went out this morning as planned. Bioware had no indication that the update failed. 2. They told us right away that the maintenance would take 8 hours, which is the normal maintenance window. This means they are restoring the backup they took last night and rerunning the upgrade routines. 3. Bioware has been so attentive to our needs and bug reports in the past that I just don't buy negligence or incompetence as the cause for the down time. They patch us like once a week in order to keep the amount of changes they deploy at one time very low; thus reducing potential for catastrophe. In Conclusion I don't expect anyone to read this and suddenly not be mad at EA/Bioware. As someone who has gone through catastrophic upgrades himself (a client was once down for 4 days, during which I didn't sleep at all, and we lost millions) through no fault of his own, I just wanted you guys to get a glimpse of Bioware's point of view. They cannot provide that kind of insight themselves, so I figured I'd do it for them. Link to comment Share on other sites More sharing options...
MagicmanNC Posted April 13, 2012 Share Posted April 13, 2012 Great read, thanks for the insight. Link to comment Share on other sites More sharing options...
Fiveskin Posted April 13, 2012 Author Share Posted April 13, 2012 Appreciated! They already gave us a free full day. I remember when I played Lord of The Rings Online and they had severe rollbacks and extended outages. They did NOTHING to compensate players who lost rare boss drops, etc... Like I said, Bioware is being proactive. Software = problems. The difference is in how you respond. So far the response is on par with my expectations. Link to comment Share on other sites More sharing options...
verymediocre Posted April 13, 2012 Share Posted April 13, 2012 Appreciated! They already gave us a free full day. I remember when I played Lord of The Rings Online and they had severe rollbacks and extended outages. They did NOTHING to compensate players who lost rare boss drops, etc... Like I said, Bioware is being proactive. Software = problems. The difference is in how you respond. So far the response is on par with my expectations. That's pretty much the standard way of things in every MMO I've ever played, you get a hollow apology with no information to explain a downtime or rollback (or flimsy information at best) and NO compensation for lost time. Bioware handled things quickly, efficiently, and they gave us a full day in compensation for half that time lost. Well done. Link to comment Share on other sites More sharing options...
KorinHyvek Posted April 13, 2012 Share Posted April 13, 2012 (edited) I agree for the most part. This downtime was unavoidable, and bugs in general don't just magically go away. Their business practices* and game design, you can criticize. But they're doing their best on the software end. *Personally, I think it might have been more prudent to have called these first six months or so a "Final Beta test", probably with a slightly lowered purchase charge for those who bought the game in this time period, like what some Indy Developers have done. Suddenly, all these complaints about "Should have been in at launch" would go away, since the game wouldn't actually have "launched" yet. But then, like I said, that's what Indy developers have done. A major company might not be able to take such an approach for some reason. I don't know what those reasons might be, but Corporations are something I don't pretend to understand. And as far as the 11th hour change goes... It wasn't a money grab. Think rationally people. Within a day of the update, they gave all those level 50s who resubscribed and a whole lot of other people a free 30 days, which costs them more money than they would have made from the "bait and switch". And even if they hadn't, the ill will the event generated would have cost them more money in the long run than the small boost they'd get immediately (and it may well still do just that). A change that late (And it wasn't just a matter of deleting code, Bioware likely had to pull an all-nighter to get 1.2 to us after that decision) almost certainly indicates something disastrous happened behind the scene like a subtle but destructive bug they nearly missed or the like. Edited April 13, 2012 by KorinHyvek Link to comment Share on other sites More sharing options...
Tuleno Posted April 13, 2012 Share Posted April 13, 2012 Nice to see someone posting some sense on here. Nerdrage is funny at first, but gets boring Link to comment Share on other sites More sharing options...
ZaRree Posted April 13, 2012 Share Posted April 13, 2012 Sounds good, think BW has been doing a decent job so far. Link to comment Share on other sites More sharing options...
Jenzali Posted April 13, 2012 Share Posted April 13, 2012 There needs to be a lot more posts like this. No matter how much anyone in the industry tries to explain that MMO's are among the most complicated software systems in existence, there will always be those whiners who think that a change is "just a line of code" or a config thing changed here or there. Nothing is as simple as you imagine it to be, Sir Backseat Programmer. Link to comment Share on other sites More sharing options...
Verg Posted April 13, 2012 Share Posted April 13, 2012 I like this thread, and wish to subscribe to your newsletter. Link to comment Share on other sites More sharing options...
Seraphynn Posted April 13, 2012 Share Posted April 13, 2012 Good post and thank you for taking the time to explain the other side of the fence... Did I miss my game today? Yes I did a lot! Considering I barely got a chance to look yesterday when the patch was deployed. But, I understand things mess up and BioWare are working their asses off to get it back online.. So I am patient.. They also did a goodwill gesture to all players of a free day.. to compensate for this incident. Thank you BioWare I know you will pull this one out of the bag Link to comment Share on other sites More sharing options...
CorellianWannabe Posted April 13, 2012 Share Posted April 13, 2012 You can defend them for breaking their patch the day after releasing it. I can't defend them for declaring that I am not a valued customer and stating that they don't appreciate my support and loyalty. Link to comment Share on other sites More sharing options...
Cadiva Posted April 13, 2012 Share Posted April 13, 2012 (edited) Anyone whose been playing MMOs since the days of EverQuest knows stuff like this happens. I don't understand the whining to be honest. When Dark Age of Camelot launched Shrouded Isles, their first EP, you couldn't actually log into the account servers to register it and that went on for at least a day that I remember, if not longer. It was utter chaos. They also had the whole Prydwen debacle (which they handled superbly imho) when there was much more than a bad patch at fault. The first week WoW launched the servers were not only unplayable, they were offline for days at a time. I think I ended up with something like six days additional time at the end of the first couple of weeks. Mistakes happen, unforeseen problems arise, it's how a company deals with them that decides whether I'm getting irritated or not and so far I haven't seen anything from BioWare to make me irritated. You can defend them for breaking their patch the day after releasing it. I can't defend them for declaring that I am not a valued customer and stating that they don't appreciate my support and loyalty. Then don't, this thread has nothing to do with the additional 30 days subs. That's been given as a thank you for those players who hit level 50 and stuck around when there wasn't a vast amount of content at end level. People who haven't hit 50 yet still have content they're playing with. Edited April 13, 2012 by Cadiva Link to comment Share on other sites More sharing options...
braton Posted April 13, 2012 Share Posted April 13, 2012 Thanks for the read! it was nice to have a thread without people who play this game nerd raging about free time. Most mmos wouldn't even give an explanation to the community and bioware has given people compensation and information on what they are doing. Seriously, how can you have them for this? Link to comment Share on other sites More sharing options...
Fredcat Posted April 13, 2012 Share Posted April 13, 2012 You're exactly right. I also work in IT, for a major retail corporation. End users have no clue how complicated this stuff can be. They are so used to everything running smoothly they think development and infrastructure management are easy. If a database gets corrupted on one of our servers it takes 5 hours to rebuild it. I'm certain the database for an MMO is MUCH larger then the POS (point of sale) databases I maintain. I'm not the slightest bit mad at EA/BioWare because I understand how this stuff works. Testing environments and live ones are not the same. You never know what might happen, the key is always have a way to roll back, which is what they've been doing all day. This stuff happens even to major companies, consumers/end users just don't understand because they don't realize it happens. They just don't get exposed to many IT/Development problems in their every day life. Link to comment Share on other sites More sharing options...
Gazzaho Posted April 13, 2012 Share Posted April 13, 2012 And at the end of the day it all comes down to this. Trash in... Trash out. They put trash in, and they got? Link to comment Share on other sites More sharing options...
Ian_Planeswalker Posted April 13, 2012 Share Posted April 13, 2012 10 million credits for the OP I feel you dude. As a gamer ever since time immemorial when there is a huge patch update things get really messy, it is really a good thing that Bioware have quick responses to problems like these, other MMO's will just tell you that there will be a maintenance and no timeframe. I guess the new generation of gamers nowadays sees it differently from the old school guys. They want everything to be perfect right from the start. Well there is never a perfect MMO, but there are good ones. Just my observation. Link to comment Share on other sites More sharing options...
Roechacca Posted April 13, 2012 Share Posted April 13, 2012 Murphy's Law is the best defense you can come up with? Remind me never to hire you as a lawyer or my PR spokesperson. That's almost as bad as the insanity plea. Link to comment Share on other sites More sharing options...
MaverickXIV Posted April 13, 2012 Share Posted April 13, 2012 As paying customers of a product that is currently brokenYou say that like software ever has a "fixed" state. Software is always broken. No software developer tries to make a "fixed" product. The goal is to make as less of a broken product as possible. Link to comment Share on other sites More sharing options...
Gazzaho Posted April 13, 2012 Share Posted April 13, 2012 (edited) 10 million credits for the OP I feel you dude. As a gamer ever since time immemorial when there is a huge patch update things get really messy, it is really a good thing that Bioware have quick responses to problems like these, other MMO's will just tell you that there will be a maintenance and no timeframe. I guess the new generation of gamers nowadays sees it differently from the old school guys. They want everything to be perfect right from the start. Well there is never a perfect MMO, but there are good ones. Just my observation. I don't get it, as far as I'm aware it wasn't a huge patch that caused today's disaster, it was a small insignificant (compared to 1.2) patch that blew the game apart, something as, presumably, trivial as fixing a delete character problem and a fix for in game ticket support, 1.2 was implemented flawlessly, for me anyway. Edited April 13, 2012 by Gazzaho Link to comment Share on other sites More sharing options...
Dorpire Posted April 13, 2012 Share Posted April 13, 2012 Im actually pleased at how fast they got it back up again> I assume they had a total fail and had to redo/rebuild the whole thing. Link to comment Share on other sites More sharing options...
Superawesomerman Posted April 13, 2012 Share Posted April 13, 2012 Two points: the engine sucks for an mmo. You do not pick an engine which has problems with massive. But this probably is not the fault of the software team. The real beef lies with the design and pr. Ilum was a shameful travesty. they were told their implementation would be terrible. lack of premade 50s on test means poor testing. Poor testing = poor patches. And their pr department, by all appearances, has never played a game in their life. Link to comment Share on other sites More sharing options...
Scudmungus Posted April 13, 2012 Share Posted April 13, 2012 (edited) I care not for excuses. *cracks whip* Bioware - try harder. Always try harder. As a consumer, it's my job to keep kicking them up the whatsit, So they don't slack off. Edited April 13, 2012 by Scudmungus Link to comment Share on other sites More sharing options...
Roechacca Posted April 13, 2012 Share Posted April 13, 2012 They hired the best PvP devs in the business. I heard this once. Can we see some credentials on that statement? Link to comment Share on other sites More sharing options...
Scudmungus Posted April 13, 2012 Share Posted April 13, 2012 They hired the best PvP devs in the business. I heard this once. Can we see some credentials on that statement? They asked the devs, they said they were the best. Link to comment Share on other sites More sharing options...
Roechacca Posted April 13, 2012 Share Posted April 13, 2012 They asked the devs, they said they were the best. well that settles that. It's on the internet so it must be true Link to comment Share on other sites More sharing options...
Recommended Posts