Your technology needs help... lots of help... lots and LOTS of help. :(

[sambeta]

Actually, the question should not be directed to the HeroEngine guys. Bioware bought an unfinished version of the HeroEngine with swaths of undocumented code. Bioware would have subsequently heavily modified the engine to suit their needs. What the HeroEngine is today is NOT what Bioware bought.

 

From: http://www.heroengine.com/2011/11/heroengine-meets-starwars/

 

Thanks for that, very informative read.

[dprijadi]

You're wrong. Completely. Totally. Someone learned iOS and... what, you're writing real-time apps via push notifications?

 

Network I/O strategy is not a 'low level' engine component that you just swap out with a new, shiny, speedier solutoin. It is an all-inclusive business-wide effort, from the design decisions, to asset creation, to the IT (and any CDN in this case if it's even there) capabilities, to the deployment scripts, to the the coder discipline to operate within the set limitations. It's not just something you download and plug-in.

 

Complete tomfoolery to think that network i/o is the job of the engine team only.

 

you definitely never do any game programming, maybe you just know a little of this and a little of that and think you know everything. Nope , your original post clearly state your lack of understanding of game design process and game development process.

 

Sorry friend, to real game programmers your posts are nonsense. your comment on iOS developer using push notification proves that you dont understand anything about multiplayer game design on iOS platform. Stop before you embarrass yourself further with your lack of knowledge.

 

you , sir , are a fake

[dprijadi]

Actually, the question should not be directed to the HeroEngine guys. Bioware bought an unfinished version of the HeroEngine with swaths of undocumented code. Bioware would have subsequently heavily modified the engine to suit their needs. What the HeroEngine is today is NOT what Bioware bought.

 

From: http://www.heroengine.com/2011/11/heroengine-meets-starwars/

 

yep, thats a pretty big risk for bioware if they use their own dev / engineer to modify hero engine. now there is engine source code divergence. though we dont know the details of which system's source code is modified by bioware engineer and which is taken intact from heroengine unmodified.

 

this will be hell for their engineer maintaining different engine codebase , assuming they modify engine sourcecode extensively.. i know their pain.. i'v been there

[JeramieCrowe]

TL;CR Breakdown:

 

Problems

• Technical needs are lowest priority, which causes the following:
  
• Network I/O is abusive Send/read bloat.
  
• Network I/O may have blocking operations. Send/Read bloat.
  
• Client has blocking operations. Read bloat.
  
• Client loading screen isn't always away from the play state. Read bloat.
  
• Client physics engine is authoritative regarding movement. Send bloat.
  

 

Solutions

• Don't let marketing run the show... they never have any idea what they are doing. It's Star Wars. It markets itself for God's sake.
  
• Set aside time to fix technical debt.
  
• Prioritize performance issues over all else until things stabilize, primarily, the network I/O and the asset loading strategy.
  
• Reduce your poly count on the client.
  
• Back off any 3D engine features for a bit. You're not going to outshine where Xbox was three years ago and it's an MMO. You can get away with some crappy graphics.
  
• Devs, leads, PMs, and the CTO needs more spinal fortitude to push back against other department priorities instead of being driven to cutting corners and create more technical debt.
  

 

THIS IS WHERE TL;CR ENDS. YOU CAN NOW GO BACK TO YOUR 140 CHARACTER POEMS.

 

So, some breakdown on the numbers of why I think that their synchronous network I/O and syncrhonous asset loading on the client is cause of all lag in SW:TOR. I'm comparing WoW (which creates the illusion of collective areas seemlessly) with SW:TOR (which uses isolated instances, the more traditional MQ room compartmentalization)

 

The tool I'm using is SmartSniff on Windows 7.

 

A few observations:

 

• SW:TOR push down packets according to area of sight. (AOS) Stand near a crowd and you get more, stand away from a crowd, you get less. This active push prioritization would lead to false positives in QA environments simply because they could not properly recreate an environment that would tax the discerning algorithm.
  
• WoW appears to not care about where you are or what you are doing, no matter where you are, no matter how many people exist in your AOS, which is interesting, especially considering the dramatically less amount of traffic you send and get from them. WoW data is also surprisingly uniform.
  
• The physics engine in the SW:TOR client appears to be authoritative regarding movement. This means that if I jump, the client broadcasts data every single video frame until I land on the ground again. They appear to be trying to run an FPS setup that typically uses UDP on a massive scale using TCP. This is proof positive that SW:TOR network i/o strat is not good and was poorly conceived.
  
• They both use TCP.
  
• SW:TOR is pulling down data the entire time during any loading screen. WoW only pulls them down at the 90% of its initial loading screen. In many cases, the loading screens are just overlays, not actual states separate from the play state.
  

 

So, without further ado, here are the numbers.

 

SW:TOR Data usage:

 

Standing in a populated area (Combat Training area) of a single instance of the Imperial Fleet for 10 seconds with 152 people in the instance:

 

Packets: 110

Data Sent: 426 bytes

Data Received: 4,877 Bytes

 

Standing in a sparse area (Far north of the Supplies area) of a single instance of the Imperial Fleet for 10 seconds with 152 people in the instance:

 

Packets: 36

Data Sent: 244 Bytes

Data Received: 1,783 Bytes

 

Standing in an impossible to populate instance (your ship):

 

Packets: 16

Data Sent: 166 Bytes

Data Received: 857 Bytes

 

Jumping.

 

Packets: 12 packets

Data Sent: 307 Bytes

Data Received: 854 Bytes

 

WoW Data usage:

 

Standing in a populated area (near the AH) of the collective instance of Stormwind for 10 seconds with hundreds of people in the zone:

 

Packets: 36

Data Sent: 42 Bytes

Data Received: 1,536 Bytes

 

Standing in a sparse area (far north @ Wollerton Steed) of a single instance of Stormwind for 10 seconds with hundreds of people in the zone:

 

Packets: 33 packets

Data Sent: 191 Bytes (I jumped.. oops)

Data Received: 1,611 Bytes

 

Standing in an impossible to populate instance (Stockades in my own group with only me in it):

 

Packets: 27

Data Sent: 56 bytes

Data Received: 1,190 bytes

 

Jumping.

 

Packets: 5

Data Sent: 163 bytes

Data Received: 463 bytes

 

To read the original scathing rant that kick started this whole shebang, click on the SHOW button! Free bunnies!

 

 

 

The Original Post

 

Former game developer here. (No gigs with Lucas, BioWare, or the other companies associated with this... although I did contract for someone who did help with the marketing push lulz) I left the industry because too many non-gamers were entering it. Some thing about the dream of accruing abusive amounts of money from a single title seems to attract the parasites, the wolves, and the incompetent.

 

I like your game. I really do. The focus on the single-player aspect is amazing. The voice acting is from top notch talent (I personally know many of the voice actors in the game, actually) and it is delivered as well as it can be given the -extreme- amount of disconnected copy that had to be read. My kudos to your producers. I'm sure many, many long hours were spent on that aspect of the game alone.

 

Let's move on to the problem. Notice I said “problem” and not “problems”. You only suffer from one problem and all of these ailments stem from it.

 

The Client

 

I've only researched the client packet i/o. I'm assuming you purchased licenses for the 3D engine and that the lead devs on the project had a solid streak of 3D game programming on their resumes.. but only a few of them them had any experience with the engine itself. This shouldn't come as a shock, as there are many, many 3D engines on the market and having one dev knowing them all would price that dev right out of the market entirely. The logic behind my guess is that if the entire dev team was deeply intimate with the engine, then they would have been able to successfully push back against the art and marketing department's insistence on using high poly count models everywhere.

 

Yes, the creativity of these departments are paramount because it's the pretty pixels us consumers froth over. However, if this factor is the foundation of developing a game (It often is the most frequent goto by artists, marketers, and MBAs alike) then it will become obvious in the final product. That being said, your leadership has long crossed over the “golden ratio” between pretty pixels and performance. It is obvious BioWare's entire objective regarding client development is to wage a holy crusade against GPUs everywhere and crush them. It's one thing to write Crysis. It's another to write something that blows up a GPU like Crysis and have absolutely none of the visual effects.

 

I'm sure the politics in your company, the silverback chest-thumping, and the e-peen flopfests all take turns stifling any attempt to rebalance this pixel-vs-performance ratio. (SSDD) When the flaws of your product are broadcasted to the world that the devs are not included in any actual decision making regarding priority or direction, it's time to fire your CTO for being spineless and probably all of his reports.

 

The Servers

 

Your networking infrastructure, however, is not as easily purchasable as an out-of-the-box 3D engine that you just drop in and start hacking away at. It is woefully taxed. I have nothing but pity on your back-end and IT department for one reason: your asset loading strategy is completely untenable in relation to your resource demand.

 

These loading screens between “zones” (See: MQ channels/rooms/subscriptions) are reminiscent to installing an operating system from a pack of floppy drives. You do know that when broadcasting client data to other clients, that shouldn't issue a blocking resource pull from the HD -every- -single- -time-, right?

 

Right?

 

I mean, I like the little pause when on a PvP server because it lets me know that I'm about to be ganked. But I know why that pause is happening. And I know that when you tested it on your local machines that were well-tuned for this sort of testing, and on staging server setups that didn't have even 1% of the load of a live environment, this issue never cropped up.

 

It happens to everyone.

 

Don't fire off blocking calls as a response to time-sensitive events that are pushed down from the server. Your client will fall apart because those events are random. Have a default placeholder asset primed in memory and ready to instantiate until the rest of the local resources (model, textures, etc) are pulled asynchronously.. Spinning plates, man. Spinning plates.

 

This blocking call is only part of the problem. I am almost positive the vast majority of these load screens are caused by the servers trying to poll live information regarding other connections... and that you've made this a blocking call as well. Network I/O is, literally, the most expensive time operation you could perform and you've made a mass polling operation a blocking call. Going from a planet to a warzone to planet is an experience similar to participating in hurdling... except each hurdle is a two hundred foot tall stone wall. This is because you've relied on blocking calls and no one caught it.

 

And no one's going to fix it. It's very easy to undo and yet... it will never be fixed. Why...?

 

...and back to your organizational problems we go.

 

Somewhere, an investor is having ******s (I can say "******s", right? American's aren't still squeamish about that word, are they? Does that put me on a terrorist list or something?) about the revenue and ROI. Yay, capitalism! All is well... except your growth is in jeopardy and both you and I know it. Anyone watching this pure marketing-driven priority of pushing out pointless doodads and trinkets instead of solving fundamental performance problems can spot it a mile away. You have too many non-devs leading the effort.

 

You will not survive your own success because your leadership will not take the ten minutes required to have someone explain to them that people won't play a game where 30% of the time is spent at loading screens due to a failure to prioritize and solve an extremely common problem in an industry-standard fashion. What they will listen to is some non-tech failed actress say that little cute tauntauns will solve these problems. B*tches love little cute tauntauns. (The * is for 'o')

 

Say it with me, BioWare:

 

YOU DO NOT HAVE A CONTENT PROBLEM. YOU HAVE A PERFORMANCE PROBLEM.

YOU DO NOT HAVE A CONTENT PROBLEM. YOU HAVE A PERFORMANCE PROBLEM.

YOU DO NOT HAVE A CONTENT PROBLEM. YOU HAVE A PERFORMANCE PROBLEM.

YOU DO NOT HAVE A CONTENT PROBLEM. YOU HAVE A PERFORMANCE PROBLEM.

YOU DO NOT HAVE A CONTENT PROBLEM. YOU HAVE A PERFORMANCE PROBLEM.

YOU DO NOT HAVE A CONTENT PROBLEM. YOU HAVE A PERFORMANCE PROBLEM.

 

Say that in your next stand-up, loud and proud. E-mail it every fifteen minutes to the CEO. You can fix these problems. It's solvable. You just have to spill some blood and not be afraid of it.

 

 

 

Lots of great technical data (all of which I understand) but I'm getting almost none of this on my end...

[GlowstickSwinger]

I can't refute anything you said technically, so I'll just attack your character because I read your posts in a way that made me fill my diapers.

 

Coming from the guy who still thinks network i/o strategy is one macports install away.

Edited April 28, 2012 by GlowstickSwinger

[GlowstickSwinger]

Lots of great technical data (all of which I understand) but I'm getting almost none of this on my end...

 

Then that confirms my other theory that I didn't want to get into: Distribution and data center geostrat. What numbers are you getting and what region of the world are you in?

[JeramieCrowe]

Then that confirms my other theory that I didn't want to get into: Distribution and data center geostrat. What numbers are you getting and what region of the world are you in?

 

Midwest US, East Coast server. My numbers are about half yours. Also, you only say "populated part of Fleet". How many is that, on average?

[GlowstickSwinger]

Midwest US, East Coast server. My numbers are about half yours. Also, you only say "populated part of Fleet". How many is that, on average?

 

About 20-30 people in the area when I tested. I'm on a West Coast server, coming from the West Coast. You even had half the data sent? That would be incredibly odd o____O

Edited April 28, 2012 by GlowstickSwinger

[ironix]

Then that confirms my other theory that I didn't want to get into: Distribution and data center geostrat. What numbers are you getting and what region of the world are you in?

 

I tend to view the "I'm having no issues" posts with an element of doubt.

 

I would highly suggest any of the people who say "loading times are fine" to go queue up for PvP from a planet like Corellia or Aldaraan and then come back and say that everything is fine. Perhaps these people don't notice these particular issues because they load the planet once, go about their questing and then eventually leave. Anyone who enjoys PvPing has their lists of planets from which they should never queue for PvP.

 

And before anyone says "get a better computer", I'm running 16GB of DDR3 2000MHz RAM, 4 120GB SSD drives in RAID 0, two ATI 4870x2's in SLI on an Intel i7 running at 3.2GHz. Even with this setup, I absolutely HATE certain planets for their insane loading times.

 

EDIT: And to top all this off, SW:TOR still hammers my video card's GPUs such that I need to manually increase the cooling fan speed to avoid 90ºC+ temperatures.

Edited April 28, 2012 by ironix

[JeramieCrowe]

About 20-30 people in the area when I tested. I'm on a West Coast server, coming from the West Coast. You even had half the data sent? That would be incredibly odd o____O

 

Now, I didn't say that. Of course, I haven't collated any data, as I haven't had a need to. I've never experienced server lag, except for when my ISP was hiccuping, but I have enough networking experience to know it was me. If you want some collated data, you'll have to give me some time...

Edited April 28, 2012 by JeramieCrowe

[JeramieCrowe]

I tend to view the "I'm having no issues" posts with an element of doubt.

 

I would highly suggest any of the people who say "loading times are fine" to go queue up for PvP from a planet like Corellia or Aldaraan and then come back and say that everything is fine. Perhaps these people don't notice these particular issues because they load the planet once, go about their questing and then eventually leave. Anyone who enjoys PvPing has their lists of planets from which they should never queue for PvP.

 

And before anyone says "get a better computer", I'm running 16GB for DDR3 2000MHz RAM, 4 120GB SSD drives in RAID 0, two ATI 4870x2's in SLI on an Intel i7 running at 3.2GHz. Even with this setup, I absolutely HATE certain planets for their insane loading times.

 

EDIT: And to top all this off, SW:TOR still hammers my video card's GPUs such that I need to manually increase the cooling fan speed to avoid 90ºC+ temperatures.

 

I won't say "get a better computer", but anyone that watches the numbers can tell you that you are loading a helluva bunch more data than most MMOs per planet/area because there IS a whole lot more in a planet/area than most other MMOs.

[GlowstickSwinger]

I tend to view the "I'm having no issues" posts with an element of doubt.

 

I would highly suggest any of the people who say "loading times are fine" to go queue up for PvP from a planet like Corellia or Aldaraan and then come back and say that everything is fine. Perhaps these people don't notice these particular issues because they load the planet once, go about their questing and then eventually leave. Anyone who enjoys PvPing has their lists of planets from which they should never queue for PvP.

 

And before anyone says "get a better computer", I'm running 16GB for DDR3 2000MHz RAM, 4 120GB SSD drives in RAID 0, two ATI 4870x2's in SLI on an Intel i7 running at 3.2GHz. Even with this setup, I absolutely HATE certain planets for their insane loading times.

 

EDIT: And to top all this off, SW:TOR still hammers my video card's GPUs such that I need to manually increase the cooling fan speed to avoid 90ºC+ temperatures.

 

You're not the only one with a beast box that I've seen have issues.

 

And bamf...!

 

http://hewiki.heroengine.com/wiki/Network_Optimization

 

I'ma read me some source and post what I find when I see it.

[GlowstickSwinger]

Now, I didn't say that. Of course, I haven't collated any data, as I haven't had a need to. I've never experienced server lag, except for when my ISP was hiccuping, but I have enough networking experience to know it was me. If you want some collated data, you'll have to give me some time...

 

Thanks anyways, I'ma read the HE wiki on the networking setup. That'll give me a much better understanding of what is going on.

[strictnine]

what do you consider "the best companies in the industry"? im truly curious

 

If you can't take my word for it, that is your problem.

 

I'm not trying to impress anyone here, I'm just saying that a lot of people are overly eager to criticize developers when they have no idea what actually goes on in game development. And the team dynamic is different everywhere you go.

Edited April 28, 2012 by strictnine

[GlowstickSwinger]

This wiki is confirming everything I said. I'm going to update the original post.

[JeramieCrowe]

You're not the only one with a beast box that I've seen have issues.

 

And bamf...!

 

http://hewiki.heroengine.com/wiki/Network_Optimization

 

I'ma read me some source and post what I find when I see it.

 

My rig:

 

Intel Core2 Duo E8400 @ 3.00GHz (not overclocked), Win7 Pro 64-bit, 8GB RAM, GTS250 vid card. No server lag, ever, on The Ebon Hawk, 10th most populated server in TOR. Never saw below 24FPS, even on Ilum, except for the occasional spike when an objective was taken, but that's not video lag.

 

Scrupulous about what starts when Windows starts, bare necessities in Processes. Not one for bells and whistles...

[ironix]

I won't say "get a better computer", but anyone that watches the numbers can tell you that you are loading a helluva bunch more data than most MMOs per planet/area because there IS a whole lot more in a planet/area than most other MMOs.

 

See, here's the issue with loading planetary resources. I'm assuming that planets like Aldaraan and Corellia take a long time to load because they're HUGE and have many associated assets. Why exactly can't Bioware have the client only load resources relevant to your immediate surroundings and then load all the other assets in the background?

[ironix]

My rig:

 

Intel Core2 Duo E8400 @ 3.00GHz (not overclocked), Win7 Pro 64-bit, 8GB RAM, GTS250 vid card. No server lag, ever, on The Ebon Hawk, 10th most populated server in TOR. Never saw below 24FPS, even on Ilum, except for the occasional spike when an objective was taken, but that's not video lag.

 

Scrupulous about what starts when Windows starts, bare necessities in Processes. Not one for bells and whistles...

 

During the Rakghoul plague event when the fleet had the mosh pit of players infecting each other, as soon as I would venture near that area, my client would literally STOP for 1-2 seconds before allowing me to proceed. This type of behaviour would most definitely support the OPs position that the client is making blocking calls when loading assets.

[JeramieCrowe]

See, here's the issue with loading planetary resources. I'm assuming that planets like Aldaraan and Corellia take a long time to load because they're HUGE and have many associated assets. Why exactly can't Bioware have the client only load resources relevant to your immediate surroundings and then load all the other assets in the background?

 

...It already does that... What we see now is the best "middle ground". Yes, it's that much data...

[JeramieCrowe]

During the Rakghoul plague event when the fleet had the mosh pit of players infecting each other, as soon as I would venture near that area, my client would literally STOP for 1-2 seconds before allowing me to proceed. This type of behaviour would most definitely support the OPs position that the client is making blocking calls when loading assets.

 

By the way, I take issue with your sig... (I know this is off-topic, but I apologize). Before launch, Georg Zoeller posted a LOT of great info on the Smuggler boards, even several times in my own post...

[ironix]

By the way, I take issue with your sig... (I know this is off-topic, but I apologize). Before launch, Georg Zoeller posted a LOT of great info on the Smuggler boards, even several times in my own post...

 

That's nice, but that's not entirely relevant to the lack of developer communication we've seen since launch. Hell, there are more developer posts on topics like light sabre crystal colour than in the smuggler/operative forums.

 

In fairness, I haven't updated the counsellor/inquisitor post count in a long time. It's probably doubled.

[JeramieCrowe]

That's nice, but that's not entirely relevant to the lack of developer communication we've seen since launch. Hell, there are more developer posts on topics like light sabre crystal colour than in the smuggler/operative forums.

 

In fairness, I haven't updated the counsellor/inquisitor post count in a long time. It's probably doubled.

 

*shrugs* With a level 50 Gunslinger and a level 30 Scoundrel, there isn't anything to discuss. Both are awesome. Smuggler is my love. But I digress.

 

Back on topic - I'll see what I find, as well, as far as the spikes are concerned...if I can get my computer to do it...

[GlowstickSwinger]

Updated the OP. I found out that what I thought was happen is actually happening. They are called Client GOMs and they load .DAT files when a player enters the area.

 

Anyone want to bet those file loads are sync operations?

[JeramieCrowe]

Updated the OP. I found out that what I thought was happen is actually happening. They are called Client GOMs and they load .DAT files when a player enters the area.

 

Anyone want to bet those file loads are sync operations?

 

Dual core requirement? Where is this in your write-up? What your describing, and what the data you present doesn't necessarily correlate. I'm playing Skyrim on the 360 and I get lag spikes. Not connected to a network at all. Not all spikes are attributable to the stack.

[GlowstickSwinger]

Dual core requirement? Where is this in your write-up? What your describing, and what the data you present doesn't necessarily correlate. I'm playing Skyrim on the 360 and I get lag spikes. Not connected to a network at all. Not all spikes are attributable to the stack.

 

I have provided conclusive evidence that their servers and their clients use blocking operations (just as I suspected) in very critical aspects of the game itself. It's in the second spoiler in the opening post.

 

Those .DAT loads are async, but the HeroScript that issues the file i/o is sync!

Edited April 28, 2012 by GlowstickSwinger