Benchmarking / Testing / What do you want to see? (if anything)

[TX_Angel]

http://www.swtor.com/community/showthread.php?t=868156&page=6

 

In the above thread, I got carried away and did a bunch of random tests on multiple computers, using FRAPS, to try and see what type of performance to expect in SWTOR on various hardware.

 

Now I'm not a game tester by trade, and I'll admit that some of my methods may not be perfect. I need to redo the Dell laptop test there and check it for errors that someone suggested might have been made.

 

However, what do you all want to see, if anything? My thought was that perhaps figuring out what really is needed to do X in the game. Of course story is less demanding, 8v8 warzones seem the most. GSF isn't so bad, I haven't benched ops yet.

 

I haven't tested AMD, but the conventional wisdom is that AMD is inferior to Intel for SWTOR. Of course it is one thing to make such a claim casually on a forum, it is another to actually benchmark it and put numbers to it.

 

When I go to Anandtech and Tom's Hardware, they test games like Battlefield and Crysis, not SWTOR. What is the interest level in having an actual detail benchmarking series on SWTOR covering a wider swath of hardware? What would you like to see it tested on, and how?

 

Open to ideas, shoot!

[Holocron]

I'd like to see you test a war zone on medium to high settings with all of the ui disabled except for one hotbar. Then repeat the test with your normal ui, and record the differences in fps. I'd also like to see this same test repeated on both a AMD cpu and an Intel one. Edited February 3, 2016 by Holocron

[FlyingUsPoo]

Intel i7-4940mx w/ 32GB RAM

Nvidia GTX 880M (8GB dedicated) / Intel HD 4600 on CPU w/ 6 GB system memory allocated

SWTOR installed on SSD

OS: WIndows 7

 

In-game settings:

 

 

%APPDATA%\..\Local\SWTOR\swtor\settings\client_settings.ini

[Renderer]

AntiAliasingLevel = 4

AtlasQuality = 0

Buckets = 4

D3DFullScreen = false

GraphicsDeviceId = 4504

GraphicsQuality = 6

Height = 1080

MeshLODQuality = 1

NativeHeight = 1080

NativeWidth = 1920

PerfTestFlags = 38780672

RefreshRate = 60

ShadowMapNumCascades = 8

ShadowMapResolution = 2048

SpeedTreeDistanceScale = 0.75

TextureAnisotropy = 16

Width = 1920

PlantDensity = 0

WindowX = 0

WindowY = 0

VerticalSyncState = false

AllowDepthOfField = true

doBlobShadows = true

doShadows = true

CodeVersionChanged = false

LastCodeVersionRun = 1

NewBucket = 4

EnableBloom = true

AllowColorRemapping = true

ShaderSet = 9

TextureQuality = 0

DYDDrawDistancePreset = 2

FullScreen = true

 

 

You wanted FRAPS in the other thread so -

 

Benchmark taken during explosion in middle of combat in center section of pylon pvp match.

System forced to use Nvidia card via Nvidia control panel:

Frames, Time (ms), Min, Max, Avg

5811, 69935, 59, 104, 83.091

 

System forced to use Integrated chip:

Frames, Time (ms), Min, Max, Avg

2050, 68266, 21, 38, 30.030

 

Integrates with same setting + grass and tress on full right

Frames, Time (ms), Min, Max, Avg

2458, 68266, 13, 28, 20.483

 

 

The Intel 3D settings are:

General Settings: Performance.

Application Optimal Mode: Enable

Multi-Sample Anti-Aliasing: Use Application Setting

Conservative Morphological Anti-Aliasing: Turn Off

 

In my experience with this game, Windows 8 and 10 don't do it any favors.

Edited February 3, 2016 by FlyingUsPoo

[TX_Angel]

Intel i7-4940mx w/ 32GB RAM

Nvidia GTX 880M (8GB dedicated) / Intel HD 4600 on CPU w/ 6 GB system memory allocated

SWTOR installed on SSD

OS: WIndows 7

 

That is a crazy mobile chip. If you don't mind my asking, what laptop is that in? I've never seen one with that before.

 

Second question, are you getting a stable turbo to 4GHz when playing SWTOR?

 

You wanted FRAPS in the other thread so -

 

Yes, thank you... I find that just looking at the FPS meter is so deceiving... At least with recorded numbers, there is no room for guessing.

 

FRAPS set to 120 second duration.

Benchmark taken during explosion in middle of combat in center section of pylon pvp match.

System forced to use Nvidia card via Nvidia control panel:

Frames, Time (ms), Min, Max, Avg

5811, 69935, 59, 104, 83.091

 

Ahh, you're running full screen, not windowed... unless you have a G-Sync panel at 120+hz? Mine never goes over 62fps recorded since my panels are 60hz panels.

 

Still, very nice numbers, an average of 83 is crazy in pylon WZ if you have it all turned up to max.

 

One thing I have not done is set my desktop to 1080p and try it at 4.2GHz on a Haswell i7 with the GTX 980 TI. Since I normally run on three monitors, I suspect that is holding the CPU back, but only a test will tell the tale.

 

System forced to use Integrated chip:

Frames, Time (ms), Min, Max, Avg

1132, 36676, 20, 39, 30.865

 

I'll be honest, that just doesn't seem possible. I tested on a i5 today with Intel HD 4600 and on a i5 with HD 530, and both frankly sucked. Like, to the point they weren't worth playing.

 

If you can run 30fps average on built in graphics, then what does everyone buy a GPU for? I have never actually found a game that runs worth a darn on built in graphics, except perhaps minecraft. It is a shame you can't physically pull the GPU out, I wouldn't be shocked if it was helping.

 

I would be interested in hearing from other players who use only Intel iGPU graphics and see what they say. Clearly more info is required.

 

In my experience with this game, Windows 8 and 10 don't do it any favors.

 

The only machine I have that doesn't have Windows 10 on it (well, it does, in dual boot), is a Core2Quad Q6600 with a ATI 5850 GPU. I did try, awhile ago, to run on that machine and I found the performance to be horrible, but that was on the Windows 10 side if I recall.

 

I'll give it a try on the Windows XP side and see if it makes any difference to the numbers. It probably won't be for a day or three however, I have actual work to do tomorrow. But still, it will be interesting.

[Ramalina]

With a Dell inspiron 17R 7720, i7 3630qm, 8GB ram, and Nvidia 650M I typically get 40 to 60 FPS with all the texture settings at high, bloom on, AA medium/low, shadows low, clutter and trees 50%, and usually friendly nameplates off.

 

Edit: I run fullscreen windowed mode for combat log parser compatibility, often I'll have a web browser, TS3 or mumble, a SWTOR log parser, and sometimes for GSF matches OBS running to stream video. I typically kill unused processes using task manager though it doesn't seem to make a huge difference if I forget. /edit

 

I have a UI similar to my old WoW raid healing one, and it displays a lot of stuff.

 

I force all my games to run on the Nvidia, I think I have pysX set to Nvidia driver controlled.

 

The Intel graphics in my experience really suck for pretty much everything other than web surfing, word processing, and watching DVDs. The one cool trick it can handle is doing the video encoding for Open Broadcasting Software when I stream, which is great since leaving the CPU and GPU free for SWTOR means that there's no significant framerate penalty for streaming.

 

 

SWTOR doesn't seem to stress the hardware that much, but it sure is picky about settings.

 

Things that have tanked SWTOR framerate for me at various times:

 

Laptop power settings

GPU power settings

Video defaulting to the Intel chipset

Intel Rapid Storage Technology turning partially off due to a HD error ( off is fine, on is fine, halfway is not ).

Dust on the heat exchanger fins ( steady 5-7 FPS, cleaning the cooling system every couple of months is a good idea).

Real Time Performance Destruction, aka Real Time Scanning settings from anti-virus software.

Nvidia updater automatically "optimizing" settings for ugliness and low framerate.

Various automatic driver and system updates that altered settings.

 

The heating system getting gunked up and defaulting to the Intel chipset were universal performance killers. For most of the software setting adjustments I've had to do, SWTOR gets crippled by things that other games don't even blink at. It's sort of like a Check Engine light for the computer.

 

I've run SWTOR on Win 7, 8.1, and 10, and OS doesn't seem to have much effect for me, assuming everything else is ok. I find 8.1 and 10 start up and handle other programs noticeably faster than 7 did, so I'm not really motivated to try going back.

 

If you're familiar with the color coding feature of the built in FPS meter, I can say in my personal experience that it tends to blame the GPU no matter what the actual problem is.

Edited February 3, 2016 by Ramalina

[TX_Angel]

System forced to use Nvidia card via Nvidia control panel:

Frames, Time (ms), Min, Max, Avg

5811, 69935, 59, 104, 83.091

 

System forced to use Integrated chip:

Frames, Time (ms), Min, Max, Avg

2050, 68266, 21, 38, 30.030

 

I copied and pasted my results from the Dell notebook in the OP listed above:

 

Run 1 - 1080p Full Screen Windowed

Ultra settings - except AA turned off, shadows turned off, grass turned off.

 

Frames - 5978

Time (ms) - 300000

Min - 11

Max - 33

Avg - 19.927

 

Note: That machine has a 2.3GHz i5 Skylake CPU that turbos to 3.2GHz (but really runs at 2.8GHz in SWTOR due to multi-core use). It also has a GTX 960M GPU.

 

That machine was purchased new in October it was sitting on a desk so it had good airflow, plugged into the wall, and wired networking.

 

I was quite surprised how low it performed in warzones, and from what you've posted, you got a faster FRAPS using the Intel HD 4600 than the Dell did with the GTX 960M.

 

Something is WAY off there. Either your numbers are wrong somehow, or my system is not running properly. I have a really, really hard time believing that Windows 7 to 10 is the cause of that, too many people now run Windows 10, it isn't an obscure OS.

 

---

 

I need to test the Dell again, see how repeatable my first numbers are. I'll also look and see if I can easily disable the 960M GPU and run it on the Skylake HD 530 GPU instead.

[ALaggyGrunt]

The easiest way to make a reproducible test is to break ToS and make a script which runs you through several places or fights mobs, or the space missions. If you did that and they found out, you'd be in trouble.

So much for reproducible.

 

The next best thing is to run the same 8- and 16-man ops and flashpoints, and get a bunch of guildies to have an open world fight somewhere, simulating what happens if everyone dogpiles everyone in an 8v8 WZ. Maybe that teleporter for the Gree event still works even though the event isn't running?

 

The other thing I'd be interested to see if you have a few thousand dollars to throw at it is:

What happens if you run those 6- or 8-core Intel CPUs with 15 and 20MB of L3 cache at the game? Or the i7 5775c with that L4 cache, or just for its particularly fast integrated graphics?

And, how much difference does memory speed and 2/3/4-channel mode (if you have a motherboard with 4-channel mode) make, if any at all?

Does it matter how much memory is on a video card?

Edited February 3, 2016 by ALaggyGrunt

[FlyingUsPoo]

That is a crazy mobile chip. If you don't mind my asking, what laptop is that in? I've never seen one with that before.

https://www.msi.com/product/notebook/GT70-2PE-DOMINATOR-PRO#hero-specification

 

Second question, are you getting a stable turbo to 4GHz when playing SWTOR?

3.7 - 3.9Ghz

 

Ahh, you're running full screen, not windowed... unless you have a G-Sync panel at 120+hz? Mine never goes over 62fps recorded since my panels are 60hz panels.

 

Still, very nice numbers, an average of 83 is crazy in pylon WZ if you have it all turned up to max.

 

One thing I have not done is set my desktop to 1080p and try it at 4.2GHz on a Haswell i7 with the GTX 980 TI. Since I normally run on three monitors, I suspect that is holding the CPU back, but only a test will tell the tale.

I actually run it fullscreen windowed.

 

Nvidia settings:

 

 

Ambient Occlusion: Off

Anisotropic filter: Off

Antialiasing - FXAA: Off

Antialiasing - Gamma correction: Off

Antialiasing - Mode: Enhance the application setting

Antialiasing - Setting: Use global setting (Application-controlled)

Antialiasing - Transparency: Off

CUDA - GPUs: Use global setting (All)

Maximum pre-rendered frames: 1

Multi-display/mixed-GPU acceleration: Single display performance mode

Power management mode: Perfer maximum performance

Shader Cache: Use global setting (On)

Texture filtering - Anisotropic sample optimization: On

Texture filtering - Negative LOD bias: Allow

Texture filtering - Quality: High performance

Texture filtering - Trilinear optimization: Off

Treaded optimization: Auto

Triple buffering: Off

Vertical sync: Off

Virtual Reality pre-rendered frames: 1

 

 

I'll be honest, that just doesn't seem possible. I tested on a i5 today with Intel HD 4600 and on a i5 with HD 530, and both frankly sucked. Like, to the point they weren't worth playing.

 

If you can run 30fps average on built in graphics, then what does everyone buy a GPU for? I have never actually found a game that runs worth a darn on built in graphics, except perhaps minecraft. It is a shame you can't physically pull the GPU out, I wouldn't be shocked if it was helping.

 

It runs, and runs smooth, but it looks like crap. There are distance artifacts, tearing, its not a pretty thing. Besides, I also play Witcher 3. NO WAY thats gonna run on the integrated card. I had to pull apart this laptop and replace all the thermal pads with paste because W3 was overheating this thing.

The Integrated graphics are great for the 3D effects Windows is introducing to desktops. They are great for Minesweeper/Solitaire/Suduko and all those other games you see Grandma playing when she thinks no one is looking (hi Farmville). It has a niche market. The primary reason it "can" run SWTOR... SWTOR is old. It runs smoothly on my system because my system has a lot of resources to back it up. Yank out 3/4 of my RAM or disable 2 cores and it won't run for crap.

 

I have a really, really hard time believing that Windows 7 to 10 is the cause of that, too many people now run Windows 10, it isn't an obscure OS.

They are very different kernels.

I'll also look and see if I can easily disable the 960M GPU and run it on the Skylake HD 530 GPU instead.

Simply use the Nvidia Control Panel to force swtor.exe to use the integrate chip. You can either drill down to the exe or use the global setting. However, unless you have the same system resources (memory, processor cache, etc), you can hardly expect to get the same results I did. It would comparing apples to chairs.

[TX_Angel]

With a Dell inspiron 17R 7720, i7 3630qm, 8GB ram, and Nvidia 650M I typically get 40 to 60 FPS with all the texture settings at high, bloom on, AA medium/low, shadows low, clutter and trees 50%, and usually friendly nameplates off.

 

Is that in a 8v8 warzone, or out in the open world solo?

 

I only ask, because the Dell machine gets a solid 60fps in the open world. Running Hoth heroics, it didn't blink an eye, ran smooth as silk (as much as the game does of course). It was in 8v8 Pylon WZ that it fell off a cliff.

 

The Intel graphics in my experience really suck for pretty much everything other than web surfing, word processing, and watching DVDs. The one cool trick it can handle is doing the video encoding for Open Broadcasting Software when I stream, which is great since leaving the CPU and GPU free for SWTOR means that there's no significant framerate penalty for streaming.

 

What software do you use for that? I've never done any game recording or streaming, but I think I might. I tried having FRAPS make a movie, but it tanks the frame rate.

[TX_Angel]

It runs, and runs smooth, but it looks like crap. There are distance artifacts, tearing, its not a pretty thing.

 

A thought just occurred to me... You're messing around with a lot of "under the hood" settings... which the vast majority of people will never do. I simply am picking the options that are in the game itself, along with whatever the defaults are in Windows and the drivers that come with it.

 

I just did some looking around in the control panel on my desktop computer, I don't see a place to pick the GPU, but that might be because this is a desktop and it physically can't select the Intel chip, the monitors are plugged into the NVidia GPU and aren't going to run off the Intel chip.

 

I wonder how much performance is lost with single channel main RAM when using the Intel chip. Didn't even think of it until just now, but both machines I tested today, the Haswell and the Skylake, both only have a single stick of DDR3 RAM in them, both at 1600. A second stick of RAM might make a difference. Since both are "prebuilt" computers, I don't know that I can put anything faster than DDR3-1600 in them.

 

My daughter's i5-6500 machine is a custom built and it has DDR3-2400 in it, I might take the GPU out of that and try it. On the plus side, it'll turbo higher than the i5-6400 I tested today, maybe that'll help

 

---

 

It would be interesting to see a HD 4600 test from you with everything set to the defaults that Windows and Intel put them at, just for comparison sake. You've got a great CPU and I suspect your RAM speed is helping, now that I think about it.

 

And of course, FPS is just a number, as you said, it looks like crap. What happens if you turn everything all the way up? When I did that, I got, if memory serves, about 16 fps average with drops to 8 or so, I'd have to go back and look to be 100% sure on that.

[TX_Angel]

It would comparing apples to chairs.

 

But what if I like apples and chairs?

 

Anyway, thank you for the details replies. I am grateful that you've replied in the details that you have, it gives me some new things to test and to refine what I'm doing.

[Ramalina]

Is that in a 8v8 warzone, or out in the open world solo?

 

I only ask, because the Dell machine gets a solid 60fps in the open world. Running Hoth heroics, it didn't blink an eye, ran smooth as silk (as much as the game does of course). It was in 8v8 Pylon WZ that it fell off a cliff.

 

 

 

What software do you use for that? I've never done any game recording or streaming, but I think I might. I tried having FRAPS make a movie, but it tanks the frame rate.

 

I tend not to run melee, so I try not to stand in the middle of 15 person particle effect extravaganzas. It wouldn't shock me if doing that dropped things to the mid 30s or even mid 20s. Though, I have to say, if I was choosing to turn off all nameplates or tinker with things to reduce particle effects loads, I'd go for the nameplates first. SWTOR nameplates have some sort of special evil when it comes to framerates. For 8 person raids it tends to stay at least in the mid 40s.

 

The software is OBS, Open Broadcasting Software. Free and open source. I have the Win version but they're converting to multi-platform Win/Mac/Linux going forward. Twitch has a slightly outdated tutorial for setting up OBS to stream to a Twitch account. You can also choose to just record instead of streaming. I normally have to use the SWTOR window as the source in order to get it to work properly. If I use the CPU or GPU for encoding there's enough lag in drawing to the screen (it seems to do the encoding first, and only then display to the monitor) to cause problems in GSF play, where .15 seconds can feel like a really long time. With the integrated intel, I think the option is called 286k or something like that, I can't even tell OBS is running.