How populated your side of the sever is (aka Emp or Rep) determines how slow/fast you load. I load in about 10 seconds on our Rep side, because there are like 1/3 the number of Rep to Emp, and on my Emp char, it takes about a half minute to load into a planet/fleet.
Nothing you can really do to change that. Now if your loading time is in the couple of minute timeframe, like it is for some of the people posting on the forums, then the problem is entirely your computer, and there is a number of things that could cause it, but since that isn't your issue here, cuz your loading time is normal, I won't go into it.