Hopefully I can provide some helpful advice.
I know you mentioned you have a 20Mbps connection, but that doesn't mean your getting 20Mbps outside of your ISP's network to the internet. I would first run some speed tests throughout the day to gauge what kind of bandwidth you're actually getting. I would reccomend Speedtest.net,
and choose a server hosted by someone other than your ISP. I would be willing to bet you're getting half that bandwidth during peak times, maybe even less.
Also, what CPU and GPU are you currently using? Certain graphical options have a larger impact on performance than others, and shadows are one of them. Try turning shadows off to see if you that helps(it should).
I think your main goal should be to determine if you are experiencing server side lag, or local lag. Considering it's only noticeable in certain environments, I would imagine it's mostly localized, with computer hardware being the main factor, but you never know. Do you notice a delay between activating a skill, and having it trigger? I played the beta(just got my early access invite) and server lag was tremendous. Sometimes I think it's hard for people to differentiate the difference.