I completely agree with OP. Ability delay is very frustrating, especially in PvP for melee classes.
I also noticed an animation issue. When I press the ability (Slash for example) and my Jedi Knight raises his lightsaber to attack, and while doing so parries an incoming blow, he suddenly stops executing a given command. Not to mention double animations of abilities even if I press them only once.
This has a great impact on my combat and overall playing experience. I quickly grow tired of slow character response and after 4-5 hours of gameplay I fall asleep.