It is annoying. When I play I find that I have to turn my screen so that the mob/player I am fighting is not in the center, particularly if I have to watch for visual cues like on the gate boss in DF. This affects all my characters, some more than others, laggy times more than lag free times.
No offense to anyone intended, but if you don't see the red text you may not even be watching the action, because the only way to play at your best and watch the action (as opposed to the cooldowns) is to spam the buttons until the abilities cool down, and spam multiple buttons in the cases where you rely on abilities not tied to the global cool downs. As we all know, there is a minor lag between when something happens, we perceive something happening and when we are able to act on it.
I guess unlike some of you people I like to watch what is happening on the screen, the graphics are part of the reason I play this game, if all I cared about was watching the cooldowns then why would I bother with playing one of the best graphical MMO's? The only way you don't notice the red text is if you are watching the cooldowns instead of the action -- and personally I don't like playing that way, at all.
update: After reading the whole thread and several others posted by the OP I think I may just try to turn off my UI since all my abilities are bound to buttons (though I am concerned with how this will affect /rc and chat). I do not point and click on hardly any abilites. Neither do I use a macro program, so advise others have posted on changing the macro delay does not apply. Having read much on the issue I have noticed that people with the most issues also have a robust key bind setup, and those without the issue point and click, though I am sure that is an oversimplification.