In case no one pointed this out, the amount you lose is dependent on the damage of the ability.
Say I have a perfect scenario and my buffs are lined up rights
Before: Blade Storm crit = 4,400k
After: Blade Storm crit = 4,100k
Approx change: 300 some damage
Let's look at a different skill
Let's say my Blade Rush...
Before: Blade Rush crit = 2,200k
After: Blade Rush crit = 2,050k
Approx change: 150 some damage
ect.
It's not a 300 damage decrease on all attacks, so your less hitting attacks do almost 0 damage. It's a 7% drop of the damage that attack does...
Disclaimer: Math might be slightly off or not reflecting current in-game numbers, but It's 2:30 am my time and I don't care, though I'm pretty sure it gets the point across... If this was already pointed out, sorry and have a nice night!