But it's only a penalty to the armor rating, which does nothing but reduce the damage I take correct? So if I don't get hit by mobs because my companion is tanking them, then wearing light armor isn't really a bad thing correct?

I have some adaptive armor and i'm aware how that works, but i'm refering to something like this that drops in Black Talon http://www.torhead.com/item/7zJKeIx/black-talon-inquisitors-robe

If i use an Orange piece of Light Armor on my Sniper and mod it with suitable stats, is there any downside to this? I assume I will have a lower armor rating, but if my companion is doing all the tanking then I can't see that being an issue.