Jaesa on the Sith Warrior side has the same problem. If you enter combat, defeat whatever it is and then exit combat, she'll put the saber away. However, as soon as she reappears to you from entering a new zone, hopping on and off your speeder, etc., the saber is back out.