Well that's just unprofessional.
You're a spy. You were sent to spy on them. They found a way to make you a double agent. That's the way Cloak and Dagger works. Some of their agents could have used a bit more professional courtesy and not used you for their own private ends, but in the greater scheme of things that is more of a human resource issue than a governmental issue. Honestly, they had complete control over you, and could have made you into a party favor.
Ultimately, joining the Empire or the Republic is a personal choice, and my Agent (when I get there) will probably remain with the Empire but the brain-washing thing was basically committed by both sides. If you're going to hold it against one, the other side balances it out to a zero sum.