The only ones that really pissed me off were the Nar Shadaa platforming one, the Belsavis lava one was really tough specifically because you kind of had to glitch your way on top of the rocks, and the Tatooine balloon one was ridiculous. I don't even want to think how many hours I spent trying to get that one, and ended up dueling someone to force jump to the STR +3 on the side.
But as everyone has said, these are by no means a mandatory undertaking. In the end it will give you a pretty nice stat boost, especially with the relic shards, but don't complain about the difficulty of something that you aren't being forced to do.