Yeah, it doesn't matter if you've finished their conversation quests or not, they'll still lose affection. I don't see what the problem is, though - either dismiss them for conversations where you'll lose affection, swap them out for another companion, or just buy them a gift to top them up every once in a while.
...and don't call them pets!