In and of itself, it's not a bad thing, and in other games I use it. The problem is that too many people use the numbers as an excuse to rage on people they feel don't meet up to their standards, even when the group is doing fine.
In other games, I actually stopped joining PUGs just because I get tired of seeing people vote kicked or getting verbally abusive over it.