Had same problem as well. Tho the first time i did BT with 4 players i didn't have any problems. In the evening i did another round, only difference i did the 2 man version. After most of the dialoge scene's I had the problem discribed here.
i had to get kicked and reinvited 5-6 times, really anoying.
This is the only thread i found about it, hope the problem be resolved quickly.