I believe that's the one, it's just there's a bunch of guys before that chasm that you can fight who aren't actually part of the quest. The prisoners themselves are on the other side of it, and there I found the groups to be much smaller. It was that initial crew I ran into that surprised me.