I'm having the exact same issue, this basically makes quickbar 6 unusable to me because I don't want to have it on display all the time, cluttering my UI.
Developers, please pay attention and fix this simple key binding! Specifically: "Quickswitch to Quickbar Page 6"