I’m also having this issue, along with many more. I posted this issue on Reddit but got down voted.
Do not buy the 8bitdo NES30 Pro Controller for RetroPie. It doesn’t work with it properly. from RetroPie
BTW, you know you’re in Joystick mode because you’ll see “Joystick” in “8Bitdo NES30 Pro Joystick”. If you we’re in Joystick mode, the name wouldn’t have “Joystick” in it.