Open the file: opt\retropie\configs\n64\mupen64plus.cfg
Locate the line that says: Joy Mapping Stop =
This is the joystick event string for stopping the emulator. For my iBuffalo SNES controller I set the exit button to Select, which is button 6, since the N64 controller has no Select button anyway.
Since I’m using button 6, I set that line to: Joy Mapping Stop = “J0B6”
J0 means player 1 and B6 means button 6.
You cannot use hot keys for this emulator, but with this configuration I simply press select while in game and it exits out to Emulationstation.