Are you having the same weird graphical issues with all 3 systems?
I’ve personally experienced this issue with the N64 emulator. Luckily, RetroPie comes with a few N64 emulator plugins for Mupen64plus.
Assuming you have RetroPie 3.0 Beta installed, once you start the game, immediately hit X or M on a keyboard after you select the game and the screen goes to a command line and it gives you the option to select different entry. Try the to see what one gives you the best performance. You can do this on a per-ROM basis, since some roms get along with certain emulators, but not others.