The Squirrel Labs 2.8″ display is AdaFruit compatible and works great with both the AdaFruit drivers and the tutorial by Phillip Burgess. The display needs a little filing and fitting, but does just squeeze in a GameBoy case. I had to carefully file about 0.5mm-1.0mm off each side making sure I did not file through any tracks. Then I drilled out and elongated the mounting holes to make it fit over 3 of the support posts in the GameBoy plastics. The 4th post had to be removed from the plastics as it is over the displays header.
The Nextion looks just like a generic SPI screen to me. If you can get Pi compatible drivers for it then it should work OK with the fbcp program required to duplicate the display output. However, it looks a little expensive for a screen of that type, particularly as it is from China. The Squirrel Labs display is roughly the same price, but that has proper Pi connections and known working drivers.