It looks like doom requires the enter key be mapped. Retroarch only maps a subset of keys to emulate a controller which unfortunately doesn’t include the enter key. And like you have seen the enter key becomes the escape key.
Quake seems to be playable in this state because it accepts something other than ‘enter’ to make selections in the menu.
Until there is a fix for retroarch to be more keyboard friendly then I think using prboom directly is the best bet.