Performance Issues with es-scraper

Home Forums RetroPie Project Everything else related to the RetroPie Project Performance Issues with es-scraper

RetroPie has a new website and forum. Please visit https://retropie.org.uk/ for the new site. The new forum is located at https://retropie.org.uk/forum/. This forum is left here as a read-only archive.

This topic contains 34 replies, has 16 voices, and was last updated by Profile photo of amccabe126 amccabe126 2 years, 6 months ago.

Viewing 35 posts - 1 through 35 (of 35 total)
  • Author
    Posts
  • #3210
    Profile photo of doc-thirst
    doc-thirst
    Participant

    First off, I wanted to say thank you to the folks who provided this great piece of software, very cool. Also to many of the posters here who asked and got answers to many of my questions already.

    Now to my problem…

    I have a fairly large (NES) rom set, somewhere in the 1500 range. A ton of these are goofy user created games and whatnot so I haven’t dismissed this is a rom related issue, but I really don’t think it is, but in the spirit of full disclosure I thought I should mention it.

    I’m able to set up everything without a problem to the point where I run es-scraper. Up until this point, everything seems fine, performance isn’t an issue, all the emulators popup quickly. However after I’ve “Re-scraped the roms” with es-scraper the system drags to a halt. It hangs when switching between roms and sometimes takes several sections to switch between emulators. Game performance doesn’t appear to be affected, just navigation of emulationstation.

    During the es-scraper process, I do get a ton of “cannot resize image” errors, pretty much on each file. I did verify that I have python-imaging installed and updated.

    After it runs, many of my images are giant both in visual size and file size (some are as big as 6meg). I’d imagine this is the problem, emulationstation is choking on these giant images. Which leads me to a few questions…

    1) Any suggestion on troublehsooting/fixing these resize image error?
    2) Is there a chance this problem is the number of roms and not the images?
    3) I can transfer these files via the network, can I just resize them manually?
    4) What is an ideal size (both physically and file size) for these?
    5) Is there anything I can provide to the community which might help you all troubleshoot?

    #3218
    Profile photo of motorhead
    motorhead
    Participant

    Same problem here, but I don’t have a ton of roms. The roms scrapped two or three weeks ago seems to have the image resized. But yesterday, before a binaries update, I re-scrapped some more roms and I found two issues:

    – Manual scrapping is not working for me, it was like auto scrapping… strange.
    – The new images added where huge.

    Problem with an update?

    #3220
    Profile photo of doc-thirst
    doc-thirst
    Participant

    Humm, hopefully that’s the case, I’m not sure what options I have as a end-user if it’s not. Are there are permission issues I might take a look at?

    #3221
    Profile photo of kilgorefry
    kilgorefry
    Participant

    I’m also thinking (hoping) it’s just a bug that came with an update that someone more skilled than I am will be able to fix.

    I noticed it for the first time last evening when I tried to do a re-scrape.

    From terminal, I’m seeing this,

    Downloading boxart..
    Image resize error

    #3227
    Profile photo of doc-thirst
    doc-thirst
    Participant

    Yep, that’s what I get too. It appears to happen on every rom, but several do have art once it’s finished, granted it’s hit or miss on how big it is, but it’s art none the less.

    #3232
    Profile photo of motorhead
    motorhead
    Participant

    I changed them manually, it’s a good option if you don’t have a lot of new roms… waiting for a fix.

    Enviado desde mi GT-I9300 mediante Tapatalk

    #3246
    Profile photo of doc-thirst
    doc-thirst
    Participant

    any suggestions on image size?

    #3251
    Profile photo of kilgorefry
    kilgorefry
    Participant

    I believe 350 pixels is the maximum width that the ES-Scraper documentation recommends.

    #3257
    Profile photo of newsworthy
    newsworthy
    Participant

    I ran ES_Scraper myself from the setup script two nights ago, and let it go to work on the 2000 ROMs I have installed. Took ages, but it too downloaded a variety of sizes for each one. Some perfect, meeting the 350px size, but many that were huge and cause browsing to be slowed right down.

    Contemplating running it manually on each rom directory one by one, forcing the 350px limit using one of the variables in the script. Is this a wise course of action? Or am I going to have to settle in for a long time of scraping again?

    Retropie Beginner/Tinkerer

    #3259
    Profile photo of doc-thirst
    doc-thirst
    Participant

    It sounds like manually scrapping each or manually resizing each (out side es-scrapper – using photoshop or paint) is viable, but this is a maintained project so there should be a fix at some point sooner than later.

    I plan to just sit tight and wait for the author to check out the problem (it might not hurt to reach out to him via git and let him know there is a issue – he may not be aware). I took this opportunity to clean up my roms and copy of a fresh list. I’m going to keep an eye on the git page and watch for an update. If I don’t see a new release in the next few weeks, I’ll go the manual route.

    #3398
    Profile photo of masterehm
    masterehm
    Participant

    Hi,

    I have the same problem and I opened a post [1]. Only now I saw this post here. And yes, somethings wrong with the image resize function in the python script.

    [1] http://blog.petrockblock.com/forums/topic/es-scrapper-emulationstation-peculiar-presentation/

    #3402
    Profile photo of masterehm
    masterehm
    Participant

    Riddle solved. There was a change [1] on Nov 20, 2013:
    Import PIL instead of just the Image class since it doesn’t resolve on all platforms.

    
    -import os, imghdr, urllib, urllib2, sys, Image, argparse, zlib, unicodedata, re
    +import os, imghdr, urllib, urllib2, sys, PIL, argparse, zlib, unicodedata, re
    

    Change PIL to Image in scraper.py [2] and resize works again.

    [1] https://github.com/elpendor/ES-scraper/commit/37c34f32ef522df2902316318ea876c1a3c11798
    [2] /home/pi/RetroPie/supplementary/ES-scraper/scraper.py

    #3405
    Profile photo of doc-thirst
    doc-thirst
    Participant

    Change PIL to Image in scraper.py [2] and resize works again.

    Was this a type-o? Shouldn’t it be the other way around?

    EDIT: OK I miss read this solution. The author changed it to PIL from Image to accommodate differnt platforms, which is what is causing our error.

    By changing it BACK to Image (as detailed above) the problem is solved. Sorry for my lack of reading comprehension.

    #3408
    Profile photo of smartroad
    smartroad
    Participant

    Woohoo I shall try this tomorrow 😀 Chers for identifying the issue and sharing the solution 😀

    #3413
    Profile photo of motorhead
    motorhead
    Participant

    Hi! Still not working for me… I have same errors resizing images, scraping its crashing, and manual scraping seems to be broken too…

    #3416
    Profile photo of masterehm
    masterehm
    Participant

    You changed manually the line from PIL to Image in scraper.py? If you use the scraper.py with the option -v, you get an Image resize error?

    #3429
    Profile photo of motorhead
    motorhead
    Participant

    Sorry, it was my fault, it’s not working if I try to run it under Retropie-Setup, but it’s working fine using commands directly.

    Thanks!!

    #3430
    Profile photo of FreezX
    FreezX
    Participant

    The solution posted by Masterehm solved it on mine too. Solved?

    #3433
    Profile photo of scurvykid
    scurvykid
    Participant

    Thanks everyone who found the solution, worked a charm!!

    #3478
    Profile photo of petrockblog
    petrockblog
    Keymaster

    Sorry, it was my fault, it’s not working if I try to run it under Retropie-Setup, but it’s working fine using commands directly.

    Which command are you using now on the command line? I would modify the RetroPie Script accordingly.

    #3481
    Profile photo of motorhead
    motorhead
    Participant

    Im using this one for manual scrapping:

    sudo python scraper.py -w 275 -m

    #3482
    Profile photo of petrockblog
    petrockblog
    Keymaster

    Thanks for that!

    Did you also try the “-newpath” option of es-scraper? I cannot make it work for some reason right now.

    #3957
    Profile photo of ahambric98
    ahambric98
    Participant

    Just wanted to start off by saying thanks so much for all the work you have already put into this. Just ordered some cases and controller adapters. Looking forward to putting the whole system together.

    I have downloaded the lastest RetroPie scripts and notice this is still an issue. Are there plans to fix this in the script itself later?

    Thanks again!

    #4187
    Profile photo of amccabe126
    amccabe126
    Participant

    Following this forum has gotten me the closest to getting it to work so far! However, running ES-scraper manually took me about 3 hours for NES games A thru C, and it completely froze in the middle of the C list. I read the advice on the ES-scraper documentation website about renaming ROM files using goodtimes. I downloaded the goodtimes and goodgui and converted all my NES file names. I’m going to retry this with a small sample manually and with a large sample in auto-scraper. It seems that ultimately I will have to get the auto scraping to work when I’m doing thousands of roms. While ES-scraper isn’t necessary to build a retropie gaming system I want to make something that looks complete since I’m making it for my brother’s birthday. Cheers.

    #4204
    Profile photo of amccabe126
    amccabe126
    Participant

    Also, does anyone know if the ES-scraper works better when you set the memory to 256 or should I leave it at default? For some reason the auto ES-scraper used from the setup GUI recommends it is set to only 16.

    #4223
    Profile photo of RetroMarine
    RetroMarine
    Participant

    Join the club…I delt with all the same frustrations as you, trust me. Truth is, the only legit way to do the scraping is manually, and even then, you will have to pay attention to the roms it auto names (which it will do even though it’s supposed to be manual). Then you have to go back through the gamelist.xml to delete the games it renamed wrong. I manually scraped all my nes, snes, genesis, and psx roms. It took FOREVER…but I think it was worth it, because you only have to do each emu once, then save the entire directory to include the gamelist.xml to your PC, and you can just reuse it over and over again.

    As far as the es_scraper crashing in the middle of your C list…did it create a partial gamelist.xml? If it did, open it up and see what game it failed to write. You may have a “C” game with a variable the scraper can’t read. For example, it happened to me with the Sega Genesis game André Agassi’s Tennis. The scraper couldn’t read the é. Once I deleted the game, the scraper was able to finish. Check it out and let me know what you find.

    #4250
    Profile photo of amccabe126
    amccabe126
    Participant

    Thanks for the feedback! After I posted this I figured out only about 75% auto scraped correctly, and sometimes it was the popular titles that didn’t make it. However, this is when they were all named incorrectly and filtered through goodtimes into new names. I’ve since found a rom download site that tends to have the roms already named correctly. In the end I will probably have to manually scrape and modify those unidentified roms a few times through until I have the base rom list easily identifiable. I might just manually scrape them in sections and weed out the crappy ones or rename them until the final product at which point I’ll of course have to do it manually from start to finish at this point.

    Two questions I am wondering is whether the SD card write speed matters (ie 4 vs 10) or whether or not changing the the memory so that the pi has 256/512 vs 16/512.)

    Another question: does anyone know what the file naming conventions are that are most easily identifiable by the ES-scraper? It seems like the most successful ones had a (U), a (!), and sometimes PRG. I’m guessing that (J) means japan and that (U) means US. Not sure what the exclamation point means. I started downloading the roms from http://www.romnation.net/srv/roms/ since they appear to be named well. I’ve been going for standard (U) without (!) if its available, ignoring the (hacks) (j) (Unl) and (0# B#) labels.

    If I had the list that ES-scraper uses to identify roms I bet this could be a lot easier does anyone know how to access this?

    Many thanks for all the input and help!

    P.S. I started deleting Doom, duke3d, apple2, x86, and cavestory so that the gui loads only the emulators/roms the user has access to since I’m doing a usb controller only interface. This leaves the input config left, which someone could accidentally run and reset the game-exit. Does anyone know how to disable the input config screen? Thanks!

    P.P.S. I discovered with the naming convention that (U) means US and (!) means verified good dump, so I’m going to test the difference between (!) roms and roms without the (!) label.

    #4256
    Profile photo of somail
    somail
    Participant

    amccabe: what romset are you using?

    If you are using nointro I can can post my xml’s which have already been manually scraped. That should save you sometime.

    #4258
    Profile photo of amccabe126
    amccabe126
    Participant

    right now I’m starting with GBC, NES, and supernintendo. I’m just downloading the roms from romnation that are marked with “” and “[!]” What are the romsets and xml’s? Thanks!

    #4261
    Profile photo of somail
    somail
    Participant

    Sorry I have to correct myself. I forgot my romsets are hyperspin compliant which means they differ a little bit from nointro ( same roms files but different naming scheme). My xml’s probably will not be able to save you any time.

    #4347
    Profile photo of boxxle
    boxxle
    Participant

    P.S. I started deleting Doom, duke3d, apple2, x86, and cavestory so that the gui loads only the emulators/roms the user has access to since I’m doing a usb controller only interface. This leaves the input config left, which someone could accidentally run and reset the game-exit. Does anyone know how to disable the input config screen? Thanks!

    To get rid of the Input Config screen, do the following from either the command line (close down EmulationStation by pressing F4) or by SSH shell:

    – sudo nano /home/pi/.emulationstation/es_systems.cfg
    – go to the very bottom of the file and add comment hashes like this:

    DESCNAME=Input Configuration
    NAME=esconfig
    PATH=/home/pi/RetroPie/roms/esconfig
    EXTENSION=.sh .SH
    COMMAND=/home/pi/RetroPie/roms/esconfig/Start.sh

    to this:

    #DESCNAME=Input Configuration
    #NAME=esconfig
    #PATH=/home/pi/RetroPie/roms/esconfig
    #EXTENSION=.sh .SH
    #COMMAND=/home/pi/RetroPie/roms/esconfig/Start.sh

    – hit ctrl+x
    – type ‘y’ then hit enter
    – done!

    This should kick you back out to the command line. You can always re-enable the input config by removing the hashtags.

    #14692

    Marlboro Gold Regular Cigarettes

    Excellent read, I just passed this onto a colleague who was doing a little research on that. And he actually bought me lunch because I found it for him smile So let me rephrase that.

    #14713
    Profile photo of InsecureSpike
    InsecureSpike
    Participant

    right now I’m starting with GBC, NES, and supernintendo. I’m just downloading the roms from romnation that are marked with “” and “[!]” What are the romsets and xml’s? Thanks!

    mate, please DON’T mention rom sites, I’m sure this is in a post somewhere Florian posted a few weeks ago,

    Sent from my iPhone using Tapatalk

    #73730
    Profile photo of kingvoxel
    kingvoxel
    Participant

    I was not able to get rid of the annoying Input Config Screen. I am running ES2.0.0-RC1. In my es_systems.cfg file I only saw meta data for various eemulators. I didnt see the following:
    DESCNAME=Input Configuration
    NAME=esconfig
    PATH=/home/pi/RetroPie/roms/esconfig
    EXTENSION=.sh .SH
    COMMAND=/home/pi/RetroPie/roms/esconfig/Start.sh

    Are you using the same version of Emulation Station as me?

    #81571
    Profile photo of amccabe126
    amccabe126
    Participant

    Making the xml lists is by far the most time-consuming part. You have to keep an eye on the scraper and catch ones that don’t register properly. Sometimes change I,II,III,IV to 1,2,3, or 4. Once you have the xml lists, the roms, the image files, etc. remaking the system only takes like 3-5 hours for me. I’m not into playing games since I don’t have the time but I like the challenge of putting this thing together. Sorry didn’t realize the rule about website names. Kingvoxel, I used the emulation station that came with the retropie image that was the latest back around march. I’m not sure if this would have been changed but there should be a text file that you can just add comment lines to take the input out. I recommend this if you don’t want the hassle of having to reconfigure your controls whenever you accidentally load that screen. I also figured out that the images for boxart take up different portions of the screen when run on different resolutions (720, 1080, etc.) and that there is a quick way of modifying the images to conform in terms of height vs width ratios which helps with the aesthetics. One problem I haven’t solved is that sometimes my buffalo gamepads will send a random signal to the pie which will essentially be a random d-pad signal, and when you have a long list that you are scrolling through this can be annoying. I was also able to combine an atrix 4g lapdock, some imported adapters and wires with this retropie to make a portable battery operated gaming system. PS does anyone know how to delete old posts? I couldnt find them listed in any of the profile stuff. Thanks.

Viewing 35 posts - 1 through 35 (of 35 total)

Forums are currently read only - please visit the new RetroPie forums at https://retropie.org.uk/forums/

Skip to toolbar