Scraping metadata in RetroPie 3 – sselph scraper

Home Forums RetroPie Project Everything else related to the RetroPie Project Scraping metadata in RetroPie 3 – sselph scraper

RetroPie has a new website and forum. Please visit https://retropie.org.uk/ for the new site. The new forum is located at https://retropie.org.uk/forum/. This forum is left here as a read-only archive.

This topic contains 39 replies, has 6 voices, and was last updated by Profile photo of charco charco 1 year, 10 months ago.

Viewing 35 posts - 1 through 35 (of 40 total)
  • Author
    Posts
  • #98257
    Profile photo of Floob
    Floob
    Moderator

    I’m sure most people know about this tool now, but here is a mini update to it, showing how to get that extra game data scraped so Emulation Station looks nicer.

    (May 2015)

    RetroPie help guides --> https://goo.gl/Yfy8kj
    Please read this before asking for help --> http://goo.gl/eLErnl

    #98258
    Profile photo of Omnija
    Omnija
    Participant

    i’ve used this method before but was sort of annoyed with how it renames the roms by adding (usa), (europe) etc. Was their a way to scrap it with out it changing the rom titles?

    #98259
    Profile photo of Floob
    Floob
    Moderator

    You could use this flag if that helps

    -use_filename=false: If true, use the filename minus the extension as the game title in xml.

    RetroPie help guides --> https://goo.gl/Yfy8kj
    Please read this before asking for help --> http://goo.gl/eLErnl

    #98261
    Profile photo of charco
    charco
    Participant

    I downloaded this scraper after watching one of your Youtube videos Floob, after I had scraped using the the ./scraper -thumb_only command I started to experience Emulation Station lockups. I think this must be connected as I have done a fresh install on a new SD card this evening and not scraped any metadata and have experienced no issues with RetroPie.

    Was this version just released today?

    #98262
    Profile photo of Floob
    Floob
    Moderator

    The latest version is 5 days old (18th May 2015).
    Although I wouldnt have thought previous versions would have caused Emulation Station lockups.

    Perhaps try scraping one system at a time and see the lockups happen again.
    Do make sure Emulation Station has quit before scraping.

    RetroPie help guides --> https://goo.gl/Yfy8kj
    Please read this before asking for help --> http://goo.gl/eLErnl

    #98263
    Profile photo of charco
    charco
    Participant

    Yes, I saw that in your video. I will make sure to check that ES has stopped before scraping. Thanks!

    I think this tool would also be a great addition to the RetroPie settings in Emulation Station.

    • This reply was modified 1 year, 11 months ago by Profile photo of charco charco.
    #98318
    Profile photo of Omnija
    Omnija
    Participant

    You could use this flag if that helps

    -use_filename=false: If true, use the filename minus the extension as the game title in xml.

    Thanks floob

    • This reply was modified 1 year, 11 months ago by Profile photo of Omnija Omnija.
    #98711
    Profile photo of braincell
    braincell
    Participant

    I too am having lockups after scraping with this tool.

    It lets me play 3 or 4 games then locks up , and it has also locked up on the system selection screen too ??

    have tried with rpi 1 and rpi 2 with different psus and different sd cards ….. its definately the scaping.

    i even copied the rom folders to a usb and did a fresh install then allowed ES to copy the roms back into the installation and it locks up again ?????

    what is causing this ???????

    #98712
    Profile photo of Floob
    Floob
    Moderator

    I’m presuming you scrape when emulation station isnt running to ensure there wont be any corrupt gamelist files?

    RetroPie help guides --> https://goo.gl/Yfy8kj
    Please read this before asking for help --> http://goo.gl/eLErnl

    #98718
    Profile photo of charco
    charco
    Participant

    What is the command to stop Emulation Station from running? I might try scraping again if that is guaranteed to be the safest way.

    #98721
    Profile photo of braincell
    braincell
    Participant

    Emulationstation is definately not running .

    As i said i even copied the rom folders onto a usb stick.

    I read through the xml files and couldnt aee any errors.

    What i dont understand is the freezing on the system selection screen ???????

    Its only a handful of xml files and a lot of thumbnail images. So is it a memory leak?

    Are the xml files parsed at startup or during sustem selection??

    Why does it take a few games to freeze up ????

    #98722
    Profile photo of Floob
    Floob
    Moderator

    Could it be related to this?
    http://blog.petrockblock.com/forums/topic/emulation-station-crashes-where-are-the-log-files/#post-98310

    RetroPie help guides --> https://goo.gl/Yfy8kj
    Please read this before asking for help --> http://goo.gl/eLErnl

    #98725
    Profile photo of braincell
    braincell
    Participant

    But without scraping wouldnt the filename still xause crashes ??

    #98726
    Profile photo of Floob
    Floob
    Moderator

    Maybe its the way ES processes the gamelist.xml?
    May be worth asking in the ES forums as well.

    RetroPie help guides --> https://goo.gl/Yfy8kj
    Please read this before asking for help --> http://goo.gl/eLErnl

    #98728
    Profile photo of charco
    charco
    Participant

    Yeah mine was locking up on the system selection screen aswell.

    #98729
    Profile photo of braincell
    braincell
    Participant

    Ok… this is rediculous . Im going to bulk rename leaving just filename then scrape from windows …. copy to usb and see what happened

    #98755
    Profile photo of Omnija
    Omnija
    Participant

    Any idea if there will be support for neogeo pocket and wonderswan?
    Doing it manual is a pain.

    #98758
    Profile photo of Floob
    Floob
    Moderator

    They are listed here (source db), so I guess it should be possible:
    http://thegamesdb.net/platforms/

    RetroPie help guides --> https://goo.gl/Yfy8kj
    Please read this before asking for help --> http://goo.gl/eLErnl

    #98770
    Profile photo of charco
    charco
    Participant

    I wonder does the issue only occur when there are a large amount of roms. I am using full rom sets on a 64GB SD card, Floob, I think you are only scraping a fairly small amount and are not experiencing the issue?

    #98778
    Profile photo of braincell
    braincell
    Participant

    Ok so i fresh installed my sd…..

    Renamed ALL my roms so my filenames only contain letters and numbers…..

    Unzipped all roms except mame ……

    Rescraped on the pi itself……

    Rebooted went through mame file without playing anything….. so far so good…..

    Went into atari 2600 and got down about 20 roms and ……. FREEZE.

    FFS.

    #98780
    Profile photo of Omnija
    Omnija
    Participant

    which pi version are you using?

    #98792
    Profile photo of braincell
    braincell
    Participant

    Both rpi 1 and 2 on 3.0beta

    #98798
    Profile photo of braincell
    braincell
    Participant

    OK ,yet ANOTHER clean install , this time did apt – update / upgrade , STILL SAME HAPPENS…… Although this time i loaded 6 games before lockup

    What is going on!!!!!!!!!!!!!!

    #98801
    Profile photo of Floob
    Floob
    Moderator

    Is this happening even if you just put, say SNES roms, in place, and dont copy other systems roms yet.
    Just wondering if it locks up with one system?

    RetroPie help guides --> https://goo.gl/Yfy8kj
    Please read this before asking for help --> http://goo.gl/eLErnl

    #98806
    Profile photo of Omnija
    Omnija
    Participant

    They are listed here (source db), so I guess it should be possible:
    http://thegamesdb.net/platforms/

    For some reason it wasn’t finding games yesterday but it is now.

    #98815
    Profile photo of sselph
    sselph
    Participant

    I can’t think of anything the script would be doing that would cause things to freeze, but I haven’t updated to the new version yet. Does this only happen if you use my scraper or does it happen with the builtin scraper as well?

    Using a gamelist.xml file does cause it to use more memory but the rpi2 has plenty so there might a leak somewhere. Have you checked the ES log, think it is at ~/.emulationstation/es_log.txt

    Auto-scraper: https://github.com/sselph/scraper

    #98818
    Profile photo of braincell
    braincell
    Participant

    Ok …. without scraping it doesnt freeze.

    When it freezes a SSH connection is still possible but theres nothing in the log file to indicate why.

    Top says theres still 600megs ram available :/

    Now copying gamelist files into emulationstations gamelist dirs….

    Nope. That broke it now it wont startup. Time for yet another reinstall……

    #98823
    Profile photo of braincell
    braincell
    Participant

    Didnt need a reinstall just a reboot but still crashes…..

    #98834
    Profile photo of braincell
    braincell
    Participant

    Deleted gamelist.xml files (left ./image folders including thumbs) and no longer crashes (or displays pix)…..

    #98835
    Profile photo of Floob
    Floob
    Moderator

    I’m assuming you are scraping with
    “scraper -thumb_only” ?

    RetroPie help guides --> https://goo.gl/Yfy8kj
    Please read this before asking for help --> http://goo.gl/eLErnl

    #98836
    Profile photo of braincell
    braincell
    Participant

    Yes thats the settings i used

    #98839
    Profile photo of charco
    charco
    Participant

    My lockups stopped when I deleted the gamelist.xml files via ssh aswell. Not scraping anything until this bug is resolved anyway.

    • This reply was modified 1 year, 10 months ago by Profile photo of charco charco.
    #100839
    Profile photo of charco
    charco
    Participant

    Trying this again and now I get

    -bash: scraper: command not found

    EDIT: I downloaded from command line to the RPI2 and it seems to be working now.

    • This reply was modified 1 year, 10 months ago by Profile photo of charco charco.
    #100908
    Profile photo of charco
    charco
    Participant

    Lucky I backed up my image yesterday, scraped for full romsets and everything working like a dream until I tried to scrape SuperGrafx, Neo Geo Pocket, and Virtual Boy. ES won’t stop freezing, even after I delete gamelists over ftp. Formatted my SD card and will start again. Lost a whole days work. If I had not backed up before scraping the Pi would be in the bin!

    #100927
    Profile photo of robertybob
    robertybob
    Participant

    Lucky I backed up my image yesterday, scraped for full romsets and everything working like a dream until I tried to scrape SuperGrafx, Neo Geo Pocket, and Virtual Boy. ES won’t stop freezing, even after I delete gamelists over ftp. Formatted my SD card and will start again. Lost a whole days work. If I had not backed up before scraping the Pi would be in the bin!

    Pretty sure having too many systems can cause Emulationstation to lockup – how can you be sure that’s not the problem?

Viewing 35 posts - 1 through 35 (of 40 total)

Forums are currently read only - please visit the new RetroPie forums at https://retropie.org.uk/forums/

Skip to toolbar