Scraper: little details to solve…

Home Forums RetroPie Project Ideas for Further Enhancements Scraper: little details to solve…

RetroPie has a new website and forum. Please visit https://retropie.org.uk/ for the new site. The new forum is located at https://retropie.org.uk/forum/. This forum is left here as a read-only archive.

This topic contains 17 replies, has 8 voices, and was last updated by Profile photo of sselph sselph 1 year, 3 months ago.

Viewing 18 posts - 1 through 18 (of 18 total)
  • Author
    Posts
  • #109067
    Profile photo of jordimayor
    jordimayor
    Participant

    First of all, my intention is getting better scrapping. It’s not criticism to the awesome and well-done job from sselph <congratulations, mate!!>

    When call scrap a game from metadata into emulationstation, they save image in the folder called .emulationstation/downloaded_images. But when I scrape a system, I go to console and roms folder and then run scraper -thumb_only that generates a gamelist.xml file and the images folder. That is a problem, because when you pack a system for backup purposes it has imagefiles in two diferent folders.

    It is possible a bad config, but I’ve recently installed retropie 3.2.1 from scratch… I’ve only installed the scraper 1.04 from console…

    Thanks in advance, and please, any suggestion will be welcome, because I’m not sure to be wrong…

    #109072
    Profile photo of sselph
    sselph
    Participant

    The current version of emulationstation looks for the gamelist.xml file first in the rom folder and if it doesn’t exist it looks in the ~/.emulationstation/gamelists/(system) folder and finally /etc/emulationstation/gamelists/(system). The images can be anywhere since the gamelist tells ES where to look.

    I prefer the gamelist and images in the rom folder since it makes it easier to copy files to and from the retropie. I actually wrote the scraper to run on my desktop and then I copied the files over to make upgrading systems easier.

    ES’s author, Aloshi, has the opposite opinion and prefers that the gamelist and images not clutter up the rom folder so the internal scraper places things in the ~/.emulationstation folder.

    My script can be invoked with command line flags to tell it where to find the roms and where you will be placing the gamelist and images. This is how it works when my script is invoked from the retropie-setup script. But keep in mind the gamelist can only be in one of three places if you expect ES to find it.

    Something like this(with (system) replaced by the system) but you need to double check the paths since I’m doing this from memory. The image path is specified twice since this script was originally designed to be run from a different system, this allowed the host and target system to have a different folder location.

    scraper -output_file ~/.emulationstation/gamelists/(system)/gamelist.xml -img_dir ~/.emulationstation/downloaded_images/(system) -img_path ~/.emulationstation/downloaded_images/(system) -rom_dir ~/retropie/roms/(system)

    Auto-scraper: https://github.com/sselph/scraper

    #110013
    Profile photo of cyborguk
    cyborguk
    Participant

    Sselph,

    I’m having a little issue getting your scraper working. I’ve ran through the install guide on github.com and my pi2 has downloaded release version v1.0.4.

    I’ve scrapped the Megadrive and Snes rom folders and it created the image folders which have many thumbnail size pictures but the gamelist.xml that got created in the roms fokder is empty.

    For mame the folder again it created the images folder, but this time no gamelist.xml file.

    Is there a command I can run to generate the gamelist.xml properly?

    Not sure why it’s gone wrong.

    I’m using Retropie 3.2.1.

    Thanks

    #110029
    Profile photo of robertybob
    robertybob
    Participant

    Sselph,

    I’m having a little issue getting your scraper working. I’ve ran through the install guide on github.com and my pi2 has downloaded release version v1.0.4.

    I’ve scrapped the Megadrive and Snes rom folders and it created the image folders which have many thumbnail size pictures but the gamelist.xml that got created in the roms fokder is empty.

    For mame the folder again it created the images folder, but this time no gamelist.xml file.

    Is there a command I can run to generate the gamelist.xml properly?

    Not sure why it’s gone wrong.

    I’m using Retropie 3.2.1.

    Thanks

    This is happening for me now on any system – started doing this a little over a week ago.

    #110032
    Profile photo of cyborguk
    cyborguk
    Participant

    Must be a bug on 3.2.1?

    #110039
    Profile photo of sselph
    sselph
    Participant

    Hi cyborguk,

    The gamelist.xml is placed in the current working directory. This is the same directory where it is looking for roms and placing images so if those are working it should be fine. You can try adding -output_file /home/pi/RetroPie/roms/(rom_dir)/gamelist.xml to make it explicit. Permissions don’t seem to be the issue since it is able to create an images folder and write to it and it should give an error in that case. The other option is the scraper didn’t find any hashes which doesn’t appear to be the case since it downloaded images. I haven’t changed anything in my script in over a month except adding and removing some hashes in my database.

    Auto-scraper: https://github.com/sselph/scraper

    #110042
    Profile photo of sselph
    sselph
    Participant

    I just saw another message where someone said the gamelist was placed in their home directory. Check to see if that is the case, if so that would be strange and I’ll have to spend some time diagnosing that.

    Auto-scraper: https://github.com/sselph/scraper

    #110052
    Profile photo of robertybob
    robertybob
    Participant

    This was happening to me using the Windows version of the scraper, I’ll try scraping using my Pi tonight and see whether it makes a difference 🙂

    *edit* appears to be only certain Sega CD games which it affects (strange!).

    • This reply was modified 1 year, 5 months ago by Profile photo of robertybob robertybob.
    #110053
    Profile photo of cyborguk
    cyborguk
    Participant

    Let me check tonight.

    Where do I add

    -output_file /home/pi/RetroPie/roms/(rom_dir)/gamelist.xml

    This seems to be where mine is but the file is pretty much empty.

    #110069
    Profile photo of cyborguk
    cyborguk
    Participant

    Ran this:

    pi@retropie ~/RetroPie/roms/megadrive $ scraper -thumb_only -output_file /home/pi/RetroPie/roms/megadrive/gamelist.xml

    No luck, images folder is created but no gamelist.xml

    #110211
    Profile photo of cyborguk
    cyborguk
    Participant

    Anyone managing to get it to work under 3.2.1? I’m at a loss now with it sadly.

    #110372
    Profile photo of cyborguk
    cyborguk
    Participant

    Formatted another SD card and installed 3.2.1 on it and put a few games in the Megadrive folder and installed the Sselph scraper manually as the experimental one doesn’t work and this time it worked.

    So my main SD card doesn’t work, what could be the issue?

    #110717
    Profile photo of Soullous
    Soullous
    Participant

    The issue seems to be that gamelist.xml is being overwritten with whatever the built-in scraper created every time EmulationStation is restarted. It doesn’t matter if gamelist.xml is saved in the rom directory or in the ~/.emulationstation/gamelist/(system) directory. They both get overwritten. I’m not exactly sure where the file is coming from that’s overwriting it, though…

    #110776
    Profile photo of Soullous
    Soullous
    Participant

    OK, so I figured out my issue. Not sure if it’s the same issue that others are having, but, I came across this:
    https://github.com/RetroPie/RetroPie-Setup/issues/807

    Emulationstation seems to load all gamelists at startup and writes them back at exit. So if emulationstation is running and you modify a gamelist any modification will be overwritten.

    So, it appears that EmulationStation cannot be running while you run the scraper. If you’re running it from SSH while EmulationStation is up, or if you try to run it from the RetroPie setup script / Experimental through EmulationStation it will scrape everything, but your gamelist.xml will get overwritten. So you need to shut down EmulationStation and drop to the console, run scraper (or sudo ~/RetroPie-Setup/retropie_setup.sh) from the command-line, and then when you restart EmulationStation again, everything should load properly.

    Ideally, the scraper should be able to handle this by telling EmulationStation to reload the gamelist somehow. I’m assuming it’s possible, since the built-in scraper can do it. In the meantime, I’d suggest adding this bit of info to the README. Excellent job on the scraper, though. Works much better than the built-in one. I’d love to see this become the default, or even perhaps just merging the searching algorithm in with the default one.

    • This reply was modified 1 year, 5 months ago by Profile photo of Soullous Soullous.
    #110778
    Profile photo of herbfargus
    herbfargus
    Moderator

    The end goal is to have it incorporated into emulationstation but that just depends on how much free time sselph has, in the meantime I think it would be wise for us to kill emulationstation in the script before the scraper is loaded from the retropie setup script as to not run into the same issues mentioned above.

    #110792
    Profile photo of buzz
    buzz
    Keymaster

    if you exit ES with f4 I think it doesn’t save any gameslists. I might look at changing the exit so you have exit + save and exit without saving options or so

    #114062
    Profile photo of zigurana
    zigurana
    Participant

    if you exit ES with f4 I think it doesn’t save any gameslists. I might look at changing the exit so you have exit + save and exit without saving options or so

    I think this is correct.
    While we are used to think that the gamelist.xml files are the endpoint, for ES they are of course only the stored values of an internal construct (in working memory) that contains all metadata of a given system. So when a certain system is exited (you go back to the system overview), it will store the current version of that internal struct in our newly scraped xml file. It does this to update the values for “# times played” etc.
    The build-in scraper alters the internal info, which is then written to file as regular.

    I would be willing to try and see what needs to happen in order to tie sselph’s scraper into ES nicely.
    Not sure if actually making it part of ES would be the way to go, though. That would make further development for the scraper more involved, and we would lose flexibility. Maybe it’s better to let the scraper write to a second set of xml files, which are then somehow imported into ES?

    #114064
    Profile photo of sselph
    sselph
    Participant

    Hi zigurana,

    Yeah this is most likely what is happening.

    As for adding my code to ES, I have submitted https://github.com/Aloshi/EmulationStation/pull/355 to ES and it is in their unstable branch. This code is somewhat out of date but it is there. Now that Go can create C modules it might be possible to refactor my code or create a small wrapper to allow ES to import it as an optional library so that it is kept more up to date. If you are a programmer and want to look in to this feel free and I’ll be happy to answer any questions.

    The other solution to this is that ES was moving to using sqlite for storage and I already have some unchecked in code to also use sqlite. When this happens ES would no longer be dumping everything to the xml on exit which would probably solve the issue.

    Auto-scraper: https://github.com/sselph/scraper

Viewing 18 posts - 1 through 18 (of 18 total)

Forums are currently read only - please visit the new RetroPie forums at https://retropie.org.uk/forums/

Skip to toolbar