Scraper not finding some games

Home Forums RetroPie Project New to RetroPie? Start Here! Scraper not finding some games

RetroPie has a new website and forum. Please visit https://retropie.org.uk/ for the new site. The new forum is located at https://retropie.org.uk/forum/. This forum is left here as a read-only archive.

This topic contains 25 replies, has 6 voices, and was last updated by Profile photo of cyborguk cyborguk 1 year, 5 months ago.

Viewing 26 posts - 1 through 26 (of 26 total)
  • Author
    Posts
  • #108858
    Profile photo of cyborguk
    cyborguk
    Participant

    Hello,

    Fully appreciate the games db can’t have all games, however can I manually update games that have no images?

    #108862
    Profile photo of herbfargus
    herbfargus
    Moderator

    You can submit missing images to thegamesdb so others can benefit if the gameadb doesn’t have an image.

    You can also transfer over your image and then press shift (or select, I can’t remember) once you’ve navigated to the game and edit the image path manually to where you transferred your image over. Just note that you’ll have to select quit from the emulationstation start menu If you want those manual changes saved.

    Otherwise you can manually edit the gamelist.xml for that system which accomplished the same thing.

    #108865
    Profile photo of cyborguk
    cyborguk
    Participant

    Where is the gamelist.xml held, can’t locate it? I was hoping to copy the images to the default location to test first.

    If all looks OK I can upload to the gamesdb, I will Google their site.

    #109073
    Profile photo of sselph
    sselph
    Participant

    If you used the builtin scraper or my scraper via the retropie-setup script the gamelist is in ~/.emulationstation/gamelists/(system).

    If you used my command line scraper it is in the rom folder.

    Auto-scraper: https://github.com/sselph/scraper

    #109080
    Profile photo of nosedeath
    nosedeath
    Participant

    sselph’s scraper is AMAZING!! Use it. so much quicker. It had just about all of them too.

    Thanks sselph!

    #109095
    Profile photo of cyborguk
    cyborguk
    Participant

    I’m checking this out tonight!!

    Thanks, I have lots of C64 games to do.

    #109096
    Profile photo of sselph
    sselph
    Participant

    I have an issue open to add C64 support and just haven’t had the time to learn about the file formats involved.

    Auto-scraper: https://github.com/sselph/scraper

    #109097
    Profile photo of cyborguk
    cyborguk
    Participant

    Oh ok, would be nice to see a scraper take hold of the Amiga, C64 and Spectrum one day.

    #109118
    Profile photo of cyborguk
    cyborguk
    Participant

    Sorry where in retropie-setup script do I install this?

    #109125
    Profile photo of chito
    chito
    Participant

    I dont understand how there arent complete gamelists for retropie with all the meta data and images. complete packs… sselph’s tool matches roms to data by Hash… if a tool like his had its own complete db with images and data specifically for ES we would be set. I get that this is the idea behind scrapping these sites but im saying lets remove that dependancy. even have a community driven open contribution wiki or something.

    #109126
    Profile photo of redhalo
    redhalo
    Participant

    Well, there isn’t complete info for all games. These sites being scraped are still constantly being updated. A “community driven open contribution wiki” is exactly what these sites are. The solution you describe is exactly what’s in place.

    #109129
    Profile photo of herbfargus
    herbfargus
    Moderator

    I think it’s better to have a scraper than complete packs as a lot images can take up space, so why add images you don’t need?

    #109130
    Profile photo of chito
    chito
    Participant

    we really should make a solution specific to retropie/es. scrapping is a messy slow solutions and less than ideal should the site change the scraper will break. Also we could add support to select different images packs (screen shoot, title, poster, cartridge, boxart) and added easily.

    • This reply was modified 1 year, 5 months ago by Profile photo of chito chito.
    #109132
    Profile photo of chito
    chito
    Participant

    or maybe thegamesdb can create a public API… otherwise we can create our own data base of meta data. and that database would only need to scrap data (data mine) off these sites maybe a few times a year.

    #109133
    Profile photo of chito
    chito
    Participant

    I think it’s better to have a scraper than complete packs as a lot images can take up space, so why add images you don’t need?

    the solution im suggesting would use a complete pack and then another tool like sselph’s could customize your gamelists for what it finds in your folders and only transfer or use what is relevant

    #109134
    Profile photo of redhalo
    redhalo
    Participant

    The “complete pack” is thegamesdb, the scraper is what we have now. You are literally describing the current use case.

    #109135
    Profile photo of chito
    chito
    Participant

    The “complete pack” is thegamesdb, the scraper is what we have now. You are literally describing the current use case.

    actually its not, switching from a scrapping solution to an api is wayyyyyyyy different and way more efficient, faster and customizable

    #109137
    Profile photo of redhalo
    redhalo
    Participant

    http://wiki.thegamesdb.net/index.php/API_Introduction

    thegamesdb.net has an api, sselph’s scraper uses the api.

    #109140
    Profile photo of chito
    chito
    Participant

    I see, well thats a relief lol, so why is it called a scrapper? its not scrapping.

    I hope they add support for the api to return your choice of game artwork (boxart, screenshot, title etc)

    #109141
    Profile photo of sselph
    sselph
    Participant

    Sorry where in retropie-setup script do I install this?

    Sorry I haven’t actually used this from the menu to see but I’ve heard people say the experimental area. Should say something like “Scraper for EmulationStation by Steven Selph”. Herbfargus would know.

    actually its not, switching from a scrapping solution to an api is wayyyyyyyy different and way more efficient, faster and customizable

    My scraper(for console roms) maps the contents of the rom(via hashing) with the ID in thegamesdb to query the API. It then formats the API response in the format suitable for the XML. This is a fairly efficient method for the systems I have built the mappings for since it only downloads what is needed and doesn’t perform any unneeded search API calls.

    There is a project that is attempting to do something like you mention. It relies on the file naming to be done according to the no-intro naming. They used my scraper to seed much of the information.
    https://github.com/digitalLumberjack/recalbox-gamelists

    Auto-scraper: https://github.com/sselph/scraper

    #109144
    Profile photo of chito
    chito
    Participant

    How hard is it to add a new system like C64 for cyborguk? I personally would love to have Neogeo pocket. I was looking at creating my xml manually 🙁

    Using the hash is way better cause i use no-intro and datomatic to create 1 game 1 rom sets. so the matching of hash to id you would’ve had to have done that manually? or did you create a tool that tried to find a match and you confirm (kinda like the way ES scraper works) either way thats a huge amount of work really appreciate it sselph.

    for arcade games (in particular mame) could you use the mameinfo dat files to create the xml gamelists? images could still be retrieved from thegamesdb

    #109145
    Profile photo of sselph
    sselph
    Participant

    so why is it called a scrapper? its not scrapping.

    That is what Aloshi called it in ES so I just used the same terminology. It does actually scrape HTML for MAME info.

    I hope they add support for the api to return your choice of game artwork (boxart, screenshot, title etc)

    The API returns all the images that are available. boxart is almost always there where screenshot, etc are less common. My script, if run from the command line, you can accept the flags -mame_img or -gdb_img depending on which system type you are processing. You can see how to use these with -help

    How hard is it to add a new system…So the matching of hash to id you would’ve had to have done that manually? or did you create a tool

    To add a new system I have to find a set of hashes for the rom set. Typically from no-intro. I have to determine how the rom data is formatted. Some systems like nes have headers which aren’t included in the hash. Some systems like megadrive and n64 have multiple encodings for the same data so I have to convert them on the fly to the correct format. Once I’ve determined how to hash a file, I started mapping hash to ID with the help of some scripts and spreadsheets. So it has some automation but still very manual.

    for arcade games (in particular mame) could you use the mameinfo dat files to create the xml gamelists?

    For arcade platforms I am not doing the best job. I use the name since that is a good key for these systems then I scrape mamedb.com/game/(name) plus some info from a history file. I should try and map the name to the correct name for the mame version, but what I have has been good enough for most people.

    Auto-scraper: https://github.com/sselph/scraper

    #109149
    Profile photo of chito
    chito
    Participant

    I started mapping hash to ID with the help of some scripts and spreadsheets. So it has some automation but still very manual.

    I tip my hat to you. Thats heaps of work. Well Im very keen to complete the NGP “scrapping”. I was going to edit my gamelist.xml manually but happy to help add this system to your tool if you like.

    #109151
    Profile photo of cyborguk
    cyborguk
    Participant

    Sselph I will look again tonight, but I did look there and couldn’t see it, it was at the end of a long day at work so my eyes were not the best.

    #109158
    Profile photo of herbfargus
    herbfargus
    Moderator

    It’s a hidden folder so you may have to make hidden folders visible to see the gamelist xmls.

    Also just for reference:

    https://github.com/RetroPie/RetroPie-Setup/wiki/scraper

    #109205
    Profile photo of cyborguk
    cyborguk
    Participant

    I managed to install from the experimental menu and it also asked which emulator to search and scrape, which I did for the NeoGeo and PSX, but not images show up, it did scan far too quickly too, did I miss a step?

Viewing 26 posts - 1 through 26 (of 26 total)

Forums are currently read only - please visit the new RetroPie forums at https://retropie.org.uk/forums/

Skip to toolbar