New version of sselph/scraper v0.9.0-beta

Welcome Page Forums RetroPie Project Everything else related to the RetroPie Project New version of sselph/scraper v0.9.0-beta

This topic contains 58 replies, has 20 voices, and was last updated by Profile photo of sselph sselph 1 year, 3 months ago.

Viewing 24 posts - 36 through 59 (of 59 total)
  • Author
    Posts
  • #114832
    Profile photo of sselph
    sselph
    Participant

    You can use the -missing missing.csv that will write all files that weren’t scraped to a file with an explanation. Either the hash was not in no-intro, the game matched but was missing in thegamesdb when I did the last matching, or the game is in thegamesdb but no image was found. You can open that file and add a column with thegamesdb ID and I can add those to the master csv.

    If you think data in the existing master csv could be helpful you can download it from:
    https://storage.googleapis.com/stevenselph.appspot.com/hash.csv.gz

    Auto-scraper: https://github.com/sselph/scraper

    #114908
    Profile photo of proxycell
    proxycell
    Participant

    Thanks a lot Steven!

    #117734
    Profile photo of vretro
    vretro
    Participant

    Is it possible to scrape a folder and use custom images which are in the /images folder (as long as they are named the same as each item)? – Also being able to use this technique on an unsupported system?

    #117742
    Profile photo of robertybob
    robertybob
    Participant

    Is it possible to scrape a folder and use custom images which are in the /images folder (as long as they are named the same as each item)? – Also being able to use this technique on an unsupported system?

    https://github.com/sselph/scraper/wiki/Flags

    Presumably this is what you’re looking for?

    -download_images
    If false, don’t download any images, instead see if the expected file is stored locally already. (default true)

    #117745
    Profile photo of vretro
    vretro
    Participant

    Bingo! Thank you Robertybob. Making it work on unsupported systems, using “-download_images false” would be my feature request then I suppose.

    #117746
    Profile photo of robertybob
    robertybob
    Participant

    Bingo! Thank you Robertybob. Making it work on unsupported systems, using “-download_images false” would be my feature request then I suppose.

    I’ve not tried it myself, but maybe

    -add_not_found -download_images false
    could work? The first flag, it seems, will add an entry to the gamelist.xml with no metadata but perhaps with the image if it’s found (?)

    *edit* I wonder whether this may be of use:

    -use_gdb false

    Use the hash.csv and theGamesDB metadata. (default true)

    I’m just speculating here, but if the games are from an unsupported system then the hash.csv won’t have any entries for them, so maybe if this flag is set to false it will still add the games to the .xml in addition to the other flags above?

    #117749
    Profile photo of sselph
    sselph
    Participant

    Yes -add_not_found -download_images=false and if the scraper doesn’t know about the extension involved you can do -extra_ext=".new" Depending on the name of your files you may need to do -image_suffix="" to tell it to name the images rom.jpg instead of rom-image.jpg

    Auto-scraper: https://github.com/sselph/scraper

    #117831
    Profile photo of vretro
    vretro
    Participant

    Thank you Sselph!

    I will try those extensions again soon, there seemed to be server issues when I first tried. These additional extensions are really handy.

    #118528
    Profile photo of mferraci
    mferraci
    Participant

    Hi Sselph, thanks for your work, I really like your scraper.

    You can open that file and add a column with thegamesdb ID and I can add those to the master csv.

    I want to improve the hash list with some psone game of my collection. Do you use the .cue hash or the first .bin of each game ?

    Mathieu.

    #118920
    Profile photo of g0nz0uk
    g0nz0uk
    Participant

    Great work Sselph. One question are there higher resolution pictures available, I use the nbba theme so images are large on the right but in mame the quality is very low, I used the thumbs only switch, should I delete the images and try without the thumbs switch?

    #118961
    Profile photo of robertybob
    robertybob
    Participant

    Great work Sselph. One question are there higher resolution pictures available, I use the nbba theme so images are large on the right but in mame the quality is very low, I used the thumbs only switch, should I delete the images and try without the thumbs switch?

    Yes, thumbs is for downloading low-res images – this is incase the Pi suffers a slowdown due to having to display large images with themes. You may find each image takes a second or two to load.

    #118962
    Profile photo of robertybob
    robertybob
    Participant

    Hi Sselph, thanks for your work, I really like your scraper.

    You can open that file and add a column with thegamesdb ID and I can add those to the master csv.

    I want to improve the hash list with some psone game of my collection. Do you use the .cue hash or the first .bin of each game ?

    Mathieu.

    I believe it’s both .cue and all .bin files

    #119006
    Profile photo of sselph
    sselph
    Participant

    Great work Sselph. One question are there higher resolution pictures available, I use the nbba theme so images are large on the right but in mame the quality is very low, I used the thumbs only switch, should I delete the images and try without the thumbs switch?

    This is because the images at mamedb are low resolution.

    I want to improve the hash list with some psone game of my collection. Do you use the .cue hash or the first .bin of each game ?

    The .cue and the main(largest) .bin should work. I used to do all the bins but there are some duplicates across games so the main bin file should work.

    Auto-scraper: https://github.com/sselph/scraper

    #119162
    Profile photo of g0nz0uk
    g0nz0uk
    Participant

    Hello,

    I’ve scraped mame not used the thumb_only switch but the images are quite low quality, around 14kb per image, how do I get them into a better resolution?

    Thanks

    #119694
    Profile photo of pausuri
    pausuri
    Participant

    Hi Sselph, is there any way to prevent adding the region to every single rom?

    Cheers

    #119700
    Profile photo of sselph
    sselph
    Participant

    The default is to use the no intro name which includes the region. You can do -use_nointro_name=false to use the name from thegamesdb which doesn’t include a region.

    Auto-scraper: https://github.com/sselph/scraper

    #119735
    Profile photo of zerojay
    zerojay
    Participant

    g0nzOuk: The answer is on this very page.

    #120846
    Profile photo of neighbourhoodnerd
    neighbourhoodnerd
    Participant

    Hi SSelph!

    I ran the scraper today via RetroPie-Setup and it did a great job. Much faster than the inbuilt scraper and produced far more results.

    Two quick questions:
    1. I want to run the scraper from the terminal instead of using the RetroPie-Setup GUI so I can add arguments. Running the command scraper doesn’t seem to work though.
    2. All my multi-disk games fail to scan. Here’s an example of how they are named in my library: Final Fantasy VII [disk 1 of 3].ISO. Is there any easy way to scrape these? I don’t mind renaming them but naturally I need a way to know which disk is which.

    #120870
    Profile photo of sselph
    sselph
    Participant

    Doesn’t seem to work isn’t very descriptive, can you provide more information?

    The scraper works for multi-disc games but doesn’t work for iso files since redump hashes are for bin/cue.

    Auto-scraper: https://github.com/sselph/scraper

    #120922
    Profile photo of neighbourhoodnerd
    neighbourhoodnerd
    Participant

    Sorry, that was a dumb way for me to explain it 😛

    I’ve seen online that the command scraper is the one to use, but if I run it on my Pi it says the command wasn’t recognised. Even though I definitely have it installed and have used it (but only via the retropie-setup app).

    #121142
    Profile photo of sselph
    sselph
    Participant

    Ah retropie doesn’t place the binary in the system path so you can’t just issue scraper but would have to give the full path. Unfortunately, I can’t figure out from the code where it places the binary so you might be better off just downloading a copy by following my instructions here:
    https://github.com/sselph/scraper#install-from-my-binaries

    Just remember you have to find the latest release value here:
    https://github.com/sselph/scraper/releases

    Auto-scraper: https://github.com/sselph/scraper

    #121448
    Profile photo of neighbourhoodnerd
    neighbourhoodnerd
    Participant

    Thanks! I got it working 🙂

    #121458
    Profile photo of enigma
    enigma
    Participant

    I need some help with an issue. I ran your scraper on my NES library of 1600-ish games and it worked perfectly. But I deleted the list to run it again without ‘use-nointro-name’ this time and now I can’t get it to run at all. It checks for new hash and then I get this error ‘got 413 Request Entity Too Large response’.

    Is it something on my side? I’ve tried running without any arguments and running it on a small list of 5 games as well, but no change. I’ve also tried waiting a week or so to see if anything would change, but it hasn’t.

    #121462
    Profile photo of sselph
    sselph
    Participant

    That is an odd error. You are getting the error from downloading the hash or later? Can you me a send me a PM with the output of the script? That error means that a request you sent to web server was too large. This usually only happens if you are uploading a file which my script doesn’t do.

    Auto-scraper: https://github.com/sselph/scraper

Viewing 24 posts - 36 through 59 (of 59 total)

Forums are currently read only - please visit the new RetroPie forums at https://retropie.org.uk/forums/

Skip to toolbar