scraper for NES and SNES using ROM hashes

Welcome Page Forums RetroPie Project Everything else related to the RetroPie Project scraper for NES and SNES using ROM hashes

This topic contains 13 replies, has 4 voices, and was last updated by Profile photo of Floob Floob 2 years, 5 months ago.

Viewing 14 posts - 1 through 14 (of 14 total)
  • Author
    Posts
  • #15067
    Profile photo of sselph
    sselph
    Participant

    Hi,

    I was reinstalling retropie and had to re-scrape my rom directory. For fun, I pulled the full game list for NES and SNES from thegamedb, manually matched everything, and mapped all the hashes to the ID in the DB. Then wrote a quick program to scrape using this data.

    I figured I’d share it in case anyone might find it useful:
    https://github.com/sselph/scraper

    Auto-scraper: https://github.com/sselph/scraper

    #15363
    Profile photo of deaftolight
    deaftolight
    Participant

    where is it

    #15372
    Profile photo of sselph
    sselph
    Participant

    Odd pretty sure I added a link it’s on github.com/sselph/scraper

    Auto-scraper: https://github.com/sselph/scraper

    #15384

    Haris

    Sselph can you write similar script for the other emulators too?

    #15411
    Profile photo of sselph
    sselph
    Participant

    It is possible to extend the script. I’d have to write code to extract the rom information if the format has headers and other information then use a list of hashes to manually build the mapping from hash to ID. If there is a platform you would like me to try I could take a look.

    Auto-scraper: https://github.com/sselph/scraper

    #15483

    Haris

    Sure man. I would like to see it in:

    Fba,turbograf16,mame,genesis,megadrive

    atari2600,gbc

    Pretty much for all i suposse:)

    #15850

    Haris

    I get an

    -bash: $: command not found

    when i try to run $ go get github.com/sselph/scraper

    why is that?

    #15976
    Profile photo of sselph
    sselph
    Participant

    I wrote the script in go so you’ll have to install that to compile it. Many distributions have packages so should be as easy as something like:
    sudo apt-get install golang
    You’ll probably need to set a GOPATH environment variable:
    http://golang.org/doc/code.html#GOPATH

    But to make things easy I compiled versions for most platforms at:
    https://github.com/sselph/scraper/releases

    I recommend running this on a machine other than the pi but I compiled one for it anyway.

    (-thumb_only flag can speed it up for testing since it is not having to download/process large images.)

    Auto-scraper: https://github.com/sselph/scraper

    #16067
    Profile photo of deaftolight
    deaftolight
    Participant

    I just get a bunch of errors:

    SIGILL: illegal instruction
    PC=0x3b484
    
    math.init·1()
            /usr/lib/go/src/pkg/math/pow10.go:34 +0x28
    math.init()
            /usr/lib/go/src/pkg/math/unsafe.go:21 +0x70
    fmt.init()
            /usr/lib/go/src/pkg/fmt/scan.go:1164 +0x70
    github.com/sselph/scraper/rom.init()
            /home/pi/go/src/github.com/sselph/scraper/rom/rom.go:83 +0x7c
    github.com/sselph/scraper/rom/snes.init()
            /home/pi/go/src/github.com/sselph/scraper/rom/snes/snes.go:48 +0x70
    main.init()
            /home/pi/go/src/github.com/sselph/scraper/scraper.go:395 +0x74
    
    goroutine 2 [runnable]:
    created by runtime.main
            /golang-1.0.2/src/pkg/runtime/proc.c:221
    
    trap    0x6
    error   0x0
    oldmask 0x0
    r0      0x7332f8
    r1      0x0
    r2      0x0
    r3      0x2
    r4      0x0
    r5      0x0
    r6      0x0
    r7      0x0
    r8      0x0
    r9      0x324fec
    r10     0x10840000
    fp      0x732eb6
    ip      0xcafebabe
    sp      0xb6ebcf24
    lr      0x3ba18
    pc      0x3b484
    cpsr    0x20000010
    fault   0x0
    
    #16071
    Profile photo of sselph
    sselph
    Participant

    I got similar issues when I tried compiling from the raspberry pi. The build command needed ‘GOARM=5 go build <blah>’ but seems the version of go in raspbian is older(1.0.2 instead of 1.1+ I’m using 1.2) and couple things don’t seem to work.

    I pushed a version that fixes one of the issues but the library that resizes images doesn’t work so you’ll have to use -thumb_only so resizing isn’t needed. Use go get -u github.com/sselph/scraper to update the package.

    I also cross compiled a version for the raspberry pi using 1.2 here. Or if you like you can do that yourself from an ubuntu machine by installing golang-go-linux-arm and doing ‘GOARM=6 GOOS=linux GOARCH=arm go build github.com/sselph/scraper’
    https://github.com/sselph/scraper/releases/download/v0.1-beta/scraper_rpi.zip

    Auto-scraper: https://github.com/sselph/scraper

    #83195
    Profile photo of grassmunk
    grassmunk
    Participant

    I just wanted to say thanks for this! You’ve saved me a tonne of time on scraping material!

    #85057
    Profile photo of Floob
    Floob
    Member

    Is there anyway it could deal with Atari Lynx games in the future?

    RetroPie help guides --> https://goo.gl/Yfy8kj
    Please read this before asking for help --> http://goo.gl/eLErnl

    #85062
    Profile photo of sselph
    sselph
    Participant

    I actually added that over the weekend try v0.5.8-beta and let me know if it is working correctly.

    Auto-scraper: https://github.com/sselph/scraper

    #85064
    Profile photo of Floob
    Floob
    Member

    Great! I’ll give it a go. Thanks 🙂

    RetroPie help guides --> https://goo.gl/Yfy8kj
    Please read this before asking for help --> http://goo.gl/eLErnl

Viewing 14 posts - 1 through 14 (of 14 total)

Forums are currently read only - please visit the new RetroPie forums at https://retropie.org.uk/forums/

Skip to toolbar