Skip to content

[Bug Report] Group scrapers don't work #5793

Closed
@feederbox826

Description

@feederbox826

Describe the bug
Group scrapers don't work at all. This encompasses multiple bugs

  • groupByURL index out of range
  • group field not found

To Reproduce
Sample "Group" scraper

name: GroupScraper

groupByURL:
  - action: scrapeXPath
    url:
      - example.com/group
    scraper: groupScraper
# avoid load errors
# movieByURL:
#   - action: scrapeXPath
#     url:
#       - example.com/movie
#     scraper: movieScraper

xPathScrapers:
  groupScraper:
    movie:
# set to "movie" to avoid load errors
#   group:
      Name:
        fixed: Group Name
  movieScraper:
    movie:
      Name:
        fixed: Movie Name

groupByURL Index out of range

When attemping to scrape with groupByURL, the following error appears

scrapeGroupURL: input: scrapeGroupURL Internal system error. Error <runtime error: index out of range [0] with length 0>
runtime error: index out of range [0] with length 0

replacing groupByURL with movieByURL magically fixes this issue and the scene scrapes perfectly fine. https://http.cat/200 is the test URL, since example.com returns 404 for everything and scrape fails

My attempt at problem solving

scrapeGroupURL leads to two likely locations, graphql (which I don't suspect) and https://github.com/stashapp/stash/blob/develop/internal/api/resolver_query_scraper.go#L102

I couldn't spot anything that particularly stood out compared to ScrapeMovieURL, my best guess because of an index lookup is that when passing it to models.ScrapedGroup, the URLs field is not being looked up properly, leading me down to https://github.com/stashapp/stash/blob/develop/internal/api/scraped_content.go#L164-L171
or
https://github.com/stashapp/stash/blob/develop/pkg/scraper/scraper.go#L173-L183
but I don't think it got that far, since it errors before it hits 404

group field not found

Uncommenting group in xPathScrapers/* yields the following error

Error loading scraper /config/scrapers/community/dev/dev.yml: yaml: unmarshal errors:
  line 22: field group not found in type scraper.mappedScraper

The issue seems to be here: as group is not defined

stash/pkg/scraper/mapped.go

Lines 848 to 855 in 4bfc93b

type mappedScraper struct {
Common commonMappedConfig `yaml:"common"`
Scene *mappedSceneScraperConfig `yaml:"scene"`
Gallery *mappedGalleryScraperConfig `yaml:"gallery"`
Image *mappedImageScraperConfig `yaml:"image"`
Performer *mappedPerformerScraperConfig `yaml:"performer"`
Movie *mappedMovieScraperConfig `yaml:"movie"`
}

Co-Existence edit:

movieByURL disallowed if groupByURL is present is overriden by errors such as, which incorrectly led me to believe that it was allowed on develop

Error loading scraper /config/scrapers/community/dev/dev.yml: yaml: unmarshal errors:
  line 8: field movieByURL already set in type scraper.config
  line 16: field group not found in type scraper.mappedScraper

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug reportBug reports that are not yet verified

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions