Description
Describe the bug
Group scrapers don't work at all. This encompasses multiple bugs
- groupByURL index out of range
group
field not found
To Reproduce
Sample "Group" scraper
name: GroupScraper
groupByURL:
- action: scrapeXPath
url:
- example.com/group
scraper: groupScraper
# avoid load errors
# movieByURL:
# - action: scrapeXPath
# url:
# - example.com/movie
# scraper: movieScraper
xPathScrapers:
groupScraper:
movie:
# set to "movie" to avoid load errors
# group:
Name:
fixed: Group Name
movieScraper:
movie:
Name:
fixed: Movie Name
groupByURL Index out of range
When attemping to scrape with groupByURL, the following error appears
scrapeGroupURL: input: scrapeGroupURL Internal system error. Error <runtime error: index out of range [0] with length 0>
runtime error: index out of range [0] with length 0
replacing groupByURL
with movieByURL
magically fixes this issue and the scene scrapes perfectly fine. https://http.cat/200
is the test URL, since example.com returns 404 for everything and scrape fails
My attempt at problem solving
scrapeGroupURL leads to two likely locations, graphql (which I don't suspect) and https://github.com/stashapp/stash/blob/develop/internal/api/resolver_query_scraper.go#L102
I couldn't spot anything that particularly stood out compared to ScrapeMovieURL, my best guess because of an index lookup is that when passing it to models.ScrapedGroup, the URLs field is not being looked up properly, leading me down to https://github.com/stashapp/stash/blob/develop/internal/api/scraped_content.go#L164-L171
or
https://github.com/stashapp/stash/blob/develop/pkg/scraper/scraper.go#L173-L183
but I don't think it got that far, since it errors before it hits 404
group field not found
Uncommenting group
in xPathScrapers/* yields the following error
Error loading scraper /config/scrapers/community/dev/dev.yml: yaml: unmarshal errors:
line 22: field group not found in type scraper.mappedScraper
The issue seems to be here: as group is not defined
Lines 848 to 855 in 4bfc93b
Co-Existence edit:
movieByURL disallowed if groupByURL is present
is overriden by errors such as, which incorrectly led me to believe that it was allowed on develop
Error loading scraper /config/scrapers/community/dev/dev.yml: yaml: unmarshal errors:
line 8: field movieByURL already set in type scraper.config
line 16: field group not found in type scraper.mappedScraper
Metadata
Metadata
Assignees
Type
Projects
Status