Question: How to remove specific url from dupefilter set on redis?

[matheus-rossi]

Description

Today, I use web scraping and LLM models to extract data from URLs. If, for some reason, I encounter an error with the LLM, my plan is to retry processing during the next scheduled run (I schedule URLs every 15 minutes).

The dupefilter is very helpful, but if a URL has already been parsed by Scrapy, I need to download it again just for that specific URL.

Is there a way to remove a specific URL from the dupefilter redis set?

[rmax]

Did you try the dont_filter request meta parameter?

[matheus-rossi]

It works...

But as I schedule the same site to be crawled every hour, it's easier for me to capture any errors and remove them from the set. This way, on the next run, it will automatically try again.

Otherwise, I will need to check each previous link somewhere else to look for a status and pass it to the request.

[rmax]

Looks like you need a slightly different implementation.

The dupefilter assumes a single crawl session and by default it uses the spider name as redis key prefix.

But it seems you want to start a new crawl session every hour, so you will need to customize the dupefilter key for each new crawl session. Or delete the entire dupefilter key set from redis before starting a new crawl.

Does it make sense?

[matheus-rossi]

When I use a specific spider for a website, I generally just delete the entire dupefilter, and it works fine.

However, we have two 'general spiders' that are shared across more than 1000 URLs. In this case, if I delete the entire dupefilter, the impact on the other URLs will be huge. This is where it would be helpful to delete only the specific URLs.

Perhaps the solution is to customize the dupefilter logic...

[toxicbuzzer]

Here is my altered dupefilter.py code. It adds the domain address from the URL to the end of the Redis key and also adds an expiry to the key:

My Redis key now appears like this: "sitemap:dupefilter:www.example.com". This way each domain expires independently.

    #CODE changes for dupefilter.py:
    
    from urllib.parse import urlparse
    
    def request_seen(self, request):
      fp = self.request_fingerprint(request)

      #-- add url to redis key (ps)
      url = urlparse(request.url).netloc
      new_key=self.key + ':' + canonicalize_url(url)

      # This returns the number of values added, zero if already exists.
      added = self.server.sadd(new_key, fp)

      #--set expiry for new key (ps)
      self.server.expire(new_key,60*1440,1) #-- NX mode (ps)

      return added == 0

You may need to update this function too, but I cant see where Scrapy Scheduler actually calls it... but seems to work without changing this for the above use case.

def close(self, reason=""):
  #--Delete data on close. Called by Scrapy's scheduler.
  self.clear()

def clear(self):
  """Clears fingerprints data."""
  self.server.delete(self.key)