Skip to content

[Bug] onlyImportantContent / Tag exclusions appear to have no effect #1564

Open
@jalexy12

Description

@jalexy12

Describe the Bug
Hello! Maybe I'm doing something silly here, but it doesn't seem excluding tags and/or the "only important content" fields seem to have any effect on the outcome.

I've tried this via the playground, via the Go SDK, and via curl, and none of these seem to produce a result I'd expect with the given page.

To Reproduce
Steps to reproduce the issue:

  1. Navigate to the playground here
  2. Hit "Run"
  3. Try various combinations of tags and/or onlyImportantContent enabled/disabled
  4. Observe that the response always includes the following:
GET ITON

Menu

- [Explore](https://www.simon.com/mall/ross-park-mall "Ross Park Mall™  Home")
- [Stores](https://www.simon.com/mall/ross-park-mall/stores)
- [Products](https://www.simon.com/mall/ross-park-mall/search)
- [Dining](https://www.simon.com/mall/ross-park-mall/stores?&filter=store_type%7CstoreType%2F2)
- [Home](https://www.simon.com/mall/ross-park-mall/home)
- [Luxury](https://www.simon.com/mall/ross-park-mall/luxury)
- [Map](https://www.simon.com/mall/ross-park-mall/map/)
- [Deals](https://www.simon.com/mall/ross-park-mall/deals)
- [News & Events](https://www.simon.com/mall/ross-park-mall/news-and-events)
- [Family](https://www.simon.com/mall/ross-park-mall/family)
- [Travel](https://www.simon.com/mall/ross-park-mall/travel-here)
- [About](https://www.simon.com/mall/ross-park-mall/about)
- [Hours](https://www.simon.com/mall/ross-park-mall/hours)

This navigation bar should very clearly be excluded - it's a header with a nav element inside - both of which are excluded in onlyMainContent and in the manual exclusion list. I've tried with one enabled, the other off, etc - no combination of these seems to exclude this header.

Expected Behavior
No navigation element included in the output

Environment (please complete the following information):

  • OS: macOS
  • Firecrawl Version: 1 (I think?)

Please let me know if I'm missing something obvious and silly here re: the behavior. Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions