Add search query and summary from Brave Search SERP when open Leo chat

[yrliou]

Resolves brave/brave-browser#40615

• Add search query and summary from Brave Search SERP to conversation
  history when summarizer-key tag is present.
• Also add a new from_brave_search_SERP property in ConversationTurn to identify the
  conversation entries from Brave Search SERP.
• Modify UI to show the page context toggle when search query and summary are
  staged.
• Staged search query and summary (at the end of the conversation history) will
  be cleared if user opted out, page was unlinked, or user navigated away to a new page

For more details, please check requirement # 1 in https://docs.google.com/document/d/1idelFPpUEcKDNcyKYf3M5yw91tuIddjrlYfpaWEJjNk/edit

Security/Privacy review: https://github.com/brave/reviews/issues/1732

Submitter Checklist:

• I confirm that no security/privacy review is needed and no other type of reviews are needed, or that I have requested them
• There is a ticket for my issue
• Used Github auto-closing keywords in the PR description above
• Wrote a good PR/commit description
• Squashed any review feedback or "fixup" commits before merge, so that history is a record of what happened in the repo, not your PR
• Added appropriate labels (QA/Yes or QA/No; release-notes/include or release-notes/exclude; OS/...) to the associated issue
• Checked the PR locally:
  • npm run test -- brave_browser_tests, npm run test -- brave_unit_tests wiki
  • npm run presubmit wiki, npm run gn_check, npm run tslint
• Ran git rebase master (if needed)

Reviewer Checklist:

• A security review is not needed, or a link to one is included in the PR description
• New files have MPL-2.0 license header
• Adequate test coverage exists to prevent regressions
• Major classes, functions and non-trivial code blocks are well-commented
• Changes in component dependencies are properly reflected in gn
• Code follows the style guide
• Test plan is specified in PR before merging

After-merge Checklist:

• The associated issue milestone is set to the smallest version that the
  changes has landed on
• All relevant documentation has been updated, for instance:
  • https://github.com/brave/brave-browser/wiki/Deviations-from-Chromium-(features-we-disable-or-remove)
  • https://github.com/brave/brave-browser/wiki/Proxy-redirected-URLs
  • https://github.com/brave/brave-browser/wiki/Fingerprinting-Protections
  • https://github.com/brave/brave-browser/wiki/Brave%E2%80%99s-Use-of-Referral-Codes
  • https://github.com/brave/brave-browser/wiki/Web-Compatibility-Exceptions-in-Brave
  • https://github.com/brave/brave-browser/wiki/QA-Guide
  • https://github.com/brave/brave-browser/wiki/P3A

Test Plan:

1. Put a query in Brave Search and click Answer with AI
2. Wait until the answer is fully generated, open Leo panel, should see the query and summary in the panel.
3. Toggle off the page context switch should clear the staged query and summary
4. Toggle it back on should see re-staged query and summary
5. Type a follow-up question from the summary should work

[brave-builds]

A Storybook has been deployed to preview UI for the latest push

[fallaciousreasoning]

LGTM - on thing which is slightly odd to me on this is that we have the data on the page already, so it feels slightly weird to make an API call to fetch it, after parsing the API key from the page.

Maybe pulling the data out from the page is more complicated though?

edit: read the spec

[fallaciousreasoning]

still lgtm

[diracdeltas]

noting that this looks like a Tor bypass if this feature were available in Tor windows, but i don't think it is at the moment

[diracdeltas]

would suggest adding a test for the not-opted-into-leo case to the test plan

[github-actions]

[puLL-Merge] - brave/brave-core@25257

Here's my review of the pull request:

Description

This PR adds functionality to fetch and display search query summaries from Brave Search SERP (Search Engine Results Page) in the AI Chat feature. It includes changes to handle the retrieval, parsing, and integration of these summaries into the conversation history. The PR also adds unit tests and browser tests to cover the new functionality.

Changes

Changes

1. browser/ai_chat/ai_chat_ui_browsertest.cc:

   • Added new test cases for fetching search query summaries
   • Implemented mock responses for different search query summary scenarios

2. components/ai_chat/content/browser/ai_chat_tab_helper.cc and ai_chat_tab_helper.h:

   • Added methods to fetch and process search query summaries
   • Integrated search query summary handling into the conversation flow

3. components/ai_chat/core/browser/conversation_driver.cc and conversation_driver.h:

   • Implemented logic to fetch, clear, and manage search query summaries
   • Added new tests for search query summary functionality

4. components/ai_chat/core/browser/utils.cc and utils.h:

   • Added utility function IsBraveSearchSERP to identify Brave Search SERP URLs

5. components/ai_chat/renderer/page_content_extractor.cc and page_content_extractor.h:

   • Implemented GetSearchSummarizerKey method to extract summarizer key from meta tags

6. components/ai_chat/core/common/mojom/ai_chat.mojom:

   • Added from_brave_search_SERP field to ConversationTurn struct

7. Various test files:

   • Updated existing tests and added new ones to cover the new functionality

8. UI components (React and iOS):

   • Updated to handle and display search query summaries

Possible Issues

1. The PR adds complexity to the conversation flow, which may require careful testing to ensure it doesn't introduce unexpected behavior in edge cases.
2. The new functionality is tightly coupled with Brave Search, which may impact maintainability if the search API changes in the future.

Security Hotspots

1. The PR introduces new network requests to fetch search query summaries. Ensure that proper security measures (e.g., HTTPS, input validation) are in place to prevent potential attacks.
2. The GetSearchSummarizerKey method uses regular expressions to parse HTML content. Ensure that this parsing is robust against malformed or malicious input to prevent potential security vulnerabilities.

Overall, the changes appear well-structured and include appropriate test coverage. However, due to the complexity of the changes and the introduction of new network requests, thorough testing and security review are recommended before merging.