Google Generative AI integration / Gemini "Enable Google Search tool" does not work

[momoz]

The problem

Sorry, I had a problem talking to Google Generative AI: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Tool use with function calling is unsupported', 'status': 'INVALID_ARGUMENT'}}
If I uncheck the "Enable Google Search tool" it all works fine, just not current or web data.

What version of Home Assistant Core has the issue?

2024.4.0

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Google Generative AI

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

Anything in the logs that might be useful for us?

Additional information

No response

[Andrerm124]

Currently it seems like the Google Generative AI can only use the Search Tool if you're in No Control mode. A bit disappointing as I'm really looking forward to having them both run concurrently!

[rialDave]

Currently it seems like the Google Generative AI can only use the Search Tool if you're in No Control mode. A bit disappointing as I'm really looking forward to having them both run concurrently!

yes, this was confirmed by Paulus of HA yesterday in the Discord "Release-Afterparty". I don't know any ETA when it will be fixed tho.

[Andrerm124]

@rialDave Cheers for the info. Do you happen to know whether there was a reason for it to not work with the Assist API?
I'm hoping it's not a limitation of Gemini itself, as it's a really strong LLM for the smart home, and this would be a deal breaker if other LLMs (Such as ChatGPT) can use Search

[rialDave]

@Andrerm124 I'm not 100% sure what's the reason behind it, but it seemed more like a bug or "overlook" thing. We will have to wait and see I guess.
btw. since you mentioned it: The openAI API (and the HA Integration for it) now also supports web search (but a bit slower and not as good as gemini (obviously) can).

[home-assistant]

Hey there @tronikos, @IvanLH, mind taking a look at this issue as it has been labeled with an integration (google_generative_ai_conversation) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of google_generative_ai_conversation can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign google_generative_ai_conversation Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

google_generative_ai_conversation documentation
google_generative_ai_conversation source
_{^{(message by IssueLinks)}}

[tronikos]

"Fixed" in #141712
This is a limitation of Gemini so there isn't anything we can do. In the meantime you can use the workaround in #141712 (comment) that we should add in the docs and close this.

[rialDave]

Thanks for the clarification! That's unfortunate, but then we'll wait and hope for a fix by Google.
What specifically is the limitation right now? Tool calling in combination with web search functionality?

[tronikos]

What specifically is the limitation right now? Tool calling in combination with web search functionality?

Exactly.

A better workaround that we should add in the docs is #141712 (comment)
@Andrerm124 Can you do it?