intermittent failure of TestGeminiAIStandard.test_tool_calling_with_no_arguments in langchain-google-genai

[msaadg]

The integration test TestGeminiAIStandard.test_tool_calling_with_no_arguments in langchain-google-genai intermittently fails with a 400 Invalid Argument error from the Google Generative AI API. Observed in PR #970 and main branch, it sometimes passes and sometimes fails, unrelated to PR changes. Marked as xfail for gemini-1.5-pro-latest.

Details:

• Error: InvalidArgument in chat_models.py during _chat_with_retry (generate_content call).
• Impact: Inconsistent test results block clean test runs; may affect tool-calling reliability.
• Steps to Reproduce:
  1. Run make test in libs/genai.
  2. Observe intermittent failure in TestGeminiAIStandard.test_tool_calling_with_no_arguments.
• Proposed Actions:
  1. Debug API request payload for invalid argument causing intermittent failure.
  2. Check compatibility with google.ai.generativelanguage_v1beta.
  3. Review test case for API alignment.
  4. Report upstream issue if confirmed and document workaround.

Further investigation needed to identify the cause of intermittent failures.

Error Trace:

=================================== FAILURES ===================================
___________ TestGeminiAIStandard.test_tool_calling_with_no_arguments ___________
Traceback (most recent call last):
  File "/workspace/libs/genai/langchain_google_genai/chat_models.py", line 196, in _chat_with_retry
    return generation_method(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/google/ai/generativelanguage_v1beta/services/generative_service/client.py", line 868, in generate_content
    response = rpc(
               ^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target
    _retry_error_helper(
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper
    raise final_exc from source_exc
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target
    result = target()
             ^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/google/api_core/timeout.py", line 130, in func_with_timeout
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: 400 Request contains an invalid argument.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
                                ^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/_pytest/python.py", line 1792, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/langchain_tests/integration_tests/chat_models.py", line 1497, in test_tool_calling_with_no_arguments
    result = model_with_tools.invoke(query)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 5431, in invoke
    return self.bound.invoke(
           ^^^^^^^^^^^^^^^^^^
  File "/workspace/libs/genai/langchain_google_genai/chat_models.py", line 1279, in invoke
    return super().invoke(input, config, stop=stop, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 372, in invoke
    self.generate_prompt(
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 957, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 776, in generate
    self._generate_with_cache(
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 1022, in _generate_with_cache
    result = self._generate(
             ^^^^^^^^^^^^^^^
  File "/workspace/libs/genai/langchain_google_genai/chat_models.py", line 1366, in _generate
    response: GenerateContentResponse = _chat_with_retry(
                                        ^^^^^^^^^^^^^^^^^
  File "/workspace/libs/genai/langchain_google_genai/chat_models.py", line 214, in _chat_with_retry
    return _chat_with_retry(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/builder/home/.cache/pypoetry/virtualenvs/langchain-google-genai-G8ioShJx-py3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/workspace/libs/genai/langchain_google_genai/chat_models.py", line 208, in _chat_with_retry
    raise ChatGoogleGenerativeAIError(
langchain_google_genai.chat_models.ChatGoogleGenerativeAIError: Invalid argument provided to Gemini: 400 Request contains an invalid argument.