claude sonnet latest: "all messages must have non-empty content except for the optional final assistant message"

[gordonwoodhull]

Hi Carson!

Thanks to chatlas, I have my quarto chatbot working on both anthropic and openai! ✨

https://github.com/gordonwoodhull/quarto-data-science-chatbot/

It took less than an hour to port from the openapi python library, and the output tool show_answer is connected much more reliably.

Chatlas also multiplexes the streaming chat and sending the documents via the tool very nicely!

I find that with anthropic claude-3-opus-20240229 my chatbot works reliably but it is very chatty. If I switch to claude-3-5-sonnet-latest with

QUARTO_DS_CHATBOT_PROVIDER=anthropic QUARTO_DS_CHATBOT_MODEL=claude-3-5-sonnet-latest shiny run ds-quarto-chatbot.py

It seems both speedier and more to-the-point, but when I ask it to revise its answer, my second input, it crashes with

Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error',
   'message': 'messages.3: all messages must have non-empty content except for the optional final assistant message'}}

Typical conversation:

1. please draw a heatmap of synthetic randomized hourly temperatures for a place that's in the 70s in the afternoon and gets down to 40 at night, using ggplot2 (or altair, or plotnine, etc.)

2. yeah that's great but please put the days in x and the hours in y and nights should be colder than days (or whatever silly thing it got wrong)

My chatbot crashes consistently on step 2 with sonnet, but works reliably with opus and with openai gpt-4o. I haven't tested others yet.

Here is the full stack trace:

Traceback (most recent call last):
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/shiny/reactive/_extended_task.py", line 131, in _execution_wrapper
    return await self._func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/shiny/ui/_chat.py", line 568, in _stream_task
    await self._append_message_stream(message)
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/shiny/ui/_chat.py", line 595, in _append_message_stream
    async for msg in message:
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/shiny/_utils.py", line 508, in __anext__
    value = next(self.iterator)
            ^^^^^^^^^^^^^^^^^^^
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/chatlas/_chat.py", line 902, in __next__
    chunk = next(self._generator)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/chatlas/_chat.py", line 632, in _chat_impl
    for chunk in self._submit_turns(
                 ^^^^^^^^^^^^^^^^^^^
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/chatlas/_chat.py", line 681, in _submit_turns
    response = self.provider.chat_perform(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/chatlas/_anthropic.py", line 244, in chat_perform
    return self._client.messages.create(**kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/anthropic/_utils/_utils.py", line 274, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/anthropic/resources/messages.py", line 888, in create
    return self._post(
           ^^^^^^^^^^^
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/anthropic/_base_client.py", line 1277, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/anthropic/_base_client.py", line 954, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/anthropic/_base_client.py", line 1058, in _request
    raise self._make_status_error_from_response(err.response) from None
anthropic.BadRequestError: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.3: all messages must have non-empty content except for the optional final assistant message'}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/shiny/reactive/_reactives.py", line 584, in _run
    await self._fn()
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/shiny/ui/_chat.py", line 583, in _handle_error
    await self._raise_exception(e)
  File "/Users/gordon/src/quarto-cli/.venv/lib/python3.12/site-packages/shiny/ui/_chat.py", line 345, in _raise_exception
    raise NotifyException(str(e), sanitize=sanitize) from e
shiny.types.NotifyException: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.3: all messages must have non-empty content except for the optional final assistant message'}}

[gordonwoodhull]

Aha, it also works reliably with claude-3-5-sonnet-20240620, only failing with latest = claude-3-5-sonnet-20241022

[cpsievert]

Very nice, thanks for sharing!

I'll eventually have a look into, but in the meantime you can utilize the new .chat(echo="all") argument to get a better sense of what the conversation looks like before the error occurs, which should help highlight where/why things are going wrong, and thus help to get a more minimal reproducible example

[gordonwoodhull]

Great! I'll try to use echo="all" to provide a reproducible example.

In the meantime, I hope the hint to try earlier sonnet is helpful to anyone who hits this error. Older sonnet is good enough for my development purposes, no complaints here.

Thanks so much for chatlas!

[gordonwoodhull]

I solved this by upgrading to anthropic 0.42.

[gordonwoodhull]

Actually, using the latest anthropic package does not fix the error.

I think that the error arises when Claude's response ends with a tool call and no chat after it. This causes a message to be put in history with content: []

Currently claude-3-5-sonnet-latest is being chatty, so I'm unable to reproduce this. Just yesterday I could see the error, and I think that filtering out messages with content.length === 0 (or not producing them) would be sufficient to avoid the error.

I'll return to this if Claude gets terse again.

[gordonwoodhull]

The error is a bit more clear with anthropic 0.42

all messages must have non-empty content except for the optional final assistant message

I get the error consistently with streaming enabled and with a system prompt to submit answers as Quarto documents using the tool I register. In this circumstance Claude will usually chat something and then submit a Quarto document and not chat further. Sometimes it shows a spinner as if it's waiting for more response, but it is done.

When I ask a follow-up question, it will crash.

In this case it seems chatlas or shiny.ui.Chat allows many of these empty-content messages to be appended.

I have made a stupid patch which solves the issue for me. As noted, it's overkill, but I do see it removing as many as 7 messages with empty content.

diff --git a/chatlas/_anthropic.py b/chatlas/_anthropic.py
index 9d7d0e9..56d6fa7 100644
--- a/chatlas/_anthropic.py
+++ b/chatlas/_anthropic.py
@@ -241,6 +241,14 @@ class AnthropicProvider(Provider[Message, RawMessageStreamEvent, Message]):
         kwargs: Optional["SubmitInputArgs"] = None,
     ):
         kwargs = self._chat_perform_args(stream, turns, tools, data_model, kwargs)
+        messages = kwargs['messages']
+        if messages:
+            # this is overkill as it will remove the one allowed empty content at the end
+            # "all messages must have non-empty content except for the optional final assistant message"
+            no_empties = [m for m in messages if len(m['content']) != 0]
+            if len(no_empties) != len(messages):
+                print(f'averting sending {len(messages) - len(no_empties)} message(s) with empty content')
+                kwargs['messages'] = no_empties
         return self._client.messages.create(**kwargs)  # type: ignore

I'd be glad to submit a repro but it won't be trivial since it involves both streaming and tool calling.

[gordonwoodhull]

I think I may be looking for something like the functionality described in #33, which IIUC helps with cases where the tool call is the final response. Will try again when that lands.