feat: Add responses API

[samvrlewis]

Adds support for the OpenAI responses API.

Doesn't have support for streaming yet.

Due to types being reexported from the root of the crate, there's a few types with existing names (but different shapes) that I've exported with Responses prefixes to avoid making ambiguous types. For example: ResponsesFilePath, ResponsesRole). Happy for feedback if there's a different way that this would be preferred. (edit: realised later it would probably be cleaner just to not export all the types at the root, as there's too much duplication).

[twitchax]

Cool.

I tried this against an existing project, and I think something might be wrong with the text() option. I get.

Error while handling: invalid_request_error: Missing required parameter: 'text.format.name'. (param: text.format.name) (code: missing_required_parameter)

I am guessing that in chat.rs,

pub enum ResponseFormat {
    /// The type of response format being defined: `text`
    Text,
    /// The type of response format being defined: `json_object`
    JsonObject,
    /// The type of response format being defined: `json_schema`
    JsonSchema {
        json_schema: ResponseFormatJsonSchema,
    },
}

might need to be...

pub enum ResponseFormat {
    /// The type of response format being defined: `text`
    Text,
    /// The type of response format being defined: `json_object`
    JsonObject,
    /// The type of response format being defined: `json_schema`
    JsonSchema(ResponseFormatJsonSchema),
}

So that name, etc. is at the "top-level" of text.format.

[samvrlewis]

Thanks for testing it @twitchax and good catch, I think you're right. I updated it to use a ResponseFormat just for this API rather than reusing the chat one. The example uses a jsonschema response format now and works for me.

fwiw, I mostly hand generated most of this as I couldn't find any generators that produced nice output. So with how complex the API is, I do worry there might be other subtle cases like this. I suppose if there are more issues like this they can be fixed as encountered though.

[twitchax]

@samvrlewis, yeah, I agree. Not sure how to fully exercise it, honestly, lol.

[twitchax]

@samvrlewis, your changes fixed this issue with parsing the RepsonseFormat, and it works for me. Not that my review matters, but LGTM. 👍

[twitchax]

Actually, getting another issue, which just seems like a serialization problem. When I switch to o3-mini, I get...

ERROR Error while handling: failed to deserialize api response: missing field `status` at line 18 column 4

Looks like there is an extra output type called reasoning that does not have a status field.

2025-05-23T08:25:55.025 app[6830d40a650698] sea [info] {
2025-05-23T08:25:55.025 app[6830d40a650698] sea [info] "id": "rs_683031120ddc8191b416bd3ad56cb6fd052c3d28f22abf09",
2025-05-23T08:25:55.025 app[6830d40a650698] sea [info] "type": "reasoning",
2025-05-23T08:25:55.025 app[6830d40a650698] sea [info] "summary": []
2025-05-23T08:25:55.025 app[6830d40a650698] sea [info] },

[samvrlewis]

Huh, yeah, there is no status.

  "output": [
    {
      "id": "rs_68305c0068848191ab9fa768452d6b090e0209c29f190939",
      "type": "reasoning",
      "summary": []
    },
    {
      "id": "msg_68305c00cac481918b57e1b255671ace0e0209c29f190939",
      "type": "message",
      "status": "completed",
      "content": [
        {
          "type": "output_text",
          "annotations": [],
          "text": "{\"title\": \"Historic Climate Accord Reached: Nations Unite to Combat Global Warming\", \"website\": \"https://www.globalnews.com\"}"
        }
      ],
      "role": "assistant"
    }
  ],

Unless I'm misunderstanding them, the docs seem to say that it should have one though..

Maybe "populated when items are returned via API" means "not in response to a request"? 🤷‍♂️

In any case, I pushed an update to make the field optional so it doesn't make deser. Thanks for the ongoing testing @twitchax!

[twitchax]

Yeah, I am guessing it is only present when summary has items?

[twitchax]

It looks like ToolDefinition may also need an mcp option, or, at least, that's what it looks like in the MCP docs.

https://platform.openai.com/docs/guides/tools-remote-mcp

[samvrlewis]

It looks like ToolDefinition may also need an mcp option, or, at least, that's what it looks like in the MCP docs.

https://platform.openai.com/docs/guides/tools-remote-mcp

Ah yeah looks like OpenAI added a bunch new tools last week: https://platform.openai.com/docs/changelog !

Have updated the PR to include them, and updated the example to use the same MCP example from the above link.

[samvrlewis]

Thought about it some more and I think for the responses types it probably makes less sense to export them all at the root of the types crate, as there's so many duplicated (but subtly different) types and naming becomes confusing. Have instead made everything available through types::responses which I think makes sense, as it seems like OpenAI intends for this API to be a superset of a lot of the previously available APIs.

[twitchax]

LGTM after that change. Had to bump a few types, but all my tests still pass, so... 👍?

Lol.

Haven't tried the MCP stuff, but I bet it works. May try it in a few weeks. If you're interested, I'm currently using this for https://github.com/twitchax/triage-bot, which is maybe 70% there? Mostly works, but needs some tweaking.

[samvrlewis]

LGTM after that change. Had to bump a few types, but all my tests still pass, so... 👍?

woohoo, thanks for trying! 🎉

Haven't tried the MCP stuff, but I bet it works.

It seems to work for at least the simple example I have in the code now that uses https://mcp.deepwiki.com/mcp. Hopefully for more complicated cases it works too.

If you're interested, I'm currently using this for https://github.com/twitchax/triage-bot, which is maybe 70% there? Mostly works, but needs some tweaking.

Looks cool! How well does SurrealDB work for retrieving context on demand in that? At work we have a somewhat similar service that tries to associate incidents with recent pull requests but it doesn't work very well, as it's really hard to give it enough context to let it figure things out. Your approach of doing the initial triaging seems a lot more promising. How is it working for you?

[twitchax]

Nice.

I haven't used it in a real context yet, but I like SurrealDB. I'd argue that any DB with a full-text search would work fairly well?

I think there is more opportunity to "agentize" some of these things, or get more context into the hands of the LLM. My approach right now uses a two-phase system. First phase (using 4.1) throws a directive, some small context, and the user's query at two separate agents. One of them does a web search, and the second one just determines "search terms" for messages (which are then FTSed on the past messages). Second phase takes the system directive, custom channel directive, any stored channel context, the web search results, the message search results, and the user's query to the assistant agent (using o3). That seems to work fairly well; however, I could see a scenario in which a "loopback" remote MCP might be better. Essentially, allow the LLM to call back to a server housing all of the context / past messages, and give it access to read queries over that data so it can "traverse" / search it however it would like.

Even if my bot-server has access to the same data, I like the idea of just having a separate process run an MCP so I don't have to deal with all of the back-and-forth function calls in the bot code. Looks like Rust is getting some love, so an MCP server may be pretty painless to just drop in.

Definitely some promising results, but I'm trying to push the envelope on what is possible.

In your case, I think that's exactly what remote MSP is for. Instead of gathering a bunch of context that will likely eat up tokens, give o3 the initial context about the incident, and then the remote MCP endpoint for GitHub. As far as I understand, o3 will then decide what sort of calls to make, and it will sort of "throw away" what it doesn't need if it "thinks" it went down the wrong direction.

[twitchax]

@samvrlewis, do you know the best way to respond with type function_call_output.

https://platform.openai.com/docs/guides/function-calling?api-mode=responses

Maybe a new enum option needs to be added to Input? It also looks like you're supposed to append the API's tool call message itself, and I'm not sure that's possible either?

Maybe, due to the explosion of options, Input just needs a Custom state that just lets you shove a raw serde_json::Value in there? Just thinking out loud.

[samvrlewis]

Maybe, due to the explosion of options, Input just needs a Custom state that just lets you shove a raw serde_json::Value in there? Just thinking out loud.

Ooh, yeah, there are a lot of input items that aren't there right now. 😬 Would be nice to have these all strictly typed but for now I've done as you suggested and added a Custom variant.

Added another example that uses the get_weather example, seems to work!

Thanks for the prompts on MCP, btw! Definitely something to explore a bit further when I have some time. I do worry about how well the model would work with a codebase of significant complexity though, if it's needing to navigate around file by file. I haven't had much success using coding tools like cursor against big repos, I usually need to give them a tighter context to get any good output. Though to be fair I'm usually focused on generating code, maybe just reading/understanding code would be easier?

[twitchax]

+1, mileage varies a ton for the "agent mode" stuff like Cursor and Copilot Agent. I have the same experience with them as you. Limiting to tests or small refactors appears to work well. Everything else tends to fold, and produce bad architecture.

For my purposes, MCP is going to be a big deal, so I've been poking at it a little more than others might.

Thanks for entertaining my random observations while I poke! Happy to help out at some point, but not certain how you and other maintainers feel about stop-gaps like Custom enum placeholders.

[64bit]

Thank you so much @samvrlewis and @twitchax for your contributions!

Thank you for doing all the heavy lifting by hand typing types and testing. Appreciate all the hard work!

Your design choice to nest types inside types::responses is a good one!