Skip to content

Commit 2ba89a7

Browse files
moonbox3glorious-beard
authored andcommitted
Python: Introduce the OpenAI Responses Agent (microsoft#11240)
### Motivation and Context OpenAI recently released their Responses API, which is their newest core API and an agentic API primitive, combining the simplicity of Chat Completions with the ability to do more agentic tasks. Azure OpenAI also recently released the Responses API, with close-to feature parity with OpenAI's SDK. It gives us enough to introduce an `AzureResponsesAgent` and an `OpenAIResponsesAgent` to Semantic Kernel. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description This PR introduces: - The `AzureResponsesAgent` and `OpenAIResponsesAgent` - Allows use of the web search tool with `OpenAI` (`Azure OpenAI` currently doesn't support this). - Allows use of the file search tool with both `OpenAI` and `Azure OpenAI`. - Provides getting started samples for both agent types. **TODO:** - Add unit test coverage for the `ResponsesAgentThreadActions` class. - Add support for the computer user agent tool. - Improve typing in the `ResponsesAgentThreadActions` class. - Closes microsoft#11026 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄
1 parent c10d516 commit 2ba89a7

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+4300
-33
lines changed

python/pyproject.toml

+1-1
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ dependencies = [
3434
"numpy >= 1.25.0; python_version < '3.12'",
3535
"numpy >= 1.26.0; python_version >= '3.12'",
3636
# openai connector
37-
"openai ~= 1.61",
37+
"openai >= 1.67",
3838
# openapi and swagger
3939
"openapi_core >= 0.18,<0.20",
4040
"websockets >= 13, < 16",

python/samples/concepts/README.md

+10
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,15 @@
5959
- [OpenAI Assistant Templating Streaming](./agents/openai_assistant/openai_assistant_templating_streaming.py)
6060
- [OpenAI Assistant Vision Streaming](./agents/openai_assistant/openai_assistant_vision_streaming.py)
6161

62+
#### [OpenAI Responses Agent](../../semantic_kernel/agents/open_ai/openai_responses_agent.py)
63+
64+
- [OpenAI Responses Message Callback Streaming](./agents/openai_responses/responses_agent_message_callback_streaming.py)
65+
- [OpenAI Responses Message Callback](./agents/openai_responses/responses_agent_message_callback.py)
66+
- [OpenAI Responses File Search Streaming](./agents/openai_responses/responses_agent_file_search_streaming.py)
67+
- [OpenAI Responses Plugins Streaming](./agents/openai_responses/responses_agent_plugins_streaming.py)
68+
- [OpenAI Responses Reuse Existing Thread ID](./agents/openai_responses/responses_agent_reuse_existing_thread_id.py)
69+
- [OpenAI Responses Web Search Streaming](./agents/openai_responses/responses_agent_web_search_streaming.py)
70+
6271
### Audio - Using services that support audio-to-text and text-to-audio conversion
6372

6473
- [Chat with Audio Input](./audio/01-chat_with_audio_input.py)
@@ -157,6 +166,7 @@
157166

158167
- [Cycles with Fan-In](./processes/cycles_with_fan_in.py)
159168
- [Nested Process](./processes/nested_process.py)
169+
- [Plan and Execute](./processes/plan_and_execute.py)
160170

161171
### PromptTemplates - Using [`Templates`](https://github.com/microsoft/semantic-kernel/blob/main/python/semantic_kernel/prompt_template/prompt_template_base.py) with parametrization for `Prompt` rendering
162172

python/samples/concepts/agents/README.md

+6-1
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ This project contains a step by step guide to get started with _Semantic Kernel
1010
- For the use of Streaming OpenAI Assistant agents, the minimum allowed Semantic Kernel pypi version is 1.11.0.
1111
- For the use of AzureAI and Bedrock agents, the minimum allowed Semantic Kernel pypi version is 1.21.0.
1212
- For the use of Crew.AI as a plugin, the minimum allowed Semantic Kernel pypi version is 1.21.1.
13+
- For the use of OpenAI Responses agents, the minimum allowed Semantic Kernel pypi version is 1.27.0.
1314

1415

1516
## Source
@@ -28,6 +29,7 @@ chat_completion_agent|How to use Semantic Kernel Chat Completion agents that lev
2829
bedrock|How to use [AWS Bedrock agents](https://aws.amazon.com/bedrock/agents/) in Semantic Kernel.
2930
mixed_chat|How to combine different agent types.
3031
openai_assistant|How to use [OpenAI Assistants](https://platform.openai.com/docs/assistants/overview) in Semantic Kernel.
32+
openai_responses|How to use [OpenAI Responses](https://platform.openai.com/docs/api-reference/responses) in Semantic Kernel.
3133

3234
## Configuring the Kernel
3335

@@ -57,6 +59,9 @@ You can explicitly create a specific implementation for the desired `Agent` that
5759
Below is a sample code snippet demonstrating thread management:
5860

5961
```python
62+
from semantic_kernel.agents import ChatCompletionAgent
63+
from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
64+
6065
USER_INPUTS = [
6166
"Why is the sky blue?",
6267
]
@@ -71,7 +76,7 @@ agent = ChatCompletionAgent(
7176
# 2. Create a thread to hold the conversation
7277
# If no thread is provided, a new thread will be
7378
# created and returned with the initial response
74-
thread: ChatCompletionAgentThread = None
79+
thread = None
7580

7681
for user_input in USER_INPUTS:
7782
print(f"# User: {user_input}")
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Copyright (c) Microsoft. All rights reserved.
2+
import asyncio
3+
import os
4+
5+
from semantic_kernel.agents import OpenAIResponsesAgent
6+
from semantic_kernel.contents.streaming_chat_message_content import StreamingChatMessageContent
7+
8+
"""
9+
The following sample demonstrates how to create an OpenAI Responses Agent.
10+
The sample shows how to have the agent answer questions about the provided
11+
document with streaming responses.
12+
13+
The interaction with the agent is via the `get_response` method, which sends a
14+
user input to the agent and receives a response from the agent. The conversation
15+
history is maintained by the agent service, i.e. the responses are automatically
16+
associated with the thread. Therefore, client code does not need to maintain the
17+
conversation history.
18+
"""
19+
20+
21+
# Simulate a conversation with the agent
22+
USER_INPUTS = [
23+
"By birthday, who is the youngest employee?",
24+
"Who works in sales?",
25+
"I have a customer request, who can help me?",
26+
]
27+
28+
29+
async def main():
30+
# 1. Create the client using OpenAI resources and configuration
31+
client, model = OpenAIResponsesAgent.setup_resources()
32+
33+
pdf_file_path = os.path.join(
34+
os.path.dirname(os.path.dirname(os.path.realpath(__file__))), "resources", "employees.pdf"
35+
)
36+
37+
with open(pdf_file_path, "rb") as file:
38+
file = await client.files.create(file=file, purpose="assistants")
39+
40+
vector_store = await client.vector_stores.create(
41+
name="step4_assistant_file_search",
42+
file_ids=[file.id],
43+
)
44+
45+
file_search_tool = OpenAIResponsesAgent.configure_file_search_tool(vector_store.id)
46+
47+
# 2. Create a Semantic Kernel agent for the OpenAI Responses API
48+
agent = OpenAIResponsesAgent(
49+
ai_model_id=model,
50+
client=client,
51+
instructions="Find answers to the user's questions in the provided file.",
52+
name="FileSearch",
53+
tools=[file_search_tool],
54+
)
55+
56+
# 3. Create a thread for the agent
57+
# If no thread is provided, a new thread will be
58+
# created and returned with the initial response
59+
thread = None
60+
61+
response_chunks: list[StreamingChatMessageContent] = []
62+
for user_input in USER_INPUTS:
63+
print(f"# User: '{user_input}'")
64+
# 4. Invoke the agent for the current message and print the response
65+
first_chunk = True
66+
async for response in agent.invoke_stream(messages=user_input, thread=thread):
67+
thread = response.thread
68+
response_chunks.append(response)
69+
if first_chunk:
70+
print(f"# {response.name}: ", end="", flush=True)
71+
first_chunk = False
72+
print(response.content, end="", flush=True)
73+
print()
74+
75+
"""
76+
# User: 'By birthday, who is the youngest employee?'
77+
# Agent: The youngest employee by birthday is Teodor Britton, born on January 9, 1997.
78+
# User: 'Who works in sales?'
79+
# Agent: The employees who work in sales are:
80+
81+
- Mariam Jaslyn, Sales Representative
82+
- Hicran Bea, Sales Manager
83+
- Angelino Embla, Sales Representative.
84+
# User: 'I have a customer request, who can help me?'
85+
# Agent: For a customer request, you could reach out to the following people in the sales department:
86+
87+
- Mariam Jaslyn, Sales Representative
88+
- Hicran Bea, Sales Manager
89+
- Angelino Embla, Sales Representative.
90+
"""
91+
92+
93+
if __name__ == "__main__":
94+
asyncio.run(main())
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Copyright (c) Microsoft. All rights reserved.
2+
import asyncio
3+
from typing import Annotated
4+
5+
from semantic_kernel.agents import AzureResponsesAgent
6+
from semantic_kernel.contents import AuthorRole, FunctionCallContent, FunctionResultContent
7+
from semantic_kernel.contents.chat_message_content import ChatMessageContent
8+
from semantic_kernel.functions import kernel_function
9+
10+
"""
11+
The following sample demonstrates how to create an OpenAI
12+
Responses Agent using either Azure OpenAI or OpenAI. The
13+
Responses Agent allow for function calling, the use of file search and a
14+
web search tool. Responses Agent Threads are used to manage the
15+
conversation state, similar to a Semantic Kernel Chat History.
16+
Additionally, the invoke configures a message callback
17+
to receive the conversation messages during invocation.
18+
"""
19+
20+
21+
# Define a sample plugin for the sample
22+
class MenuPlugin:
23+
"""A sample Menu Plugin used for the concept sample."""
24+
25+
@kernel_function(description="Provides a list of specials from the menu.")
26+
def get_specials(self) -> Annotated[str, "Returns the specials from the menu."]:
27+
return """
28+
Special Soup: Clam Chowder
29+
Special Salad: Cobb Salad
30+
Special Drink: Chai Tea
31+
"""
32+
33+
@kernel_function(description="Provides the price of the requested menu item.")
34+
def get_item_price(
35+
self, menu_item: Annotated[str, "The name of the menu item."]
36+
) -> Annotated[str, "Returns the price of the menu item."]:
37+
return "$9.99"
38+
39+
40+
intermediate_steps: list[ChatMessageContent] = []
41+
42+
43+
async def handle_intermediate_steps(message: ChatMessageContent) -> None:
44+
intermediate_steps.append(message)
45+
46+
47+
async def main():
48+
# 1. Create the client using Azure OpenAI resources and configuration
49+
client, model = AzureResponsesAgent.setup_resources()
50+
51+
# 2. Create a Semantic Kernel agent for the OpenAI Responses API
52+
agent = AzureResponsesAgent(
53+
ai_model_id=model,
54+
client=client,
55+
name="Host",
56+
instructions="Answer questions about the menu.",
57+
plugins=[MenuPlugin()],
58+
)
59+
60+
# 3. Create a thread for the agent
61+
# If no thread is provided, a new thread will be
62+
# created and returned with the initial response
63+
thread = None
64+
65+
user_inputs = ["Hello", "What is the special soup?", "What is the special drink?", "How much is that?", "Thank you"]
66+
67+
try:
68+
for user_input in user_inputs:
69+
print(f"# {AuthorRole.USER}: '{user_input}'")
70+
async for response in agent.invoke(
71+
messages=user_input,
72+
thread=thread,
73+
on_intermediate_message=handle_intermediate_steps,
74+
):
75+
thread = response.thread
76+
print(f"# {response.name}: {response.content}")
77+
finally:
78+
await thread.delete() if thread else None
79+
80+
# Print the final chat history
81+
print("\nIntermediate Steps:")
82+
for msg in intermediate_steps:
83+
if any(isinstance(item, FunctionResultContent) for item in msg.items):
84+
for fr in msg.items:
85+
if isinstance(fr, FunctionResultContent):
86+
print(f"Function Result:> {fr.result} for function: {fr.name}")
87+
elif any(isinstance(item, FunctionCallContent) for item in msg.items):
88+
for fcc in msg.items:
89+
if isinstance(fcc, FunctionCallContent):
90+
print(f"Function Call:> {fcc.name} with arguments: {fcc.arguments}")
91+
else:
92+
print(f"{msg.role}: {msg.content}")
93+
94+
"""
95+
Sample Output:
96+
97+
# AuthorRole.USER: 'Hello'
98+
# Host: Hi there! How can I assist you with the menu today?
99+
# AuthorRole.USER: 'What is the special soup?'
100+
# Host: The special soup is Clam Chowder.
101+
# AuthorRole.USER: 'What is the special drink?'
102+
# Host: The special drink is Chai Tea.
103+
# AuthorRole.USER: 'How much is that?'
104+
# Host: Could you please specify the menu item you are asking about?
105+
# AuthorRole.USER: 'Thank you'
106+
# Host: You're welcome! If you have any questions about the menu or need assistance, feel free to ask.
107+
108+
Intermediate Steps:
109+
AuthorRole.ASSISTANT: Hi there! How can I assist you with the menu today?
110+
AuthorRole.ASSISTANT:
111+
Function Result:>
112+
Special Soup: Clam Chowder
113+
Special Salad: Cobb Salad
114+
Special Drink: Chai Tea
115+
for function: MenuPlugin-get_specials
116+
AuthorRole.ASSISTANT: The special soup is Clam Chowder.
117+
AuthorRole.ASSISTANT:
118+
Function Result:>
119+
Special Soup: Clam Chowder
120+
Special Salad: Cobb Salad
121+
Special Drink: Chai Tea
122+
for function: MenuPlugin-get_specials
123+
AuthorRole.ASSISTANT: The special drink is Chai Tea.
124+
AuthorRole.ASSISTANT: Could you please specify the menu item you are asking about?
125+
AuthorRole.ASSISTANT: You're welcome! If you have any questions about the menu or need assistance, feel free to ask.
126+
"""
127+
128+
129+
if __name__ == "__main__":
130+
asyncio.run(main())

0 commit comments

Comments
 (0)