Skip to content

fix tool code generation in case of multiline descriptions #613

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 14, 2025

Conversation

sysradium
Copy link
Contributor

@sysradium sysradium commented Feb 11, 2025

If a tool decorated with @tool contains a multiline comment:

@tool
def ssh_execute_command_with_agent(host: str, port: int, username: str, command: str) -> str:
    """This tool connects to a specified SSH server using the SSH agent for authentication
    and executes a given command on the server. It returns the output of the command
    or any errors encountered during execution.

    Args:
        host: The hostname or IP address of the server.
        port: The SSH port to connect to (default is 22).
        username: The SSH username for authentication.
        command: The command to execute on the server.

    Returns:
        A string containing the command's output or any errors encountered.

    """

It breaks to_dict. It produces something like this:

from smolagents import Tool
            from typing import Any, Optional

            class SimpleTool(Tool):
                name = "ssh_execute_command_with_agent"
                description = This tool connects to a specified SSH server using the SSH agent for authentication
and executes a given command on the server. It returns the output of the command
or any errors encountered during execution.
                inputs = {"host":{"type":"string","description":"The hostname or IP address of the server."},"port":{"type":"integer","description":"The SSH port to connect to (default is 22)."},"username":{"type":"string","description":"The SSH username for authentication."},"command":{"type":"string","description":"The command to execute on the server."}}
                output_type = "string"

    def forward(self, host: str, port: int, username: str, command: str) -> str:

@sysradium sysradium mentioned this pull request Feb 11, 2025
Copy link
Contributor

@aymeric-roucher aymeric-roucher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @sysradium !

Copy link
Member

@albertvillanova albertvillanova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! It would be awesome to have a regression test.

Copy link
Member

@albertvillanova albertvillanova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering about an alternative approach.

@sysradium
Copy link
Contributor Author

@albertvillanova frankly speaking I wanted to refactor tests a bit :) They became a bit too hard to maintain and can be improved (for example by removing side-effects). But maybe I can squeeze in a test for this specific case. No problem

@sysradium sysradium force-pushed the fix-tool-code-generation branch from 1068ec9 to 030a83f Compare February 13, 2025 00:01
@sysradium
Copy link
Contributor Author

sysradium commented Feb 13, 2025

@aymeric-roucher
Fix a similar problem with description generation when tool is defined as a class. Added tests.

For the future do you have any particular attachment to old-school unittest.TestCase, or as I go I can change then and/or use pytest tests with fixture and parametrization?

@sysradium sysradium force-pushed the fix-tool-code-generation branch from 030a83f to 41d276a Compare February 13, 2025 08:55
@aymeric-roucher
Copy link
Contributor

@albertvillanova about @sysradium's question above: which do you prefer? I see you've introduced many pytest tests, are these preferred?

@sysradium
Copy link
Contributor Author

sysradium commented Feb 13, 2025

I advise to switch to pytest, because you are using pytest runner anyways, but loose the benefits pytest brings, e.g.: https://docs.pytest.org/en/stable/how-to/unittest.html#pytest-features-in-unittest-testcase-subclasses

classes are a nice way to group tests, probably, but instead you can group them using files.

@albertvillanova
Copy link
Member

@albertvillanova about @sysradium's question above: which do you prefer? I see you've introduced many pytest tests, are these preferred?

We are using pytest to run our test suite, so better use pytest tests:

smolagents/Makefile

Lines 16 to 18 in 0153e37

# Run smolagents tests
test:
pytest ./tests/

uv run pytest ./tests/test_agents.py

However:

  • no need to migrate all tests: unittest tests are supported by pytest
  • the use of test classes is also possible in pytest: we align test file names with those of the library under test

@aymeric-roucher aymeric-roucher merged commit d9da0a7 into huggingface:main Feb 14, 2025
3 checks passed
@sysradium sysradium deleted the fix-tool-code-generation branch February 14, 2025 11:37
@sysradium
Copy link
Contributor Author

@albertvillanova we just loose a bit of feature if we keep using classes. For example, you can't use tables-tests, or mark tests as deprecated, make them conditioned on python version.

I am not saying we need to go a rewrite everything, but maybe newer test can be just pure functions instead.

I know it is a heated discussion, and quite often people still push for classes, that's why I decided to ask first :) Currently it is a bit of a mix in this project, I want to have some alignment on that.

@albertvillanova
Copy link
Member

@sysradium when I say we can use test classes in pytest, I don't mean to use unittest classes, but just regular classes.

And I remember I have already deprecated and skipped tests that were implemented as pytest class methods.

Here the link to Pytest docs about grouping tests in a class: https://docs.pytest.org/en/stable/getting-started.html#group-multiple-tests-in-a-class

@sysradium
Copy link
Contributor Author

@albertvillanova my bad. Understood. Thank you! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants