Sketch code generator

[aliabid94]

Now we can use code generation directly in gradio sketch.

I added a "_value_description" property to components so they can describe what values they can have, which the LLM code generator needs to know. This was necessary, because often it was not enough to just read the docstring of a component - the value expected depends often on other kwargs (especially type=).

[gradio-pr-bot]

🪼 branch checks and previews

•	Name	Status	URL
	Spaces	ready!	Spaces preview
	Website	ready!	Website preview
🦄	Changes	detected!	Details

---

Install Gradio from this PR

pip install https://gradio-pypi-previews.s3.amazonaws.com/8408294762bce28ea5aa66145dfeee681f76e18b/gradio-5.21.0-py3-none-any.whl

Install Gradio Python Client from this PR

pip install "gradio-client @ git+https://github.com/gradio-app/gradio@8408294762bce28ea5aa66145dfeee681f76e18b#subdirectory=client/python"

Install Gradio JS Client from this PR

npm install https://gradio-npm-previews.s3.amazonaws.com/8408294762bce28ea5aa66145dfeee681f76e18b/gradio-client-1.13.1.tgz

Use Lite from this PR

<script type="module" src="https://gradio-lite-previews.s3.amazonaws.com/8408294762bce28ea5aa66145dfeee681f76e18b/dist/lite.js""></script>

[gradio-pr-bot]

🦄 change detected

This Pull Request includes changes to the following packages.

Package	Version
gradio	minor

• Maintainers can select this checkbox to manually select packages to update.

With the following changelog entry.

Sketch code generator

Maintainers or the PR author can modify the PR title to modify this entry.

Something isn't right?

• Maintainers can change the version label to modify the version bump.
• If the bot has failed to detect any changes, or if this pull request needs to update multiple packages to different versions or requires a more comprehensive changelog entry, maintainers can update the changelog file directly.

[abidlabs]

@aliabid94 works great! Just a few nits on the usage:

1. Could we lower the duration of the success modals? Feels very long after seeing a few of them:

2. If you are logged out, you have a nice way to have a user input their HF token. Perhaps mention that the token needs to have permissions to call inference providers

Likewise, if the token doesn't have permissions to call inference providers and you get an error, it might be a good idea to catch and explain that error

3. My main frustration is that sometimes the generated code uses external libraries that are not in the context, which makes the rendered app unusable. Perhaps we should encourage the LLM not to use external libraries that are not gradio dependencies

[abidlabs]

Not a huge fan of adding ._value_description, because of the increased maintenance involved. Particularly as the usage of this parameter is quite decoupled from its place in the code. i.e. we are adding these to component classes but we are using these in a completely different part of the code (Sketch). Is it truly necessary -- can't the LLM understand how to combine the docstring with the type of the component?

Suggested change

	self._value_description = "a tuple of type [image: str, annotations: list[tuple[mask: str, label: str]]] where 'image' is the path to the base image and 'annotations' is a list of tuples where each tuple has a 'mask' image filepath and a corresponding label."
	self._value_description = "a tuple of type [image: str, annotations: list[tuple[mask: str, label: str]]] where 'image' is the path to the base image and 'annotations' is a list of tuples where each tuple has a 'mask' image filepath and a corresponding label."

[abidlabs]

I think it's worth adding the other supported tasks here: https://huggingface.co/docs/huggingface_hub/en/guides/inference#supported-providers-and-tasks

I'm sure people will try things like image generation, which right now causes the generated functions to use obscure dependencies:

[aliabid94]

Could we lower the duration of the success modals? Feels very long after seeing a few of them:

Done

If you are logged out, you have a nice way to have a user input their HF token. Perhaps mention that the token needs to have permissions to call inference providers

Done.

Likewise, if the token doesn't have permissions to call inference providers and you get an error, it might be a good idea to catch and explain that error

Done.

My main frustration is that sometimes the generated code uses external libraries that are not in the context, which makes the rendered app unusable. Perhaps we should encourage the LLM not to use external libraries that are not gradio dependencies

Done.

Not a huge fan of adding ._value_description, because of the increased maintenance involved. Particularly as the usage of this parameter is quite decoupled from its place in the code. i.e. we are adding these to component classes but we are using these in a completely different part of the code (Sketch). Is it truly necessary -- can't the LLM understand how to combine the docstring with the type of the component?

I don't love it either but the description of value and type together aren't enough. By default, it just uses the type of value if no _value_description is provided (see how gr.Textbox doesn't need a _value_description for example, because it is enough to say that value will just be a string). But for other components, many other kwargs that are needed to know the final type. For example, it is necessary for the LLM to know the choices for a dropdown, so that it knows the values the function can receive (or return if dropdown is an output), and it needs to know if the dropdown is multiselect to know whether to expect/return a list or a single string.

I think it's worth adding the other supported tasks here

Ok I plann on adding most tasks in a follow up PR but I've added image generation for now.

[freddyaboulton]

Nice @aliabid94 ! Just tried out some couple of examples (image gen via API, chatbot via inference provider, simple image editing) and the experience was great. I also don't love ._value_description but was impressed at how well the generated function inferred the correct types.

Some feedback on the experience:

• When you go back to edit a sketch after "Save and Render", it would be nice if the function tab also opened.
• It would be nice if the LLM's generation used the previous generation in its context. In my simple image editing example, I asked the LLM to generate code that either flipped an image, applied a sepia filter, or turned it black and white depending on a drop down value. After saving and rendering, I noticed the sepia logic had a bug. So I went back and pasted the error message and told it to fix the sepia logic. But it rewrote everything and although it fixed the sepia filter it broke the black and white filter lol.
• The LLM doesn't know about streaming outputs. It generated the correct code for using the LLM inference providers but then when I asked it to stream the response it generated this:

def chat(multimodaltextbox, chatbot):
    import huggingface_hub
    from io import StringIO
    import sys

    # Initialize the inference client
    client = huggingface_hub.InferenceClient()

    # Prepare the input for the chat model
    messages = chatbot + [{'role': 'user', 'content': multimodaltextbox['text']}]

    # Function to capture streaming output
    class Capturing(list):
        def __enter__(self):
            self._stdout = sys.stdout
            sys.stdout = self._stringio = StringIO()
            return self
        def __exit__(self, *args):
            self.extend(self._stringio.getvalue().splitlines())
            del self._stringio    # free up some memory
            sys.stdout = self._stdout

    # Capture the streaming response
    with Capturing() as output:
        response = client.chat_completion(messages, stream=True)

    # Append each streamed response to the chatbot
    for line in output:
        chatbot.append({'role': 'assistant', 'content': line})

    # Clear the textbox for the next message
    cleared_textbox = {'text': '', 'files': []}

    return chatbot, cleared_textbox

[aliabid94]

When you go back to edit a sketch after "Save and Render", it would be nice if the function tab also opened.

Done.

It would be nice if the LLM's generation used the previous generation in its context. In my simple image editing example, I asked the LLM to generate code that either flipped an image, applied a sepia filter, or turned it black and white depending on a drop down value. After saving and rendering, I noticed the sepia logic had a bug. So I went back and pasted the error message and told it to fix the sepia logic. But it rewrote everything and although it fixed the sepia filter it broke the black and white filter lol.

The LLM doesn't know about streaming outputs. It generated the correct code for using the LLM inference providers but then when I asked it to stream the response it generated this:

I think there's still a bit of work to be done on improving the prompting, I'll do these in a follow up PR