Skip to content

feat: add custom Docker image support and enhanced configuration for DockerExecutor #1057

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

kingdomad
Copy link
Contributor

@kingdomad kingdomad commented Mar 23, 2025

Add Custom Docker Image Support and Enhanced Configuration for DockerExecutor

Description

This PR enhances the DockerExecutor class by adding the ability to specify custom Docker image names, control the image building process, and configure container runtime parameters. Previously, the executor would always build a new image with a hardcoded name ("jupyter-kernel") on initialization and had limited container configuration options.

Changes

  • Added new parameters to DockerExecutor.__init__:

    • image_name: Allows specifying a custom Docker image name (default: "jupyter-kernel")
    • rebuild_image: Controls whether to rebuild the image even if it exists (default: True)
    • container_run_kwargs: Enables passing additional keyword arguments to the Docker container run command
  • Implemented image existence checking logic:

    • If rebuild_image is False, the executor will check if the specified image already exists
    • If the image exists, it will be used directly without rebuilding
    • If the image doesn't exist, it will be built automatically
  • Allows users to customize container settings while maintaining core functionality

Usage Example

# Use existing image without rebuilding
executor = DockerExecutor(
    additional_imports=["numpy", "pandas"],
    logger=logger,
    image_name="my-custom-jupyter",
    rebuild_image=False
)

# Force rebuild of an image
executor = DockerExecutor(
    additional_imports=["numpy", "pandas"],
    logger=logger,
    image_name="jupyter-kernel",
    rebuild_image=True
)

# Use custom container configuration
executor = DockerExecutor(
    additional_imports=["numpy", "pandas"],
    logger=logger,
    container_run_kwargs={
        "extra_hosts": {"host.docker.internal": "host-gateway"},
        "environment": {
            "HTTP_PROXY": "http://host.docker.internal:7890",
            "HTTPS_PROXY": "http://host.docker.internal:7890",
        },
    }
)

@kingdomad kingdomad changed the title feat: add custom Docker image name support feat: add custom Docker image xupport and enhanced configuration for DockerExecutor Mar 24, 2025
@kingdomad kingdomad changed the title feat: add custom Docker image xupport and enhanced configuration for DockerExecutor feat: add custom Docker image support and enhanced configuration for DockerExecutor Mar 24, 2025
@aymeric-roucher
Copy link
Collaborator

One comment above: otherwise we should be good to go!

@kingdomad
Copy link
Contributor Author

One comment above: otherwise we should be good to go!

Done!

Copy link
Member

@albertvillanova albertvillanova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@albertvillanova albertvillanova merged commit 0eefeb6 into huggingface:main Apr 14, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants