Runtime stage for Dockerfile

[mtaron]

Hello! I was just trying out llgtrt yesterday and was blown away by how easy it was it get LoRA adapters up and running. I was struggling for over a week to get Phi4 + LoRA to work with TensorRT-LLM + Triton Server and finally gave up. The performance of this server is phenomenal as well and I am so happy not to have to mess with these gigantic config.pbtxt files that change every release too. Wow, I am so impressed with this project! Thank you!

One thing I noticed though is that the docker image is quite large - 35 GB as measured by dive. It includes a bunch of stuff like the Rust toolchain, a bunch of Python libs, etc. that are not required for runtime. Much of the fault lies with the NVIDIA tensorrt container, which is a total mess.

I was able to get the image size down to 7.7 GB (78% smaller) by having this as the final stage:

FROM nvcr.io/nvidia/cuda:12.8.1-runtime-ubuntu24.04 AS llgtrt_prod

RUN DEBIAN_FRONTEND=noninteractive apt-get update \
    && apt-get upgrade -y \
    && apt-get install -y --no-install-recommends \
        # These are runtime dependencies of tensorrt_llm
        libpython3.12-dev \
        libopenmpi-dev \
    && rm -rf /var/lib/apt/lists/*

COPY --from=llgtrt_builder /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs /usr/local/lib
COPY --from=llgtrt_builder /usr/lib/x86_64-linux-gnu/libnvinfer.so.10 /usr/local/lib/libnvinfer.so.10
COPY --from=llgtrt_builder /workspaces/llgtrt/target/release/llgtrt /usr/local/bin/llgtrt

I haven't tested/don't know the full range of llgtrt capabilities, but Phi4 + five LoRA adapters works great with this as the final image.

I recommend producing two docker images, one for model building (that should include the lora export script, btw) and one for runtime. To avoid breaking changes, the runtime stage could be published as llgtrt:<version>-runtime.

Having a smaller image also helps with security (just surface area) and auto-scale time (doesn't take as long to pull the image).

[mmoskal]

Thank you for your kind words!

A few weeks back we added python plugin support #17 for vision models (they are not supported with C++ only). Any idea how would this interact with the cut-down image?

[mtaron]

Hm, I'm not sure - might just need to apt install python. One of the scientists on my team wants to use a vision model in his next project so I'll probably use this vision support soon and work it out. In the mean time, here is the full Dockerfile I'm using to build with the C++11 ABI. It copied over the prebuilt libs from the tritonserver image. There's probably a better way to do all /usr/local/lib and /usr/local/include stuff, but I'm not a C++/Rust dev so just hacking till it boots. This image is 9.2 GB uncompressed. I guess that delta is turning on C++11 ABI since I don't think other things should have changed? In any case, might be helpful for others.

FROM nvcr.io/nvidia/tritonserver:25.03-trtllm-python-py3 AS builder

# tensorrt_llm built in the base image uses the C++11 ABI
ENV USE_CXX11_ABI=1

# install the rust toolchain
ENV RUSTUP_HOME=/usr/local/rustup \
    CARGO_HOME=/usr/local/cargo \
    PATH=/usr/local/cargo/bin:$PATH \
    RUST_VERSION=1.83.0

RUN curl https://sh.rustup.rs -sSf | sh -s -- \
     -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION \
     && rustup component add rustfmt \
     && cargo install rustfilt

# copy tensorrt and tensortrt_llm lib and include files to somewhere they can be found by the linker
RUN cp /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/* /usr/local/lib \
    && cd /usr/local/lib \
    && ln -s libnvinfer_plugin_tensorrt_llm.so libnvinfer_plugin_tensorrt_llm.so.10 \
    && cp -r /usr/local/tensorrt/targets/x86_64-linux-gnu/lib/* /usr/local/lib \
    && cp /usr/local/tensorrt/targets/x86_64-linux-gnu/include/* /usr/local/include \
    # link stub, so that it builds without nvidia-runtime
    && cd /usr/local/cuda/lib64 && ln -s stubs/libnvidia-ml.so libnvidia-ml.so.1

# install build dependencies
RUN DEBIAN_FRONTEND=noninteractive apt-get update \
    && apt-get upgrade -y \
    && apt-get install -y --no-install-recommends \
        cmake \
        clang \
        pkg-config \
        libssl-dev \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /workspace/llgtrt

COPY . .

RUN /workspace/llgtrt/scripts/build.sh --clean --cxx11abi $USE_CXX11_ABI

FROM nvcr.io/nvidia/cuda:12.8.1-runtime-ubuntu24.04

# install runtime dependencies
RUN DEBIAN_FRONTEND=noninteractive apt-get update \
    && apt-get upgrade -y \
    && apt-get install -y --no-install-recommends \
        libpython3.12-dev \
        libopenmpi-dev \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs /usr/local/lib
COPY --from=builder /usr/local/lib/libnvinfer.so.10 /usr/local/lib/libnvinfer.so.10
COPY --from=builder /workspace/llgtrt/target/release/llgtrt /usr/local/bin/llgtrt
COPY --from=builder /workspace/llgtrt/scripts/launch-llgtrt.sh /usr/local/bin/launch-llgtrt.sh

# use a non-root user
USER ubuntu

# override entrypoint for runtime
ENTRYPOINT [ "/usr/local/bin/launch-llgtrt.sh" ]