Feature Request: Optimize for Nvidia Jetson Series' truly Unified Memory Architecture

[Yangxiaoz]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Currently, the code does not differentiate for Nvidia Jetson series edge boards where the CPU and GPU share unified physical memory. Regardless of whether the GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 flag is enabled, weight tensors are still copied via Host-to-Device transfers. This appears unnecessary for Unified Memory (UM) or pinned memory scenarios, as both architectures allow direct access without explicit duplication

Motivation

As shown in the figure above, even after enabling GGML_CUDA_ENABLE_UNIFIED_MEMORY=1, pointers allocated via cudaMallocManaged remain accessible to the CPU, yet explicit copy operations (MemcpyHostToDevice)) still occur in this code path. For Jetson's UMA architecture, could we optimize this behavior to eliminate redundant transfers and achieve true zero-copy access?

Possible Implementation

No response

[Yangxiaoz]

Furthermore, on Jetson devices, pinned memory allocated for GPU access (i.e., host_buffer) appears to be flagged by ggml's internal logic as unsupported by the CUDA device backend (device_cuda). This limitation has blocked my attempts to implement Small hack experiment in my local branch.For examples, Jetson devices with compute capability >7.2, pinned memory can be accessed simultaneously by both the CPU and GPU, but it is not cached by CUDA. This implies that the overhead spent on maintaining I/O coherency in Unified Memory is unnecessary for pinned memory.