You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When collecting a lazy query with the GPU engine, we can currently only materialise the end result in CPU memory. If we want to end up with the final result "at rest" as parquet (or other on-disk format), we incur the cost of moving to CPU memory and then writing from there. libcudf has GPU-accelerated IO writers as well as readers, so it would be nice to be able to sink straight from GPU memory.
I think this is doable in the same way we currently hook into collect, by adding an engine argument.
The text was updated successfully, but these errors were encountered:
Description
When collecting a lazy query with the GPU engine, we can currently only materialise the end result in CPU memory. If we want to end up with the final result "at rest" as parquet (or other on-disk format), we incur the cost of moving to CPU memory and then writing from there. libcudf has GPU-accelerated IO writers as well as readers, so it would be nice to be able to sink straight from GPU memory.
I think this is doable in the same way we currently hook into
collect
, by adding anengine
argument.The text was updated successfully, but these errors were encountered: