Avoiding Redundant Rust ↔ Python ↔ Rust Conversions for Known Callback Shapes?

[squidfunk]

Hi all 👋

I'm currently working on a project using PyO3 in which I export Rust functions to Python that can then be passed back as callbacks to other exported Rust functions. In this setup, I'm wondering if there's a way to avoid the overhead of Rust → Python → Rust conversions, specifically in cases where I know the exact shape of the function being called.

Minimal Example

Let’s say I have a Rust struct Foo and a function just_print_foo exposed to Python:

use pyo3::prelude::*;

#[pyclass]
#[derive(Clone, Debug, Default)]
pub struct Foo {
    #[pyo3(get, set)]
    pub bar: usize,
}

#[pyfunction]
fn just_print_foo(value: Foo) {
    println!("Foo: {value:?}");
}

Now, I define another function that receives a Python callback and calls it with a Foo:

#[pyfunction]
fn call_lambda(f: Bound<'_, PyAny>) -> PyResult<()> {
    let foo = Foo::default();
    // Foo is marshalled Rust -> Python
    let result = f.call1((foo,))?;
    // Foo is marshalled Python -> Rust
    let foo = result.extract::<Foo>()?;
    println!("Foo: {:?}", foo);
    Ok(())
}

This works fine, but involves two marshaling steps (Rust → Python and back). What I’m wondering is: if I know ahead of time that the callback is actually a Rust function (e.g., just_print_foo), is there a way to "recover" that original Rust function — perhaps via downcasting or some other mechanism — and bypass Python entirely?

Hypothetical API

Something like this is what I have in mind:

#[pyfunction]
fn call_lambda(f: Bound<'_, PyAny>) -> PyResult<()> {
    let foo = Foo::default();
    // Hypothetical: downcast to the original Rust function
    let wrapped_f = f.downcast_to_wrapped_type::<fn(Foo)>()?;
    wrapped_f(foo); // Direct Rust call — no marshalling
    Ok(())
}

I understand this might be tricky (or not even possible), given how PyO3 handles wrapping and registration of functions, but is there any current mechanism or planned feature that could help in cases like this?

Apologies if this is a naive question — I’m still fairly new to PyO3 and getting familiar with how these internals work. I also didn't find any similar discussions, so apologies if this has been asked already – happy for a link to prior discussions as well.

Thanks in advance for any insights!

[squidfunk]

I've been exploring this a bit further and stumbled across PyCapsule, which might actually be the right mechanism for what I'm trying to do. Here's a rough example I put together. I'm using a capsule to wrap a Rust function pointer and pass it through Python, then retrieve it on the Rust side and call it directly — bypassing Python → Rust marshalling:

fn rust_func(foo: Foo) {
    println!("{foo:?}");
}

#[pyfunction]
fn get_callback_in_capsule(py: Python<'_>) -> PyObject {
    let ptr: fn(Foo) = rust_func;
    let name = CString::new("rust_func").unwrap();
    PyCapsule::new(py, ptr, Some(name)).unwrap().into()}

#[pyfunction]
fn call_callback_in_capsule(object: PyObject, foo: Foo) -> PyResult<()> {
    Python::with_gil(|py| {
        let capsule = object.downcast_bound::<PyCapsule>(py)?;
        let f = unsafe { capsule.reference::<fn(Foo)>()? };

        // Call with default Foo (no marshalling)
        f(Foo::default());

        // Call with value from Python (already marshalled)
        f(foo);

        Ok(())
    })
}

This seems to work as a low-level workaround — the unsafe block is obviously something to be careful about, but as long as the capsule contains exactly what I expect, it's viable. I also need to find a way to export the function to Python, so it can be called in Python, and only retrieve the original function as part of the capsule when I pass it as a callback to Rust.

Does this seem like a reasonable direction? Is there a better/more idiomatic way to do this in PyO3?