PyInit_ symbol export errors on Windows PyPy builds

[jerstlouis]

As I understand it, PyInit_ is a CPython thing.

The code here:

cffi/src/cffi/vengine_gen.py

Line 59 in 2b81170

prefix = 'PyInit_'

         if sys.platform == 'win32':
            if sys.version_info >= (3,):
                prefix = 'PyInit_'
            else:
                prefix = 'init'

seems to always generate a .def file which will cause a linker failure on trying to export the non-existing PyInit_ symbol.

I would expect a _cffi_pypyinit_ prefix for PyPy.

This makes CFFI unusable on Windows / PyPy platforms, and is quite a difficult problem to track down.

(e.g., https://github.com/ecere/pyecrt/actions/runs/14982225821/job/42088776136#step:7:3509)

CFFI extension code is generated as such:

#ifdef PYPY_VERSION
PyMODINIT_FUNC
_cffi_pypyinit_myext(const void *p[])
{
    p[0] = (const void *)0x2601;
    p[1] = &_cffi_type_context;
#if PY_MAJOR_VERSION >= 3
    return NULL;
#endif
}
#  ifdef _MSC_VER
     PyMODINIT_FUNC
#  if PY_MAJOR_VERSION >= 3
     PyInit_myext(void) { return NULL; }
#  else
     initmyext(void) { }
#  endif
#  endif
#elif PY_MAJOR_VERSION >= 3
PyMODINIT_FUNC
PyInit_myext(void)
{
  return _cffi_init("myext", 0x2601, &_cffi_type_context);
}
#else
PyMODINIT_FUNC
initmyext(void)
{
  _cffi_init("myext", 0x2601, &_cffi_type_context);
}
#endif

[arigo]

Additional context: this seems to be about MINGW32 and not MSVC. This is not an officially supported platform, but if you want to provide (and test!) a fix, it would be accepted.

[jerstlouis]

@arigo Is the context where this PyInit_ is set here specific to MinGW?

I believe this applies with PyPy regardless of the compiler (MSVC or MinGW-w64).

Is PyPy still supported by CFFI?

[arigo]

Yes, I see that the issue applies regardless of the compiler. However I don't claim to understand the original reason for the .def file issue, nor do I have a way to test it, because it appears to work fine with MSVC on Windows 11 with PyPy. I guess the .def content is ignored in this case? PyPy's own tests run the CFFI tests too, so I'd expect major breakage to be noticed. It seems that we just never tested the combination of PyPy+CFFI+MinGW32. If you can provide a patch or pull request, I can make sure it also works on PyPy+CFFI+MSVC.

[jerstlouis]

Thanks @arigo for looking into this. The patch might be as simple as if <pypy>: prefix = _cffi_pypyinit_ else: ...

Perhaps the .def file is not used by the MSVC builds?

as for testing it, I'm not familiar with the environment at all -- I just couldn't get my PyPy / Windows builds to pass on cibuildwheels.

If you can provide instructions on how to set up the test environment I can try.

[arigo]

I'm afraid I don't know how to set up a test environment, particularly one which would run MinGW-w64. Nor do I know if and how you can use your github action to run with a custom patch inside PyPy (which contains its own copy of CFFI). There are probably really hackish ways to do that (like run some Python script that patches the code in cffi\vengine_gen.py), but I don't know if there is an official way. If you want to investigate it more, try to ask pypy-dev@python.org or the #pypy channel on IRC (irc.libera.chat). I'm also fine if you simply give a pull request here that "seems" to do the right thing; it will be tested against MSVC, at least. Note that you'd still have to wait until the next release of PyPy (and maybe CFFI before that, or maybe not; I can convince them to pull the changes from CFFI early).

[jerstlouis]

Thanks @arigo, I'll try discussing this in #pypy and see what they think, and can probably try patching vengine_gen.py in the cibuildwheels envirionment -- that seems like the easiest way for me to test given that I am not familiar with PyPy at all.

[arigo]

Yes, as long as you remember you need to patch the one that is distributed with PyPy. Running or installing programs inside PyPy shouldn't load the official CFFI from the python-cffi/cffi repository nor from the pypi package index, but just use the one that is included in lib_pypy/cffi/ with PyPy. [edit: on Windows it's Lib\cffi\]