pycryptodom fails to run with CFFI and PYTHONOPTIMIZE=2 due to using cffi in-line ABI mode

[schlenk]

Hi,

when using the cffi backend with optimize (PYTHONOPTIMIZE=2) the loading of cryptodome fails due to a bug in pycparser (see eliben/pycparser#198 ). See below for the error.

It should be possible to work around that issue, if pycryptodome used the 'out-of-line' feature of cffi, to compile/produce the API wrappers upfront during installation. See https://cffi.readthedocs.io/en/latest/cdef.html#ffibuilder-set-source-preparing-out-of-line-modules

As a side effect, using the out-of-line mode for cffi would speed up import times and reduce the memory footprint.

C:\>c:\Python27_x64\python.exe -OO -m Cryptodome.SelfTest
This leads to an error on load.
c:\Python27_x64\lib\site-packages\pycparser\c_parser.py:20: RuntimeWarning: parsing methods must have __doc__ for pycparser to work properly class CParser(PLYParser): WARNING: There was a problem loading the table file: KeyError('p_direct_id_declarator_1',)

[Legrandin]

I have never been so keen on using 'out-of-line', which introduces an unnecessary level of wrappers to be honest.

On the other hand, it should also be possible to check sys.flags.optimize in Util._raw_api and fallback to ctypes in case the optimization is 2. This is certainly an ugly workaround with some possible performance impact, but also simple, to be kept until either pycparser or cffi fix this problem.

[Legrandin]

Fixed with be4088c.

[github-user-en]

Hello,

I am facing this issue on Python 3.4.

Traceback (most recent call last):
File "C:\Python34\lib\site-packages\Crypto\Util_raw_api.py", line 83, in
raise ImportError("CFFI with optimize=2 fails due to pycparser bug.")
ImportError: CFFI with optimize=2 fails due to pycparser bug.

I am using pyCryptodome with Youtube-DL. I've built a local EXE of Youtube-DL using py2exe. So Py2exe should be putting PyCryptodome into the single EXE. When I run the EXE with any command, I get this error. For example,

C:\Users\Beyonce\AppData\Roaming\youtube-dl>youtube-dl.exe https://www.youtube.com/watch?v=qqWSMAoQOVg
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\Crypto\Util_raw_api.py", line 83, in
raise ImportError("CFFI with optimize=2 fails due to pycparser bug.")
ImportError: CFFI with optimize=2 fails due to pycparser bug.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 16, in
File "", line 2214, in _find_and_load
File "", line 2203, in _find_and_load_unlocked
File "", line 1191, in _load_unlocked
File "", line 1161, in load_backward_compatible
File "C:\Python34\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "C:\Users\Beyonce\youtube-dl\youtube_dl_init.py", line 15, in
from .options import (
File "", line 2214, in _find_and_load
File "", line 2203, in _find_and_load_unlocked
File "", line 1191, in _load_unlocked
File "", line 1161, in _load_backward_compatible
File "C:\Python34\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "C:\Users\Beyonce\youtube-dl\youtube_dl\options.py", line 8, in
from .downloader.external import list_external_downloaders
File "", line 2214, in _find_and_load
File "", line 2203, in _find_and_load_unlocked
File "", line 1191, in _load_unlocked
File "", line 1161, in load_backward_compatible
File "C:\Python34\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "C:\Users\Beyonce\youtube-dl\youtube_dl\downloader_init.py", line 5, in
from .hls import HlsFD
File "", line 2214, in _find_and_load
File "", line 2203, in _find_and_load_unlocked
File "", line 1191, in _load_unlocked
File "", line 1161, in _load_backward_compatible
File "C:\Python34\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "C:\Users\Beyonce\youtube-dl\youtube_dl\downloader\hls.py", line 6, in
from Crypto.Cipher import AES
File "", line 2214, in _find_and_load
File "", line 2203, in _find_and_load_unlocked
File "", line 1191, in _load_unlocked
File "", line 1161, in load_backward_compatible
File "C:\Python34\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "C:\Python34\lib\site-packages\Crypto\Cipher_init.py", line 27, in
from Crypto.Cipher._mode_ecb import _create_ecb_cipher
File "", line 2214, in _find_and_load
File "", line 2203, in _find_and_load_unlocked
File "", line 1191, in _load_unlocked
File "", line 1161, in _load_backward_compatible
File "C:\Python34\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "C:\Python34\lib\site-packages\Crypto\Cipher_mode_ecb.py", line 29, in
from Crypto.Util._raw_api import (load_pycryptodome_raw_lib,
File "", line 2214, in _find_and_load
File "", line 2203, in _find_and_load_unlocked
File "", line 1191, in _load_unlocked
File "", line 1161, in load_backward_compatible
File "C:\Python34\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "C:\Python34\lib\site-packages\Crypto\Util_raw_api.py", line 200, in
PyObject_GetBuffer = ctypes.pythonapi.PyObject_GetBuffer
File "C:\Python34\lib\ctypes_init.py", line 364, in getattr
func = self.getitem(name)
File "C:\Python34\lib\ctypes_init.py", line 369, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: function 'PyObject_GetBuffer' not found
'rct' is not recognized as an internal or external command,
operable program or batch file.
'q' is not recognized as an internal or external command,
operable program or batch file.
'esrc' is not recognized as an internal or external command,
operable program or batch file.
'source' is not recognized as an internal or external command,
operable program or batch file.
The system cannot find the path specified.
'cad' is not recognized as an internal or external command,
operable program or batch file.
'uact' is not recognized as an internal or external command,
operable program or batch file.
'ved' is not recognized as an internal or external command,
operable program or batch file.
'url' is not recognized as an internal or external command,
operable program or batch file.
'usg' is not recognized as an internal or external command,
operable program or batch file.