When loading the python core dll using LoadLibraryA/W the encodings module cannot load. #126905
Comments
|
What's more likely is that your directory structure doesn't match a default CPython layout and you aren't initializing correctly. The encodings module is usually the first one that relies on having correct search paths set. Check out https://docs.python.org/3/c-api/init_config.html#init-config and consider initializing with explicit module search paths. |
static void my_Py_Initialize() {
if (Py_IsInitialized() != 0) {
/* bpo-33932: Calling Py_Initialize() twice does nothing. */
return;
}
PyStatus status;
PyConfig config;
PyFile_SetOpenCodeHook((Py_OpenCodeHookFunction)OpenCodeHook, NULL);
PyConfig_InitIsolatedConfig(&config);
wchar_t path[MAX_PATH] = { L'\0' };
GetCurrentDirectoryW(MAX_PATH, path);
status = PyWideStringList_Append(&config.module_search_paths, path);
memset(path, 0, MAX_PATH);
GetModuleFileNameW(GetModuleHandle(NULL), path, MAX_PATH);
status = PyWideStringList_Append(&config.module_search_paths, path);
wcscat(path, L"/site-packages");
status = PyWideStringList_Append(&config.module_search_paths, path);
config.install_signal_handlers = 1;
// config.site_import = 0;
config.module_search_paths_set = 1;
if (PyStatus_Exception(status)) {
PyConfig_Clear(&config);
Py_ExitStatusException(status);
}
status = Py_InitializeFromConfig(&config);
PyConfig_Clear(&config);
if (PyStatus_Exception(status)) {
Py_ExitStatusException(status);
}
}I call this in my code with my own |
|
What does your I'm pretty sure we only handle |
static PyObject *__cdecl OpenCodeHook(PyObject *path, void *userData) {
const char *_path = PyUnicode_AsUTF8(path);
if (strstr(_path, ".exe") != NULL) {
DWORD size;
LPVOID resource = InternalLoadResource(GetModuleHandle(NULL), MAKEINTRESOURCEA(LIBS_ZIP), &size);
PyObject *resource_bytes = PyBytes_FromStringAndSize((const char*)resource, size);
if (resource_bytes == NULL) {
printf("%s\n", "Failed to convert the Win32 Resource Data to a 'bytes' object.");
return NULL; // Error occurred
}
PyObject *BytesIO = _PyImport_GetModuleAttrString("_io", "BytesIO");
if (BytesIO == NULL) {
printf("%s\n", "Failed to get the 'BytesIO' class from module '_io'.");
Py_DECREF(resource_bytes);
return NULL; // Error occurred
}
// Create a BytesIO instance using the bytes object
PyObject *args = PyTuple_New(1);
Py_INCREF(resource_bytes);
if (args == NULL || PyTuple_SetItem(args, 0, resource_bytes) < 0) {
printf("%s\n", "Failed to create the args tuple for creating an 'BytesIO' instance.");
Py_DECREF(BytesIO);
Py_DECREF(args);
Py_DECREF(resource_bytes);
return NULL; // Error occurred
}
// PyObject *args = PyTuple_Pack(1, resource_bytes);
// if (args == NULL) {
// printf("%s\n", "Failed to create the args tuple for creating an 'BytesIO' instance.");
// Py_DECREF(BytesIO);
// return NULL; // Error occurred
// }
PyObject *bytesio_instance = PyObject_CallObject(BytesIO, args);
Py_DECREF(BytesIO);
Py_DECREF(args);
Py_DECREF(resource_bytes);
if (bytesio_instance == NULL) {
printf("%s\n", "Failed to create the 'BytesIO' instance.");
return NULL; // Error occurred
}
return bytesio_instance; // This is the BytesIO instance
}
else if (PyOS_stricmp(_path, "") != 0) { // ignore empty string.
printf("%s\n", _path);
// FILE *file = fopen(_path, "r");
// fread
}
Py_RETURN_NONE;
}I probably should finish implementing the |
|
My suspicion is that the default importer is doing some other checks first and deciding that the file does not exist. It only has to try an For what you're trying to do, you really are going to need to create an importer. You can look at how I did my DLL packer if you're interested (it's a bit convoluted, but I'm sure you can find your way around), or else the importlib docs and specifically the importlib.abc section should give you the info you need to implement this. |
Strange, it should have looked inside of the zip file that I return as a |
|
So, I tested it with the normal The result is sadly the same |
|
Turns out I managed to get more information my appending this to my debug console command args when running my application in debug (on Release config): Output in that file: For some reason the thing is assuming that the magic number that is freshly compiled when I run the steps to make an embed output zip file and then extracting the Edit: The static void my_Py_Initialize() {
if (Py_IsInitialized() != 0) {
/* bpo-33932: Calling Py_Initialize() twice does nothing. */
return;
}
PyStatus status;
PyConfig config;
PyFile_SetOpenCodeHook((Py_OpenCodeHookFunction)OpenCodeHook, NULL);
PyConfig_InitIsolatedConfig(&config);
config.install_signal_handlers = 1;
config.site_import = 0;
config.user_site_directory = 0;
config.module_search_paths_set = 1;
config.write_bytecode = 0;
printf("%s%i%s\n", "config.site_import is value: '", config.site_import, "'");
printf("%s%i%s\n", "config.user_site_directory is value: '", config.user_site_directory, "'");
PyModuleSearchPath_Append(status, config, applicationFileName)
PyModuleSearchPath_Append(status, config, applicationSitePackages)
PyModuleSearchPath_Append(status, config, applicationFileDirectory)
status = Py_InitializeFromConfig(&config);
printf("%s%i%s\n", "config.site_import is value: '", config.site_import, "'");
printf("%s%i%s\n", "config.user_site_directory is value: '", config.user_site_directory, "'");
PyConfig_Clear(&config);
if (PyStatus_Exception(status)) {
Py_ExitStatusException(status);
}
}With this however, despite setting BOTH |
At a guess, you probably updated the CPython repo without regenerating your ZIP file, and someone changed the bytecode version in the meantime. That's very likely to happen when you're pulling from
If you're going to insist on building from If you try instead with the last release of 3.13, we'll have a better idea of what it's supposed to be doing. Or even the last alpha of 3.14 where we know it passed all the tests. A random commit from |
|
I figured out the TLS problem when trying to Turns out the DllMain function in the Python Core DLL does not initialize the TLS slots (Thread Local Storage) directly at all and so it magically works only when the dll is present inside of the exe's IAT (IMPORT_ADDRESS_TABLE). https://learn.microsoft.com/en-us/windows/win32/dlls/dllmain The best option to fix this is to patch the DllMain to always initialize the proper TLS slots (using the TSS API's) for the python core to use in the spots that msdocs page documents. |
I wasn't aware there was a TLS problem? We allocate TLS slots when requested through our So your proposed fix would be a significant refactor to significant parts of Python's deepest infrastructure impacting all platforms. Before we go looking into that, perhaps you could describe the problem you were seeing? |
Basically, in my exe I have it first check if the python core dll exists in it's folder at runtime, following a version check prior to loading it and then if one or both of those checks fail I write the dll file from it's win32 resource section to that folder before calling The issue with that is that since Python 3.12 where Normally people have the Python Core DLL it their IAT (Import Address Table) unless they want to package their embed application as a single file with the python core, it's C extensions, and the zip of the stlib all inside of the win32 resource section of their exe and then later have them written to disk at runtime. During the normal usage (without The main problem: The problem starts when needing to use Even py2exe suffers from this same problem and it implements it's own |
The DLLMain provided by MSVC will always initialise |
I am loading it with |
|
Perhaps you can describe what isn't working for you? All the documentation and source code suggests that |
|
So, I think I have an option, however it would require me to modify
Note: if the static configurations is approved, there will never be any shipping of prebuilt lib files of such configuration as I feel it is only for advanced embedding, as such they can simply add the cpython repository as an This works great, unless I just so happen to depend on packages such as
As long as I can static link in every pyd file that is possible for me to load along with the python core itself, my code should work without problems as a single file exe. |
|
We'd love to have static lib build configurations, though as you've recognised it's a significant project. If you want to take it on, our requirements would be that it has to work from a command-line build (i.e. |
Bug report
Bug description:
I am trying to embed python inside of an exe where I don't want it inside of the exe's
IMPORT_ADDRESS_TABLEand expected that when loading the python core with LoadLibraryA/W from the Windows API to work out of box just like it does when it is in the IMPORT_ADDRESS_TABLE.I suspect the problem might be because the IMPORT_ADDRESS_TABLE is eventually calling the undocumented and non-exported
LdrpHandleTlsDatafunction and that theLoadLibraryA/WAPI's might not be calling it but I am not entirely sure.I even went as far as implementing my own copy of LoadLibraryA/W that does call this function in a creative way and it still does not work and I am not sure why. I expected for the normal LoadLibraryA/W functions to work out of box though without manually implementing the logic behind those functions to test if this was the case.
Why do I want to manually load the python core dll instead of letting the IMPORT_ADDRESS_TABLE and Windows to load it for me?
This is because I am embedding python and I am doing it with the python core itself inside of it's win32 resource section and then writing it out to disk before attempting to LoadLibraryA/W it. This is so I can then distribute a single standalone exe file that extracts the python core, uses the zip file with the standard library, c extensions (memory loaded), and any site-packages inside of the zip file from within the win32 resource section as well for properly loading up the python interpreter.
All of the logic behind loading the zip file with all of it's contents within the embed exe's win32 resource section works flawlessly when the core dll is in the IMPORT_ADDRESS_TABLE though. Sadly I doubt manually loading the python core and then injecting the python core dll inside of the currently running process's IMPORT_ADDRESS_TABLE (to make it use the existing handle from manually loading said dll) via code would work though as I suspect the code behind the IAT parts of the exe in memory is made readonly by the OS.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Windows
The text was updated successfully, but these errors were encountered: