bpo-44394: Update libexpat copy to 2.4.1

[vstinner]

Update the vendored copy of libexpat to 2.4.1 (from 2.2.8) to get the
fix for the CVE-2013-0340 "Billion Laughs" vulnerability. This copy
is most used on Windows and macOS.

https://bugs.python.org/issue44394

[vstinner]

cc @tiran: please have a look at the XML vulnerability documentation change. I'm not sure that pyexpat is used by all Python XML parsers.

[vstinner]

I used cpython_rebuild_expat_dir.sh script attached to https://bugs.python.org/issue44394 to created this PR, then I manually reverted the following change:

diff --git a/Modules/expat/expat_external.h b/Modules/expat/expat_external.h
index f2b75dda8e..8829f77091 100644
--- a/Modules/expat/expat_external.h
+++ b/Modules/expat/expat_external.h
-
-/* Namespace external symbols to allow multiple libexpat version to
-   co-exist. */
-#include "pyexpatns.h"
-

I tested this PR with the command: ./configure --cache-file=../python-config.cache --with-pydebug CFLAGS=-O0 --with-system-ffi && make clean && make && ./python -m test -v test_pyexpat.

test_pyexpat pass successfully.

Manual test to ensure that the Python pyexpat module is not linked to the system libexpat:

$ ldd $(./python -c 'import pyexpat; print(pyexpat.__file__)')
	linux-vdso.so.1 (0x00007ffd90b6b000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa78c10c000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fa78bf3d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fa78c193000)

libexpat is not listed in the library dependencies, so it's ok.

[tiran]

I'm against marking it safe until Python no longer supports libexpat <= 2.4.0.

[hartwork]

That's a fair point. Any ideas how to best communicate it in this table?

[vstinner]

Vulnerable or Safe depends on the libexpat version, that's what I wrote in the footnote (1). I explain how how to check manually if your Python is vulnerable or not.

@tiran How do you want to explain that it depends on the libexpat version in this table, if you are unhappy with "Safe (1)"?

[tiran]

Do **Vulnerable** (1) until all relevant Linux distros have fixed libexpat: all supported CentOS streams, Debian stables, RHELs, Ubuntu LTS, etc.

[hartwork]

@vstinner the two new API functions and the new error codes do not seem included, yet. Did you see https://bugs.python.org/msg395642 ?

[hartwork]

Manual test to ensure that the Python pyexpat module is not linked to the system libexpat:

$ ldd $(./python -c 'import pyexpat; print(pyexpat.__file__)')
	linux-vdso.so.1 (0x00007ffd90b6b000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa78c10c000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fa78bf3d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fa78c193000)

libexpat is not listed in the library dependencies, so it's ok.

@vstinner I'm not entirely sure what the idea is with that test but libexpat is listed for me on Linux:

# ldd $(python -c 'import pyexpat; print(pyexpat.__file__)')
        linux-vdso.so.1 (0x00007ffca4502000)
        libexpat.so.1 => /usr/lib64/libexpat.so.1 (0x00007ffa92b10000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffa92af1000)
        libc.so.6 => /lib64/libc.so.6 (0x00007ffa92938000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ffa92b73000)

[vstinner]

@vstinner the two new API functions and the new error codes do not seem included, yet. Did you see https://bugs.python.org/msg395642 ?

That's a new feature, it cannot be backported to older Python versions. I'm not interested to write a PR to implement it.

This PR is restricted to updated libexpat so it can be backported to all Python versions which still accept security fixes.

[hartwork]

@vstinner the two new API functions and the new error codes do not seem included, yet. Did you see https://bugs.python.org/msg395642 ?

That's a new feature, it cannot be backported to older Python versions. I'm not interested to write a PR to implement it.

This PR is restricted to updated libexpat so it can be backported to all Python versions which still accept security fixes.

The new API and error codes are part of the security fix.

[github-actions]

This PR is stale because it has been open for 30 days with no activity.

[miss-islington]

Thanks @vstinner for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.6, 3.7, 3.8, 3.9, 3.10.
🐍🍒⛏🤖

[bedevere-bot]

GH-28031 is a backport of this pull request to the 3.10 branch.

[bedevere-bot]

GH-28032 is a backport of this pull request to the 3.9 branch.

[bedevere-bot]

GH-28033 is a backport of this pull request to the 3.8 branch.

[miss-islington]

Sorry, @vstinner and @ambv, I could not cleanly backport this to 3.7 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 3fc5d84046ddbd66abac5b598956ea34605a4e5d 3.7

[miss-islington]

Sorry @vstinner and @ambv, I had trouble checking out the 3.6 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 3fc5d84046ddbd66abac5b598956ea34605a4e5d 3.6

[ambv]

@ned-deily, this is marked as needing backport to 3.6 and 3.7 as well. Since there's conflicts, please let me know if I should work on that.

[bedevere-bot]

GH-28042 is a backport of this pull request to the 3.7 branch.

[vstinner]

Thanks for the update @ambv! I failed to find time to update this PR this summer ;-)