-
-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Use only aomenc and dav1d AVIF codecs to decrease wheel size #8858
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 20 commits
8b4e66e
a2cbce7
84485e1
ab55e54
8a6e5af
deb1c9d
6d7f396
7aca46c
8bc14e2
621bcd5
1c40ea5
331354a
8cac43f
ce01f19
8ab93db
a005e28
6c918a0
5573242
8a174f5
f40eed2
8905480
f20ce67
5a8313c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -51,6 +51,7 @@ LIBWEBP_VERSION=1.5.0 | |
BZIP2_VERSION=1.0.8 | ||
LIBXCB_VERSION=1.17.0 | ||
BROTLI_VERSION=1.1.0 | ||
LIBAVIF_VERSION=1.2.1 | ||
|
||
function build_pkg_config { | ||
if [ -e pkg-config-stamp ]; then return; fi | ||
|
@@ -98,6 +99,62 @@ function build_harfbuzz { | |
touch harfbuzz-stamp | ||
} | ||
|
||
function build_libavif { | ||
if [ -e libavif-stamp ]; then return; fi | ||
|
||
python3 -m pip install meson ninja | ||
|
||
if [[ "$PLAT" == "x86_64" ]] || [ -n "$SANITIZER" ]; then | ||
build_simple nasm 2.16.03 https://www.nasm.us/pub/nasm/releasebuilds/2.16.03 | ||
fi | ||
|
||
local build_type=MinSizeRel | ||
local lto=ON | ||
|
||
local libavif_cmake_flags=() | ||
|
||
if [ -n "$IS_MACOS" ]; then | ||
lto=OFF | ||
libavif_cmake_flags+=( | ||
-DCMAKE_C_FLAGS_MINSIZEREL="-Oz -DNDEBUG -flto " \ | ||
-DCMAKE_CXX_FLAGS_MINSIZEREL="-Oz -DNDEBUG -flto" \ | ||
-DCMAKE_SHARED_LINKER_FLAGS_INIT="-Wl,-S,-x,-dead_strip_dylibs" \ | ||
) | ||
else | ||
if [[ "$MB_ML_VER" == 2014 ]] && [[ "$PLAT" == "x86_64" ]]; then | ||
build_type=Release | ||
fi | ||
libavif_cmake_flags+=(-DCMAKE_SHARED_LINKER_FLAGS_INIT="-Wl,--strip-all,-z,relro,-z,now") | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If I compare the wheel sizes from https://github.com/python-pillow/Pillow/actions/runs/14773502497?pr=8858, If this line is purely about reducing file size, then I don't think it's fulfilling that role. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah, it's because there is a typo. I'll fix it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It actually had to do with a cmake policy that was disabling LTO on linux. See the build here: https://github.com/fdintino/Pillow/actions/runs/14777836226 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In the PR description, I've updated the paragraph that begins with "In absolute terms,..." and the last column of the table with the latest wheel and shared library file sizes. You can look at the github diff under the list of edits to see the change. |
||
fi | ||
|
||
local out_dir=$(fetch_unpack https://github.com/AOMediaCodec/libavif/archive/refs/tags/v$LIBAVIF_VERSION.tar.gz libavif-$LIBAVIF_VERSION.tar.gz) | ||
# CONFIG_AV1_DECODER=0 is a flag for libaom (included as a subproject of | ||
# libavif) to disable the compilation and inclusion of aom's AV1 decoder. | ||
fdintino marked this conversation as resolved.
Show resolved
Hide resolved
|
||
# CONFIG_AV1_HIGHBITDEPTH=0 is another flag for libaom that disables support | ||
# for encoding high bit depth images. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should not disbale support for encoding high bit depth images. Most AVIF HDR images are 10 bits. Can we avoid this? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Pillow does not yet support high bit depth images (see #1888). It is currently a work-in-progress, but at the moment there can only ever be 8-bit images passed to the C API. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for the reply. Nit: Change "another flag" to "a flag" because we are removing the previous paragraph (lines 131-132). We can look into disabling support for decoding high bit depth images in the dav1d library. It has this build option:
But it seems hard to pass this build option to dav1d through libavif's cmake command line. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we might still want to allow decoding of 10-bit images, even if that necessarily means downsampling to 8 bit to load them into Pillow. |
||
(cd $out_dir \ | ||
&& CMAKE_POLICY_VERSION_MINIMUM=3.5 cmake \ | ||
-DCMAKE_INSTALL_PREFIX=$BUILD_PREFIX \ | ||
-DCMAKE_INSTALL_LIBDIR=$BUILD_PREFIX/lib \ | ||
-DCMAKE_INSTALL_NAME_DIR=$BUILD_PREFIX/lib \ | ||
-DBUILD_SHARED_LIBS=ON \ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Did you mean to change There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I did. Using the linker to generate the shared library file allows it to remove unused objects. There is a mechanism for including shared library dependencies of a python extension inside a wheel file on macOS and linux (delocate and auditwheel, respectively) but there is no such standard mechanism for doing the same on windows. So the windows library has to be built statically. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The savings are modest, about 0.5 MB, but there isn't any cost to switching to a shared library. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When you say 0.5mb, I presume that's per wheel. Which wheel(s) were you using to check that? |
||
-DAVIF_LIBSHARPYUV=LOCAL \ | ||
-DAVIF_LIBYUV=LOCAL \ | ||
-DAVIF_CODEC_AOM=LOCAL \ | ||
-DCONFIG_AV1_DECODER=0 \ | ||
fdintino marked this conversation as resolved.
Show resolved
Hide resolved
|
||
-DCONFIG_AV1_HIGHBITDEPTH=0 \ | ||
-DAVIF_CODEC_AOM_DECODE=OFF \ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Frankie: Is it possible to build with the libaom and dav1d packages in Linux distributions (i.e., There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The shared libraries linked from the pillow extension are all included in the distributed binary, so that people installing Pillow don't need to install the libraries themselves. This is why libjpeg-turbo, libpng, libtiff, and other libraries are included in the wheel files on PyPI. Nothing would be gained by using libaom from the linux distribution. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If someone were to build Pillow from source, they could use libavif from the distro, or build libavif against system libraries. The binary wheels are tagged for python version, system architecture, and ABI compatibility (musl and two different versions of glibc). They are intended to be installed in any compatible environment without any other dependencies. |
||
-DAVIF_CODEC_DAV1D=LOCAL \ | ||
-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=$lto \ | ||
-DCMAKE_C_VISIBILITY_PRESET=hidden \ | ||
-DCMAKE_CXX_VISIBILITY_PRESET=hidden \ | ||
-DCMAKE_BUILD_TYPE=$build_type \ | ||
"${libavif_cmake_flags[@]}" \ | ||
. \ | ||
&& make install) | ||
touch libavif-stamp | ||
} | ||
|
||
function build { | ||
build_xz | ||
if [ -z "$IS_ALPINE" ] && [ -z "$SANITIZER" ] && [ -z "$IS_MACOS" ]; then | ||
|
@@ -132,6 +189,7 @@ function build { | |
build_tiff | ||
fi | ||
|
||
build_libavif | ||
build_libpng | ||
build_lcms2 | ||
build_openjpeg | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
Copyright (c) 2016, Alliance for Open Media. All rights reserved. | ||
|
||
Redistribution and use in source and binary forms, with or without | ||
modification, are permitted provided that the following conditions | ||
are met: | ||
|
||
1. Redistributions of source code must retain the above copyright | ||
notice, this list of conditions and the following disclaimer. | ||
|
||
2. Redistributions in binary form must reproduce the above copyright | ||
notice, this list of conditions and the following disclaimer in | ||
the documentation and/or other materials provided with the | ||
distribution. | ||
|
||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS | ||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT | ||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS | ||
FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE | ||
COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, | ||
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, | ||
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; | ||
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER | ||
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT | ||
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN | ||
ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE | ||
POSSIBILITY OF SUCH DAMAGE. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
Copyright © 2018-2019, VideoLAN and dav1d authors | ||
All rights reserved. | ||
|
||
Redistribution and use in source and binary forms, with or without | ||
modification, are permitted provided that the following conditions are met: | ||
|
||
1. Redistributions of source code must retain the above copyright notice, this | ||
list of conditions and the following disclaimer. | ||
|
||
2. Redistributions in binary form must reproduce the above copyright notice, | ||
this list of conditions and the following disclaimer in the documentation | ||
and/or other materials provided with the distribution. | ||
|
||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND | ||
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED | ||
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE | ||
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR | ||
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES | ||
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; | ||
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND | ||
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | ||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS | ||
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
libavif 1.3.0 has been released - https://github.com/AOMediaCodec/libavif/releases/tag/v1.3.0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the suggested change. I suggest two related changes if you will only be building with libavif v1.3.0 or later.
CONFIG_AV1_DECODER=0
in your build files (including the comments). libavif v1.3.0 or later setsCONFIG_AV1_DECODER
to 0 if you pass-DAVIF_CODEC_AOM_DECODE=OFF
to libavif."-DCMAKE_POLICY_VERSION_MINIMUM=3.5",
. We fixed all the oldcmake_minimum_required()
versions in libavif v1.3.0. You should not need this workaround now.