Reduce AVIF wheel size?

[hugovk]

The AVIF .so has made the wheels about 4 times bigger: #8722 (comment)

Is there anything we can do to make it smaller?

[kmilos]

Is there anything we can do to make it smaller?

Maybe there is no need to carry multiple encoders and decoders? E.g. libaom covers both ends (and is libavif preferred for encoding), no need to build w/ and ship dav1d and rav1e+svt-av1 as well? That being said, libaom seems to be the largest of them all (at least on my machine/platform), and dav1d seems to be preferred for decoding speed...

[liRONCO11]

Package size changed from 5.1M unpacked to 54M PIL, is that mandatory?

[cgohlke]

The AVIF .so has made the wheels about 4 times bigger:

See also #5201 (comment)

[doublex]

The ability to test/switch between codecs is a handy feature.

[OscarVanL]

This made me sad too because my PIL venv install went from 6.4MB to 59MB which made my container image much bigger :(

[fdintino]

If I had to choose a codec for removal, it would probably be rav1e. Development on rav1e has been a bit stagnant, and in my opinion AOM has come out ahead of it with the 3.12.0 release. Rav1e is also quite large. I'm running a build now without it to figure out what its contribution to the wheel size is.

[wantehchang]

I recommend including libaom for the encoder and dav1d for the decoder.

@fdintino Frankie: To build an encoder-only version of libaom, pass -DCONFIG_AV1_DECODER=0 to libaom's cmake configure command. Then pass -DAVIF_CODEC_AOM_DECODE=OFF to libavif's cmake configure command.

[radarhere]

Experimenting with pillow-11.2.0.dev0-cp313-cp313-musllinux_1_2_x86_64.whl, I find

• Currently, 19.8mb
• Without dav1d, 18.8mb (AOM, RAV1E and SVT encoders, AOM decoder)
• Without svt, 16.5mb (AOM and RAV1E encoders, AOM and DAV1D decoders)
• Without rav1e, 12.8mb (AOM and SVT encoders, AOM and DAV1D decoders)
• Without dav1d, rav1e and svt, 8.5mb (AOM encoder and decoder)

[hugovk]

(For comparison, pillow-11.1.0-cp313-cp313-musllinux_1_2_x86_64.whl was 4.4 MB in the last release.)

[fdintino]

I've updated #8858 to only include the dav1d decoder and the aom encoder in the wheel. It roughly halves the size of the 11.2.0 wheels.

[hugovk]

Thanks all for working on this!

Here's a comparison of file size in MB between #8858 (only aomenc and dav1d codecs) and #8869 (only aom codec), comparing each size to that of 11.1.0 from January.

filename	11.1.0	11.2.0	bigger	aomenc & dav1d	bigger	aom only	bigger
pillow.tar.gz	46.7	47.0	x1.0	47.0	x1.0	47.0	x1.0
cp310-cp310-macosx_10_10_x86_64	3.2	11.4	x3.5	6.3	x2.0	6.4	x2.0
cp310-cp310-macosx_11_0_arm64	3.1	8.9	x2.9	5.2	x1.7	5.7	x1.8
cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64	4.3	17.1	x4.0	8.1	x1.9	7.9	x1.8
cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64	4.4	18.9	x4.3	8.6	x2.0	8.1	x1.8
cp310-cp310-manylinux_2_28_aarch64	4.4	17.6	x4.0	6.8	x1.6	8.3	x1.9
cp310-cp310-manylinux_2_28_x86_64	4.5	19.6	x4.4	7.5	x1.7	8.4	x1.9
cp310-cp310-musllinux_1_2_aarch64	4.3	18.0	x4.2	7.0	x1.6	8.5	x2.0
cp310-cp310-musllinux_1_2_x86_64	4.4	19.8	x4.5	7.6	x1.7	8.5	x1.9
cp310-cp310-win32	2.3	2.3	x1.0	2.3	x1.0	2.3	x1.0
cp310-cp310-win_amd64	2.6	13.8	x5.3	7.2	x2.8	7.1	x2.7
cp310-cp310-win_arm64	2.4	2.4	x1.0	2.4	x1.0	2.4	x1.0
cp311-cp311-macosx_10_10_x86_64	3.2	11.4	x3.5	6.3	x2.0	6.4	x2.0
cp311-cp311-macosx_11_0_arm64	3.1	8.9	x2.9	5.2	x1.7	5.7	x1.8
cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64	4.3	17.1	x4.0	8.1	x1.9	7.9	x1.8
cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64	4.4	18.9	x4.3	8.6	x2.0	8.1	x1.8
cp311-cp311-manylinux_2_28_aarch64	4.4	17.6	x4.0	6.8	x1.6	8.3	x1.9
cp311-cp311-manylinux_2_28_x86_64	4.5	19.6	x4.4	7.5	x1.7	8.4	x1.9
cp311-cp311-musllinux_1_2_aarch64	4.3	18.0	x4.2	7.0	x1.6	8.5	x2.0
cp311-cp311-musllinux_1_2_x86_64	4.4	19.8	x4.5	7.6	x1.7	8.5	x1.9
cp311-cp311-win32	2.3	2.3	x1.0	2.3	x1.0	2.3	x1.0
cp311-cp311-win_amd64	2.6	13.8	x5.3	7.2	x2.8	7.1	x2.7
cp311-cp311-win_arm64	2.4	2.4	x1.0	2.4	x1.0	2.4	x1.0
cp312-cp312-macosx_10_13_x86_64	3.2	11.4	x3.5	6.3	x1.9	6.5	x2.0
cp312-cp312-macosx_11_0_arm64	3.1	8.9	x2.9	5.2	x1.7	5.7	x1.8
cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64	4.3	17.1	x3.9	8.1	x1.9	7.9	x1.8
cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64	4.4	18.9	x4.3	8.6	x2.0	8.1	x1.8
cp312-cp312-manylinux_2_28_aarch64	4.4	17.6	x4.0	6.8	x1.6	8.3	x1.9
cp312-cp312-manylinux_2_28_x86_64	4.5	19.6	x4.4	7.5	x1.7	8.4	x1.9
cp312-cp312-musllinux_1_2_aarch64	4.3	18.0	x4.2	7.0	x1.6	8.5	x2.0
cp312-cp312-musllinux_1_2_x86_64	4.4	19.8	x4.5	7.6	x1.7	8.5	x1.9
cp312-cp312-win32	2.3	2.3	x1.0	2.3	x1.0	2.3	x1.0
cp312-cp312-win_amd64	2.6	13.8	x5.3	7.2	x2.8	7.1	x2.7
cp312-cp312-win_arm64	2.4	2.4	x1.0	2.4	x1.0	2.4	x1.0
cp313-cp313-macosx_10_13_x86_64	3.2	11.4	x3.5	6.3	x1.9	6.5	x2.0
cp313-cp313-macosx_11_0_arm64	3.1	8.9	x2.9	5.2	x1.7	5.7	x1.8
cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64	4.3	17.1	x3.9	8.1	x1.9	7.9	x1.8
cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64	4.4	18.9	x4.3	8.6	x2.0	8.1	x1.8
cp313-cp313-manylinux_2_28_aarch64	4.4	17.6	x4.0	6.8	x1.6	8.3	x1.9
cp313-cp313-manylinux_2_28_x86_64	4.5	19.6	x4.4	7.5	x1.7	8.4	x1.9
cp313-cp313-musllinux_1_2_aarch64	4.3	18.0	x4.2	7.0	x1.6	8.5	x2.0
cp313-cp313-musllinux_1_2_x86_64	4.4	19.8	x4.5	7.6	x1.7	8.5	x1.9
cp313-cp313-win32	2.3	2.3	x1.0	2.3	x1.0	2.3	x1.0
cp313-cp313-win_amd64	2.6	13.8	x5.3	7.2	x2.8	7.1	x2.7
cp313-cp313-win_arm64	2.4	2.4	x1.0	2.4	x1.0	2.4	x1.0
cp313-cp313t-macosx_10_13_x86_64	3.2	11.4	x3.5	6.3	x1.9	6.5	x2.0
cp313-cp313t-macosx_11_0_arm64	3.1	8.9	x2.9	5.2	x1.7	5.7	x1.8
cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64		17.1		8.1		8.0
cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64	4.4	19.0	x4.3	8.7	x2.0	8.1	x1.8
cp313-cp313t-manylinux_2_28_aarch64		17.7		6.8		8.3
cp313-cp313t-manylinux_2_28_x86_64	4.5	19.7	x4.3	7.5	x1.7	8.5	x1.9
cp313-cp313t-musllinux_1_2_aarch64		18.1		7.0		8.6
cp313-cp313t-musllinux_1_2_x86_64	4.5	19.8	x4.4	7.6	x1.7	8.5	x1.9
cp313-cp313t-win32	2.3	2.3	x1.0	2.3	x1.0	2.3	x1.0
cp313-cp313t-win_amd64	2.6	13.8	x5.2	7.2	x2.8	7.1	x2.7
cp313-cp313t-win_arm64	2.4	2.4	x1.0	2.4	x1.0	2.4	x1.0
cp39-cp39-macosx_10_10_x86_64	3.2	11.4	x3.5	6.3	x2.0	6.4	x2.0
cp39-cp39-macosx_11_0_arm64	3.1	8.9	x2.9	5.2	x1.7	5.7	x1.8
cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64	4.3	17.1	x4.0	8.1	x1.9	7.9	x1.8
cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64	4.4	18.9	x4.3	8.6	x2.0	8.1	x1.8
cp39-cp39-manylinux_2_28_aarch64	4.4	17.6	x4.0	6.8	x1.6	8.3	x1.9
cp39-cp39-manylinux_2_28_x86_64	4.5	19.6	x4.4	7.5	x1.7	8.4	x1.9
cp39-cp39-musllinux_1_2_aarch64	4.3	18.0	x4.2	6.9	x1.6	8.5	x2.0
cp39-cp39-musllinux_1_2_x86_64	4.4	19.8	x4.5	7.6	x1.7	8.5	x1.9
cp39-cp39-win32	2.3	2.3	x1.0	2.3	x1.0	2.3	x1.0
cp39-cp39-win_amd64	2.6	13.8	x5.3	7.2	x2.8	7.1	x2.7
cp39-cp39-win_arm64	2.4	2.4	x1.0	2.4	x1.0	2.4	x1.0
pp310-pypy310_pp73-macosx_10_15_x86_64	3.2	11.4	x3.6	6.3	x2.0	6.5	x2.0
pp310-pypy310_pp73-macosx_11_0_arm64	3.1	8.9	x2.9	5.2	x1.7	5.7	x1.8
pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64	3.4	16.1	x4.7	7.1	x2.1	6.9	x2.0
pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64	3.4	17.8	x5.2	7.6	x2.2	7.0	x2.0
pp310-pypy310_pp73-manylinux_2_28_aarch64	3.4	16.6	x4.8	5.8	x1.7	7.3	x2.1
pp310-pypy310_pp73-manylinux_2_28_x86_64	3.5	18.5	x5.3	6.5	x1.8	7.3	x2.1
pp310-pypy310_pp73-win_amd64	2.6	13.8	x5.3	7.2	x2.8	7.1	x2.7
pp311-pypy311_pp73-macosx_10_15_x86_64		11.4		6.3		6.5
pp311-pypy311_pp73-macosx_11_0_arm64		8.9		5.2		5.7
pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64		16.1		7.1		6.9
pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64		17.8		7.6		7.0
pp311-pypy311_pp73-manylinux_2_28_aarch64		16.6		5.8		7.3
pp311-pypy311_pp73-manylinux_2_28_x86_64		18.5		6.5		7.3
pp311-pypy311_pp73-win_amd64		13.8		7.2		7.1
total	249.3	1,055	x4.4	505.7	x2.0	538.1	x2.2
average	3.5	13.7		6.2		6.6

Note 11.1.0 had 71 files compared to 81 for 11.2+, so a small part of the increase is down to extra wheels.

The initial 11.2.0 was 4.4 times bigger than 11.1.0.

These new PRs are 2.0 times (aomenc and dav1d) and 2.2 times bigger (aom), much more reasonable.

In #8869, @fdintino said:

I think aom is a no-brainer for the encoder, but dav1d is basically the industry standard decoder. It is the decoder used by every major browser: Edge, Safari, Chrome, and Firefox.

And @wantehchang said:

We should only consider removing dav1d as a last resort. Although the decoder in libaom is the reference implementation and we test it with open-source and commercial AV1 test vectors and fuzzing regularly at Google, dav1d is likely to have better software security because it is being used in far more applications, including some that perform additional fuzzing of dav1d.

From this, #8858 (aomenc and dav1d codecs) sound like a better choice, plus the files are a bit smaller.

[PalmtopTiger]

Hi. Have you considered using the Limited API to drastically reduce the number of wheels? Also, you won't need to rebuild wheels for every Python release.
I've noticed that packages don't usually use the Limited API. Is there a reason for this?

[sspencer-arine]

This has burst the bubble on already constrained image sizes for serverless images - and a great many serverless functions use pillow.

I'd love to see support for this migrated to an extras and available through standard package tool discovery.

[aclark4life]

@PalmtopTiger @sspencer-arine Since we're late to deliver 11.2 in #8722 (in accordance with our own release schedule), those suggestions are more likely to be implemented in 11.3 (or 12?)

I'm not familiar with the Limited API … and maybe we could do extras … but right now I think we're just trying to get the release out the door with wheels sizes that everyone agrees are within some acceptable range.

[sspencer-arine]

@aclark4life thank you - but is there guidance on how to omit AVIF and any other things that came in since 11.1 if we did a local build and distributed that internally as a custom 11.2 wheel?

[9001]

is there guidance on how to omit AVIF and any other things that came in since 11.1 if we did a local build and distributed that internally as a custom 11.2 wheel?

@sspencer-arine you don't really need to build it from source -- it is sufficient to remove avif support from the official whl by deleting PIL/_avif.*

$ ls -al pillow-11.2.0-cp312-cp312-win_amd64.whl 
-rw-r--r--.  ed ed 13807352  pillow-11.2.0-cp312-cp312-win_amd64.whl

$ zip -d pillow-11.2.0-cp312-cp312-win_amd64.whl PIL/_avif.pyi PIL/_avif.cp312-win_amd64.pyd 
deleting: PIL/_avif.cp312-win_amd64.pyd
deleting: PIL/_avif.pyi

$ ls -al pillow-11.*-cp312-cp312-win_amd64.whl 
-rw-r--r--.  ed ed 2680119  pillow-11.2.0-cp312-cp312-win_amd64.whl
-rw-r--r--.  ed ed 2626369  pillow-11.1.0-cp312-cp312-win_amd64.whl

this is not official advice, I'm just another user :-)

[radarhere]

is there guidance on how to omit AVIF and any other things that came in since 11.1 if we did a local build and distributed that internally as a custom 11.2 wheel?

If you're building from source, then by default Pillow won't include libavif if it's not present in your environment. If it is present, or you don't know and would like to ensure that it's not included, you can disable it using a build option - -C avif=disable

[radarhere]

Those who were concerned about wheel sizes will be glad to know that Pillow 11.2.1 has now been released without libavif included in the wheels.

[petrprikryl]

Hey, are there any plans/ideas how to make it smaller and ship it without need of extra building on project side?

[aclark4life]

@petrprikryl I suspect we'll have more decisions made before the next release but ideas welcome!

[fdintino]

@petrprikryl #8858 has every trick I've been able to come up with to reduce the binary size. On most platforms the increased uncompressed size is between 3 and 5 MB. I doubt it can be made much smaller—after some optimizations I made yesterday I am out of ideas—so the decision will probably center around whether that is a size to allow its inclusion in the distributed wheels.

[hugovk]

Thank you for all your work on #8858, it's much appreciated!

With a quick calculation, the total size of all the wheels is up from ~260 MB to ~469 MB, about 1.8 times bigger. That's much better than the ~4 times we had originally and I think it's an acceptable increase.