Skip to content

Add support for animated WebP files #2761

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Nov 5, 2017

Conversation

jd20
Copy link
Contributor

@jd20 jd20 commented Sep 27, 2017

Fixes #2755 .

Changes proposed in this pull request:

  • Extend the C API to support WebPAnimEncoder / WebPAnimDecoder
  • Use the AnimEncoder / AnimDecoder API's, when mux support is compiled, except for when writing single-frame images (because internally the WebPAnimXXX code doesn't support RGB, only RGBA)
  • Also add support for XMP metadata chunks, and reading/writing metadata chunks to animated webp files
  • Added support for specifying some additional encoder options, and updated docs accordingly. I've not exposed every possible option, as there are way too many, if someone else would like to tackle that feel free to (maybe encapsulate them in a WebPConfig object, so the C method arguments don't become ridiculously long).
  • Add some tests to cover the new functionality

Notes

  • The WebPAnimEncoder / WebPAnimDecoder API is a bit more high-level than the Muxer API, meaning you can't specify frame fragments and offsets, but is a lot simpler overall (renders the final canvas for you, and takes care of disposal modes, blending, and everything). I suppose if someone really wanted ultimate control, they could re-implement the lower-level API, but it would be a lot more work (and seems you lose the benefit of libwebp doing things like keyframe insertion and compression heuristics for you).
  • Internally, the WebPAnimXXX API's use RGBA for everything, even if the source and final output data is YUV. So, be aware of this, if performance is a huge concern. For most use cases, there shouldn't be much noticeable difference. I've put in some simple hacks, to still support the case of reading / writing images without alpha ("RGB" mode), but internally libwebp is still converting these to RGBA.
  • The old behavior for unsupported modes (anything not "RGB" or "RGBA") was to fail with an IOError, I've changed it so that other modes are silently converted to the proper mode (with or without alpha). This should make working with WebP and Pillow more user-friendly, just something to be aware of.

Testing

AppVeyor's reporting errors for me even before my changes, but I was able to successfully run the full tests, both with Python2 and Python3, with and without --disable-webpmux, using libwebp 0.6.0 and Python 2.7.13 / 3.6.0 on MacOS 10.12.6.

- Support writing metadata chunks with WebPAnimEncoder
- Add XMP metadata support to legacy WebPEncode wrapper
- Cleanup unused mux code in legacy WebPDecode wrapper
- Fix some bugs present when compiled without WebP Mux support
- Fix conversion from L/P/PA modes when saving WebP files
- Update existing tests, and add new ones for WebP animation and metadata support
@jd20
Copy link
Contributor Author

jd20 commented Sep 27, 2017

Not sure the cause of the AppVeyor errors, but the Travis CI errors regarding for loop initial declarations I'll submit a fix for soon.

@jd20
Copy link
Contributor Author

jd20 commented Sep 27, 2017

WebP 0.5.0 added the WebPAnimEncoder and Decoder API, so will need an additional version check for that. Probably need to break up the version checks into: HAVE_WEBPMUX and HAVE_WEBPANIM, and bring metadata support back into WebPDecode_wrapper, for the case where you have a really old verison of libwebp, but still need the mux features.

@wiredfool
Copy link
Member

  • Both currently supported Ubuntu LTS releases are still on webp 0.4.x, and those are mainstream platforms. Centos 6, the designated old library distro, is there as well.
  • Windows is c99 (I think) but the result is that all declarations of variables have to be at the beginning of the block. So WebPAnimEncoderOptions enc_options; at _webp.c:69 needs to be up with the other declarations. I see a few others from a quick scan.
  • We've recently been being tighter about allowed modes for saving items (e.g., transparency isn't allowed in JPEG anymore), but in this case, it looks like it's always lossless conversion to RGB(A), rather than dropping a transparency channel. That should be acceptable.

_webp.c Outdated

mux = WebPMuxCreate(&webp_data, 1);
if (mux == NULL) {
fprintf(stderr, "ERROR: Could not re-mux to add metadata.\n");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines should set a PythonError and return NULL

_webp.c Outdated

// HACK: If original mode was RGB, we need to strip alpha before passing back, this
// is needed because internally WebPAnimDecoder doesn't suppor ta non-alpha mode
uint32_t size = decp->info.canvas_width * 4 * decp->info.canvas_height;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be a py_ssize_t, and there should be a check for overflow, or some sort of guarantee that this will never overflow due to limits on info.canvas_width.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, looking at this, I'm not sure why we're not just returning this as RGBA or RGBX

@jd20
Copy link
Contributor Author

jd20 commented Sep 27, 2017

Thanks for the comments, I can work on those fixes today, good point about RGBX, I didn't realize Pillow supported it, that's why I thought I needed to convert to RGB first.

@jd20
Copy link
Contributor Author

jd20 commented Sep 27, 2017

Also, the original webp code used fprintf for libwebp errors, and I more or less just extended that pattern not knowing better, should I replace all those cases in _webp.c with real Python exceptions?

@wiredfool
Copy link
Member

Also, the original webp code used fprintf for libwebp errors, and I more or less just extended that pattern not knowing better, should I replace all those cases in _webp.c with real Python exceptions?

Yes, go ahead and upgrade them.

The other thing that I'd like to look at is making the webp decoding work in a similar manner to the other decoders -- namely doing the bulk of the work in the load phase rather than the open phase. That's one of the things that the long neglected webp decoder PR did. I'm not sure if the best course is to merge this first, or to try to do both at the same time.

@jd20
Copy link
Contributor Author

jd20 commented Sep 27, 2017

The PR I submitted is doing lazy loading, if using the WebPAnimDDecoder API. I copy all the data to memory and parse out canvas info and metadata chunks in open(), but the actual image data doesn't get decoded until load() is called.

@wiredfool
Copy link
Member

Ok, I wasn't quite precise there -- Reading in the entire file in open is not really in spec, especially for potentially large animations. It also winds up doing the image decode as a chunk, then writing to a string, which is then run through the raw decoder.

This is a drawback of the original webp implementation, and not really your issue, but it's tightly related to what you're building. We've added some features to the decoder api that may make it easier to fix this, but I need to dig a bit to check.

@jd20
Copy link
Contributor Author

jd20 commented Sep 27, 2017

Right, it's not streaming from the file, decoding chunks as it goes along. The WebPAnimDecoder API doesn't support that either unfortunately (I can decode frame-by-frame, but not anything more granular than that). You really have to drop down to the Demux API to get the kind of control needed. So there'd be a lot more work required, to get it fully optimal.

- Change WebPAnimEncoder/Decoder to use RGBX mode instead of RGB (since internally it is using RGBA always)
….h headers meet the ABI version requirements

- Add WEBPMUX support back to WebPDecode_wrapper (to support older versions of libwebp that have mux support, but not animation)
- Add HAVE_WEBPANIM flag, and use it appropriately
- Update documentation / tests
- Add a test to cover RGBX import
- HandleMuxError function needs to be tied to WEBP_HAVEMUX, not WEBP_HAVEANIM
@jd20
Copy link
Contributor Author

jd20 commented Sep 28, 2017

Pushed latest changes, which addresses all the issues in comments (except doing block decoding, that's too out of scope for me at this time). Travis build is passing now too, AppVeyor has been failing for some unrelated reason, I'm not familiar enough to know how to fix.

@radarhere radarhere added the WebP label Sep 28, 2017
@wiredfool
Copy link
Member

Appveyor's failing on a variable definition after a command in the block.

diff --git a/_webp.c b/_webp.c
index dcb2e29..fc3ac52 100644
--- a/_webp.c
+++ b/_webp.c
@@ -34,6 +34,8 @@ static const char* const kErrorMessages[-WEBP_MUX_NOT_ENOUGH_DATA + 1] = {
 };
 
 PyObject* HandleMuxError(WebPMuxError err, char* chunk) {
+    char message[100];
+    
     assert(err <= WEBP_MUX_NOT_FOUND && err >= WEBP_MUX_NOT_ENOUGH_DATA);
 
     // Check for a memory error first
@@ -42,7 +44,6 @@ PyObject* HandleMuxError(WebPMuxError err, char* chunk) {
     }
 
     // Create the error message
-    char message[100];
     if (chunk == NULL) {
         sprintf(message, "could not assemble chunks: %s", kErrorMessages[-err]);
     } else {

@jd20
Copy link
Contributor Author

jd20 commented Sep 29, 2017

Ok, now I see the appveyor errors for _webp.c, going through those...

@hugovk
Copy link
Member

hugovk commented Sep 30, 2017

Maybe: instead of importing from the internal _WEBP in the unit tests, import the public WebPImagePlugin.py interface instead? Generally we should only test public stuff, and we want to test how end users should use the API.

@jd20
Copy link
Contributor Author

jd20 commented Sep 30, 2017

@hugovk The majority of the webp tests do use WebPImagePlugin.py, the only direct usage of the _webp module is to 1) check presence of webp features, for which there's no public equivalent, and 2) couple of the original webp tests which already existed, like test_version and test_WebPEncode_with_invalid_args. These tests could be dropped if we don't think they're useful, but I didn't create them originally so I left them alone when adding the new tests.

@@ -35,6 +35,11 @@ def check_webp_mux(self):
self.assertEqual(features.check('webp_mux'),
_webp.HAVE_WEBPMUX)

@unittest.skipUnless(HAVE_WEBP, True)
def check_webp_anim(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want this to run on the CI, which may be useful for debugging, call it test_ instead of check_. Could change the other two too.

@hugovk
Copy link
Member

hugovk commented Sep 30, 2017

Here's the problem (using this branch on macOS Sierra):

$ python
Python 2.7.13 (default, Dec 18 2016, 07:03:39)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import _webp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(PIL/_webp.so, 2): Symbol not found: _sprintf_s
  Referenced from: PIL/_webp.so
  Expected in: flat namespace
 in PIL/_webp.so
>>> 

After replacing _sprintf_s with sprintf:

$ python
Python 2.7.13 (default, Dec 18 2016, 07:03:39)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import _webp
>>> 

I'll make a PR into your branch.


Without the fix, setup.py said "WEBP support available":
https://travis-ci.org/python-pillow/Pillow/jobs/281460879#L2635

But moments later selftest.py said "WEBP support not installed":
https://travis-ci.org/python-pillow/Pillow/jobs/281460879#L2654

That's because setup.py looks for library headers and things, but selftest.py uses Pillow's features.check("webp"), which is essentially doing the same try: from PIL import _webp... and fails. Let's use features for consistency.

Copy link
Member

@hugovk hugovk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor flake8 stuff:

PIL/WebPImagePlugin.py:55:80: E501 line too long (88 > 79 characters)
PIL/WebPImagePlugin.py:122:26: E261 at least two spaces before inline comment
PIL/WebPImagePlugin.py:130:30: E261 at least two spaces before inline comment
PIL/WebPImagePlugin.py:130:80: E501 line too long (81 > 79 characters)
PIL/WebPImagePlugin.py:135:19: E261 at least two spaces before inline comment
PIL/WebPImagePlugin.py:168:1: E302 expected 2 blank lines, found 1
PIL/WebPImagePlugin.py:207:80: E501 line too long (103 > 79 characters)
PIL/WebPImagePlugin.py:209:73: E261 at least two spaces before inline comment
PIL/WebPImagePlugin.py:209:80: E501 line too long (97 > 79 characters)
PIL/WebPImagePlugin.py:240:20: E713 test for membership should be 'not in'
PIL/WebPImagePlugin.py:256:80: E501 line too long (101 > 79 characters)
PIL/WebPImagePlugin.py:276:1: E302 expected 2 blank lines, found 1
Tests/test_file_webp.py:11:1: E302 expected 2 blank lines, found 1
Tests/test_file_webp.py:42:80: E501 line too long (87 > 79 characters)
Tests/test_file_webp_animated.py:1:1: F401 'helper.hopper' imported but unused
Tests/test_file_webp_animated.py:11:1: E302 expected 2 blank lines, found 1
Tests/test_file_webp_animated.py:71:80: E501 line too long (84 > 79 characters)
Tests/test_file_webp_animated.py:106:80: E501 line too long (84 > 79 characters)
Tests/test_file_webp_animated.py:134:80: E501 line too long (80 > 79 characters)
Tests/test_file_webp_lossless.py:11:1: E302 expected 2 blank lines, found 1
Tests/test_file_webp_metadata.py:11:1: E302 expected 2 blank lines, found 1


im = Image.open(temp_file)
self.assertEqual(im.n_frames, 2)
print("Test File: " + temp_file)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this print...

# Compare first frame to original
im.load()
im.save(temp_file2)
print("Frame 1: " + temp_file2)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... and remove this print.

@property
def n_frames(self):
if not _webp.HAVE_WEBPANIM:
return 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about setting self._n_frames = 1 in _open when not _webp.HAVE_WEBPANIM?

Then these two lines can be removed...

@property
def is_animated(self):
if not _webp.HAVE_WEBPANIM:
return False
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... and also these two lines can be removed.

_webp.c Outdated

/*
* Check the versions from mux.h and demux.h, to ensure the WebPAnimEncoder and
* WebPAnimDecoder API's are present (initial support was added in 0.5.0). The
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plural of API is "APIs".


def test_n_frames(self):
"""
Ensure that webp format sets n_frames and is_animated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

webp -> WebP


def test_seeking(self):
"""
Create an animated webp file, and then try seeking through
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

webp -> WebP

_webp.c Outdated
@@ -66,17 +590,19 @@ PyObject* WebPEncode_wrapper(PyObject* self, PyObject* args)
return ret;
}
#else
{
{
/* I want to truncate the *_size items that get passed into webp
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

webp -> WebP

@@ -661,8 +661,13 @@ The :py:meth:`~PIL.Image.Image.save` method supports the following options:
If present and true, instructs the WEBP writer to use lossless compression.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WEBP -> WebP

@@ -672,6 +677,51 @@ The :py:meth:`~PIL.Image.Image.save` method supports the following options:
The exif data to include in the saved file. Only supported if
the system webp library was built with webpmux support.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

webp -> WebP

@hugovk
Copy link
Member

hugovk commented Sep 30, 2017

WebP 0.5.0 added the WebPAnimEncoder and Decoder API, so will need an additional version check for that. Probably need to break up the version checks into: HAVE_WEBPMUX and HAVE_WEBPANIM, and bring metadata support back into WebPDecode_wrapper, for the case where you have a really old verison of libwebp, but still need the mux features.

Both currently supported Ubuntu LTS releases are still on webp 0.4.x, and those are mainstream platforms. Centos 6, the designated old library distro, is there as well.

  • Ubuntu 16.04 LTS has EOL in April 2021
  • Ubuntu 14.04 LTS has EOL in April 2019
  • CentOS 6 has EOL in November 2020

https://linuxlifecycle.com/

Right now we're only installing and therefore testing code with WebP >= 0.5.0, which means we could very well break the untested, legacy code during the next four years. Is it worth installing an old WebP on the CI to also test the legacy code?

We can see the untested code here:

I think it'd be useful to put a note at the top of the code to mention when different bits can be removed/deprecated, as we're otherwise we're likely to forget. What do you think?

@wiredfool
Copy link
Member

@hugovk All of the dockers (except for alpine) are installing webp from the distro packages.

Part of the point of the docker runs is to have a diversity of library versions so that we're not just running against the latest and greatest that no one is using because they're all on distro packages.

@wiredfool
Copy link
Member

That's because setup.py looks for library headers and things, but selftest.py uses Pillow's features.check("webp"), which is essentially doing the same try: from PIL import _webp... and fails. Let's use features for consistency.

Those are two distinct checks. One is for what we should be compiling, one is what successfully compiled.

@hugovk
Copy link
Member

hugovk commented Oct 1, 2017

@hugovk All of the dockers (except for alpine) are installing webp from the distro packages.

Part of the point of the docker runs is to have a diversity of library versions so that we're not just running against the latest and greatest that no one is using because they're all on distro packages.

Aha, great! Checking the CentOS 6 build, it has webpmux but not animation, as expected, and skips the TestFileWebpAnimation tests as expected. But the legacy mode _open is not covered:
https://coveralls.io/builds/13510797/source?filename=PIL%2FWebPImagePlugin.py#L35

And looks like we're only collecting coverage from the first 8 jobs: https://coveralls.io/builds/13510797

Checking logs, here's a problem with coveralls-merge on CentOS that doesn't occur for standard 2.7:

Successfully built coveralls-merge docopt pycparser
Installing collected packages: docopt, coveralls, coveralls-merge, asn1crypto, enum34, pycparser, cffi, ipaddress, cryptography, pyOpenSSL
Successfully installed asn1crypto-0.23.0 cffi-1.11.0 coveralls-1.2.0 coveralls-merge-0.0.2 cryptography-2.0.3 docopt-0.6.2 enum34-1.1.6 ipaddress-1.0.18 pyOpenSSL-17.3.0 pycparser-2.18
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.13/bin/coveralls-merge", line 11, in <module>
    sys.exit(main())
  File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/coveralls_merge/core.py", line 84, in main
    m.collect()
  File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/coveralls_merge/core.py", line 51, in collect
    self.merge(self.parse(f))
  File "/home/travis/virtualenv/python2.7.13/lib/python2.7/site-packages/coveralls_merge/core.py", line 36, in parse
    return json.loads(src.read())
  File "/opt/python/2.7.13/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/opt/python/2.7.13/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/python/2.7.13/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

@wiredfool
Copy link
Member

I'm going to bet that the error there is because there's no data coming in from the coveralls run, not that there's something funky in coveralls-merge.

Avoid 'Symbol not found: _sprintf_s' on 'from PIL import _webp'
@hugovk
Copy link
Member

hugovk commented Oct 1, 2017

Yep, some divergence already with lcov:

Setting up lcov (1.10-1build1) ...
Setting up libgd-gd2-perl (1:2.46-3.1build1) ...
Capturing coverage data from .
Found gcov version: 4.8.4
Scanning . for .gcda files ...
geninfo: ERROR: no .gcda files found in .!

Versus:

Setting up lcov (1.10-1build1) ...
Setting up libgd-gd2-perl (1:2.46-3.1build1) ...
Capturing coverage data from .
Found gcov version: 4.8.4
Scanning . for .gcda files ...
Found 1 data files in .
Processing _webp.gcda
Finished .info-file creation

One thing: Docker and native both share after_success.sh script but not install.sh or script.sh. Are the Dockers built with lcov/CFLAGS="-coverage? Where is the Pillow build defined for Docker?

@jd20
Copy link
Contributor Author

jd20 commented Oct 1, 2017

@hugovk I've pushed the formatting changes you suggested, the only part I didn't understand was to use the features module in setup.py. I'm really not familiar with how setup.py works, is it as simple as replacing the _find_include_file and _find_library_file calls with features.check('webp')? I thought this code wasn't just checking for existence of webp support, but also determining which files to bundle up during setup process?


if (_webp.WebPDecoderVersion() < 0x0200):
self.skipTest('lossless not included')

# WebPAnimDecoder only retuns RGBA or RGBX, never RGB
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

returns

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed the typo (was also in test_file_webp.py too).

@hugovk
Copy link
Member

hugovk commented Oct 2, 2017

@jd20 Thanks! Let's not worry about the features stuff too much, it was a bit of a red herring, the main problem there was the sprintf thing. It'd be nice if we can figure out why we're not collecting/building coverage properly for Docker builds, and then we know the legacy code remains tested properly.

@wiredfool
Copy link
Member

@hugovk That's a discussion that's probably better held in the docker-images repo.

@jd20
Copy link
Contributor Author

jd20 commented Oct 3, 2017

Anything else I can do, to help push this PR along?

@jd20
Copy link
Contributor Author

jd20 commented Oct 19, 2017

Just checking in again, if there's anything I can do to move this PR along?

@wiredfool
Copy link
Member

I think you've pushed it as far as possible. I need to review where we were on this and make one last pass before merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants