Fix intermittent unit/reference test failures

[timvandermeij]

To debug this, pull request #12124 included the browser name in the logs for the unit tests too.

The following is a list of unit test failures we have seen:

• TEST-UNEXPECTED-FAIL | cleans up document resources during rendering of page | in firefox | Unhandled promise rejection: Error: shall fail cleanup in http://127.0.0.1:38165/node_modules/systemjs/dist/system.js line 4 > eval (line 1585) (Linux; fixed in Attempt to reduce intermittent failures in the "cleans up document resources during rendering of page" unit-test #12126)
• TEST-UNEXPECTED-FAIL | creates pdf doc from non-existent URL | in chrome | Unhandled promise rejection: MissingPDFException: Missing PDF "http://127.0.0.1:38165/test/pdfs/non-existent.pdf". (Linux)
• TEST-UNEXPECTED-FAIL | gets default page mode | in chrome | Unhandled promise rejection: AbortException: Worker was terminated. (Linux; fixed in Prevent Uncaught (in promise) AbortException when running the unit-tests #12144/Ignore fetch() errors, in PDFFetchStreamRangeReader, once the request has been aborted #12136)
• TEST-UNEXPECTED-FAIL | gets non-existent outline | in chrome | Unhandled promise rejection: AbortException: Worker was terminated. (Linux; fixed in Prevent Uncaught (in promise) AbortException when running the unit-tests #12144/Ignore fetch() errors, in PDFFetchStreamRangeReader, once the request has been aborted #12136)
• TEST-UNEXPECTED-FAIL | gets operatorList with JPEG image (issue 4888) | in chrome | Unhandled promise rejection: AbortError: The user aborted a request. (Windows; fixed in Prevent Uncaught (in promise) AbortException when running the unit-tests #12144/Ignore fetch() errors, in PDFFetchStreamRangeReader, once the request has been aborted #12136)
• TEST-UNEXPECTED-FAIL | should correctly render PDFs in parallel | in firefox | Unhandled promise rejection: AbortError: The operation was aborted. (line 369) (Windows; fixed in Prevent Uncaught (in promise) AbortException when running the unit-tests #12144/Ignore fetch() errors, in PDFFetchStreamRangeReader, once the request has been aborted #12136)
• TEST-UNEXPECTED-FAIL | multiple render() on the same canvas | in firefox | Failed: shall fail rendering (Windows; fixed in Attempt to reduce intermittent failures in the "multiple render() on the same canvas" unit-test #12171)

It's clear from this overview that most problems are related to unexpected action abortions, both in Chrome/Firefox and on Windows/Linux. This wasn't happening a few days ago, so maybe something regressed? The second one is also interesting since we explicitly catch the exception in the unit test.

Pull request #12125 tries to improve the situation for the reference test failures.

[timvandermeij]

It seems to have something to do with 47ab676, also the first PR where we noticed the issues. If I revert that commit, the unit tests on Linux seem to pass all the time. Strange...

[Snuffleupagus]

If I revert that commit, the unit tests on Linux seem to pass all the time.

Most likely the updated Jasmine-packages are responsible in that case. Perhaps we should just revert those changes for now, and also pin the packages at the old ~3.5.0 versions?

(Obviously that wouldn't really fix things, only hide the errors for now, and future updates would be blocked until the problems have been identified and addressed.)

[timvandermeij]

I have made #12125 which should solve the problems for now so the unit tests all pass again. This issue will remain open for the follow-up because those upgrades need to be unblocked.

[timvandermeij]

I filed an upstream issue for Jasmine: jasmine/jasmine#1840

[timvandermeij]

I received a reply for the Jasmine developers and it seems like a pre-existing issue in our unit tests that just surfaced in 3.6 because of behavior changes in Jasmine. The idea is to to track down the test that causes it by trying to get it reproducible, for example by running the tests until failures happen and keeping track of the random seed so we can replay it. Another approach is to track it down backwards, i.e., finding what can throw the AbortException and tracing it back to a particular test, but that may be more difficult.

[timvandermeij]

Only one unit test failure remains after which the Jasmine update is unblocked.

[Snuffleupagus]

TEST-UNEXPECTED-FAIL | creates pdf doc from non-existent URL | in chrome | Unhandled promise rejection: MissingPDFException: Missing PDF "http://127.0.0.1:38165/test/pdfs/non-existent.pdf".

I've been able to reproduce this a handful of times locally with Jasmine 3.6.1, even on Windows, but it's extremely intermittent[1] to the point that I cannot do any meaningful debugging :-(

---

[1] Something like 1-2 failures for 100 runs, even with a constant Jasmine seed.

[timvandermeij]

I'll also give this a try. Perhaps it's more easily reproduced on Linux for some reason, or we can find a way to make it more reproducible so it's hopefully easier to find a fix.

[timvandermeij]

I'm not having any "luck" so far with reproducing the failure at all on Linux. If you still have it, could you post the random seed so I can try with that?