Skip to content

Improve test bot stability #12125

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jul 26, 2020
Merged

Improve test bot stability #12125

merged 2 commits into from
Jul 26, 2020

Conversation

timvandermeij
Copy link
Contributor

@timvandermeij timvandermeij commented Jul 26, 2020 •
edited
Loading

No description provided.

@timvandermeij timvandermeij mentioned this pull request Jul 26, 2020
Merged
@timvandermeij
Copy link
Contributor Author

/botio test

@pdfjsbot
Copy link

From: Bot.io (Linux m4)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.67.70.0:8877/7fcc43c7835a0e1/output.txt

@pdfjsbot
Copy link

From: Bot.io (Windows)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.215.176.217:8877/7b771b5bcbb25d3/output.txt

@pdfjsbot
Copy link

From: Bot.io (Linux m4)

Failed

Full output at http://54.67.70.0:8877/7fcc43c7835a0e1/output.txt

Total script time: 26.49 mins

  • Font tests: Passed
  • Unit tests: FAILED
  • Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/7fcc43c7835a0e1/reftest-analyzer.html#web=eq.log

@pdfjsbot
Copy link

From: Bot.io (Windows)

Failed

Full output at http://54.215.176.217:8877/7b771b5bcbb25d3/output.txt

Total script time: 60.00 mins

@timvandermeij timvandermeij mentioned this pull request Jul 26, 2020
Closed
7 tasks
@timvandermeij timvandermeij marked this pull request as draft July 26, 2020 12:57
@timvandermeij timvandermeij changed the title Upgrade Puppeteer to version 5 and remove extraction log suppression code Upgrade Puppeteer to version 5 Jul 26, 2020
@timvandermeij
Copy link
Contributor Author

/botio test

@pdfjsbot
Copy link

From: Bot.io (Windows)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.215.176.217:8877/a1da348904f4db2/output.txt

@pdfjsbot
Copy link

From: Bot.io (Linux m4)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.67.70.0:8877/81c5c22bb415e98/output.txt

@timvandermeij
Copy link
Contributor Author

Hm, not sure what's up with Windows here. Maybe an old process is still running?

@pdfjsbot
Copy link

From: Bot.io (Linux m4)

Failed

Full output at http://54.67.70.0:8877/81c5c22bb415e98/output.txt

Total script time: 15.09 mins

  • Font tests: Passed
  • Unit tests: Passed
  • Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/81c5c22bb415e98/reftest-analyzer.html#web=eq.log

@pdfjsbot
Copy link

From: Bot.io (Windows)

Failed

Full output at http://54.215.176.217:8877/a1da348904f4db2/output.txt

Total script time: 60.00 mins

@timvandermeij timvandermeij changed the title Upgrade Puppeteer to version 5 Upgrade Puppeteer Jul 26, 2020
@timvandermeij
Copy link
Contributor Author

/botio test

@pdfjsbot
Copy link

From: Bot.io (Linux m4)

Received

Command cmd_test from @timvandermeij received. Current queue size: 1

Live output at: http://54.67.70.0:8877/4ace673cccca99e/output.txt

@pdfjsbot
Copy link

From: Bot.io (Windows)

Received

Command cmd_test from @timvandermeij received. Current queue size: 1

Live output at: http://54.215.176.217:8877/e0a9f9917efe837/output.txt

@pdfjsbot
Copy link

From: Bot.io (Linux m4)

Failed

Full output at http://54.67.70.0:8877/4ace673cccca99e/output.txt

Total script time: 15.12 mins

  • Font tests: Passed
  • Unit tests: FAILED
  • Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/4ace673cccca99e/reftest-analyzer.html#web=eq.log

@pdfjsbot
Copy link

From: Bot.io (Windows)

Failed

Full output at http://54.215.176.217:8877/e0a9f9917efe837/output.txt

Total script time: 60.00 mins

@timvandermeij timvandermeij marked this pull request as ready for review July 26, 2020 18:40
@timvandermeij timvandermeij changed the title Upgrade Puppeteer Stabilize the bots Jul 26, 2020
@timvandermeij
Limit the allowed versions for Jasmine and Puppeteer
c7c6c90 
Jasmine >= 3.6.0 causes intermittent test failures because of random
task abortions.

Puppeteer >= 4.0.0 causes ENOTEMPTY/EBUSY errors during shutdown on the
Windows bot.

Moreover, `jasmine-core` is a dependency of `jasmine` so it doesn't have
to be required separately.
@timvandermeij
Fix a bug in the temporary folder check in the test runner
65e76a3 
The `noPrompt` option doesn't exist and should be `noPrompts`.
@timvandermeij timvandermeij changed the title Stabilize the bots Improve unit test stability Jul 26, 2020
@timvandermeij
Copy link
Contributor Author

/botio unittest

@pdfjsbot
Copy link

From: Bot.io (Windows)

Received

Command cmd_unittest from @timvandermeij received. Current queue size: 0

Live output at: http://54.215.176.217:8877/37ba79989946246/output.txt

@pdfjsbot
Copy link

From: Bot.io (Linux m4)

Received

Command cmd_unittest from @timvandermeij received. Current queue size: 0

Live output at: http://54.67.70.0:8877/de226b1bad336dc/output.txt

@pdfjsbot
Copy link

From: Bot.io (Linux m4)

Success

Full output at http://54.67.70.0:8877/de226b1bad336dc/output.txt

Total script time: 3.77 mins

  • Unit Tests: Passed

@pdfjsbot
Copy link

From: Bot.io (Windows)

Success

Full output at http://54.215.176.217:8877/37ba79989946246/output.txt

Total script time: 4.71 mins

  • Unit Tests: Passed

@timvandermeij
Copy link
Contributor Author

/botio unittest

@pdfjsbot
Copy link

From: Bot.io (Linux m4)

Received

Command cmd_unittest from @timvandermeij received. Current queue size: 0

Live output at: http://54.67.70.0:8877/ecdbf2f0c35b081/output.txt

@pdfjsbot
Copy link

From: Bot.io (Windows)

Received

Command cmd_unittest from @timvandermeij received. Current queue size: 0

Live output at: http://54.215.176.217:8877/ed50f1b7c99154b/output.txt

@timvandermeij timvandermeij changed the title Improve unit test stability Improve test bot stability Jul 26, 2020
@pdfjsbot
Copy link

From: Bot.io (Linux m4)

Success

Full output at http://54.67.70.0:8877/ecdbf2f0c35b081/output.txt

Total script time: 3.82 mins

  • Unit Tests: Passed

@pdfjsbot
Copy link

From: Bot.io (Windows)

Success

Full output at http://54.215.176.217:8877/ed50f1b7c99154b/output.txt

Total script time: 4.72 mins

  • Unit Tests: Passed

@timvandermeij
Copy link
Contributor Author

/botio test

@mozilla mozilla deleted a comment from pdfjsbot Jul 26, 2020
@mozilla mozilla deleted a comment from pdfjsbot Jul 26, 2020
@mozilla mozilla deleted a comment from pdfjsbot Jul 26, 2020
@mozilla mozilla deleted a comment from pdfjsbot Jul 26, 2020
@timvandermeij
Copy link
Contributor Author

/botio test

@pdfjsbot
Copy link

From: Bot.io (Linux m4)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.67.70.0:8877/af841d2bf6f413c/output.txt

@pdfjsbot
Copy link

From: Bot.io (Windows)

Received

Command cmd_test from @timvandermeij received. Current queue size: 0

Live output at: http://54.215.176.217:8877/d4a00c80dfd2fad/output.txt

@pdfjsbot
Copy link

From: Bot.io (Linux m4)

Failed

Full output at http://54.67.70.0:8877/af841d2bf6f413c/output.txt

Total script time: 26.92 mins

  • Font tests: Passed
  • Unit tests: Passed
  • Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/af841d2bf6f413c/reftest-analyzer.html#web=eq.log

@pdfjsbot
Copy link

From: Bot.io (Windows)

Failed

Full output at http://54.215.176.217:8877/d4a00c80dfd2fad/output.txt

Total script time: 28.87 mins

  • Font tests: Passed
  • Unit tests: Passed
  • Regression tests: FAILED

Image differences available at: http://54.215.176.217:8877/d4a00c80dfd2fad/reftest-analyzer.html#web=eq.log

Snuffleupagus
Snuffleupagus approved these changes Jul 26, 2020
View reviewed changes
Copy link
Collaborator

@Snuffleupagus Snuffleupagus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me; thanks for getting the tests working again!

@timvandermeij timvandermeij merged commit c7eb79c into mozilla:master Jul 26, 2020
@timvandermeij timvandermeij deleted the puppeteer branch July 26, 2020 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Snuffleupagus Snuffleupagus Snuffleupagus approved these changes

Assignees
No one assigned
Labels
test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@timvandermeij @pdfjsbot @Snuffleupagus