Run the integration tests in a random order #19809
Conversation
To avoid being able to introduce dependencies between tests this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test.
To avoid being able to introduce dependencies between tests this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test.
To avoid being able to introduce dependencies between tests this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test.
To avoid being able to introduce dependencies between tests this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test.
To avoid being able to introduce dependencies between tests this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test.
To avoid being able to introduce dependencies between tests this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test.
To avoid being able to introduce dependencies between tests this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test.
To avoid being able to introduce dependencies between tests this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test.
To avoid being able to introduce dependencies between tests this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test.
To avoid being able to introduce dependencies between tests this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test.
To avoid being able to introduce dependencies between tests this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test.
To avoid being able to introduce dependencies between tests this commit makes sure that we close the document between tests so that we can't accidentally rely on state set by a previous test.
|
/botio integrationtest |
From: Bot.io (Windows)ReceivedCommand cmd_integrationtest from @timvandermeij received. Current queue size: 0 Live output at: http://54.193.163.58:8877/594b6bff145e301/output.txt |
From: Bot.io (Linux m4)ReceivedCommand cmd_integrationtest from @timvandermeij received. Current queue size: 0 Live output at: http://54.241.84.105:8877/6409bb54ab3421a/output.txt |
timvandermeij
changed the title
This commit configures Jasmine to no longer run the tests in a fixed order, which combined with the previous isolation commits avoids being able to accidentally introduce dependencies between integration tests.
timvandermeij
force-pushed
the
integration-tests-isolate-all-and-random
branch
from
1d0363b to
5f46791
Compare
From: Bot.io (Linux m4)FailedFull output at http://54.241.84.105:8877/6409bb54ab3421a/output.txt Total script time: 13.27 mins
|
From: Bot.io (Windows)FailedFull output at http://54.193.163.58:8877/594b6bff145e301/output.txt Total script time: 32.04 mins
|
|
/botio integrationtest |
From: Bot.io (Windows)ReceivedCommand cmd_integrationtest from @timvandermeij received. Current queue size: 0 Live output at: http://54.193.163.58:8877/65c80c8fe4e8df5/output.txt |
From: Bot.io (Linux m4)ReceivedCommand cmd_integrationtest from @timvandermeij received. Current queue size: 0 Live output at: http://54.241.84.105:8877/d7588f1d2560985/output.txt |
From: Bot.io (Linux m4)FailedFull output at http://54.241.84.105:8877/d7588f1d2560985/output.txt Total script time: 13.20 mins
|
From: Bot.io (Windows)FailedFull output at http://54.193.163.58:8877/65c80c8fe4e8df5/output.txt Total script time: 32.68 mins
|
|
Looking at the logs it seems the above test timed out every time, on both Linux and Windows; was that just unlucky timing or does it indicate a problem with the test-case itself? |
|
Looking back at recent logs it looks like this started after the Puppeteer (and therefore Chrome) update in #19807, which unfortunately didn't notice at the time because of the other intermittent failures; sorry for that! It only fails in Chrome, and I can reproduce it locally and have collected some debugging information, so I'll create a separate issue to track this because it's a change/regression in Chrome that is unrelated to the work in this PR. |
|
Thank you as well for all the reviews! |
|
I have created #19811 for the |
Previous PRs for this issue removed the existing dependencies between tests, and this commit series finishes the work by preventing creation of new ones. Closing the viewer between tests also simplifies the code and further reduces room for intermittent failures because viewer state can no longer be retained between tests.
This PR is easier to review per commit because the isolation and associated cleanup are bundled per suite.
Fixes #19065.