live-tests: add regression tests suite #35837
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @alafanechere and the rest of your teammates on |
alafanechere
changed the title
alafanechere
force-pushed
the
augustin/03-04-add_regression_tests
branch
from
34c6b5f to
99fc98f
Compare
alafanechere
force-pushed
the
augustin/03-04-add_regression_tests
branch
from
95c6731 to
4e06bbd
Compare
alafanechere
force-pushed
the
augustin/03-01-regression-test_declare_debug_mode
branch
from
90587ad to
d78a92b
Compare
alafanechere
force-pushed
the
augustin/03-04-add_regression_tests
branch
2 times, most recently
from
6699069 to
db3029d
Compare
alafanechere
force-pushed
the
regression-tests-package
branch
3 times, most recently
from
0a4f755 to
d9d08c7
Compare
There was a problem hiding this comment.
Looks great @alafanechere! I only reviewed files changed in the last two commits because looks like the rest won't have a diff once the merge conflict is fixed.
One main comment, about the use of fixtures.
| return ConnectorRunner(dagger_client, **spec_target_execution_inputs.to_dict()) | ||
|
|
||
|
|
||
| @pytest.fixture |
There was a problem hiding this comment.
Declaring this (and execution results for other commands) as a pytest fixture has the effect of executing them before any tests run. Is there a reason you prefer to do it this way rather than getting the result as part of the test? I have a slight preference for the latter - to me, executing the calls feels like part of the test behavior, so should fail during execution of a specific test.
Also, will declaring these as fixtures end up running all commands, even if we only want to run a single test? Or will they be executed lazily?
There was a problem hiding this comment.
I think its indeed a fair point of discussion.
As these fixtures are indeed lazily used on test request I found it useful to declare this logic in fixture.
IMO it avoids a lot of code duplication if we create multiple tests performing assertions on command execution results, and it makes sure the execution report is created.
alafanechere
force-pushed
the
augustin/03-04-add_regression_tests
branch
2 times, most recently
from
5f749cc to
a837536
Compare
alafanechere
force-pushed
the
augustin/03-04-add_regression_tests
branch
from
a837536 to
f34082a
Compare
What
Relates to https://github.com/airbytehq/airbyte-internal-issues/issues/6483
This PR boostraps a regression test suite.
It basically declares fixtures with execution results of commands on the control and target connectors version.
It implements a basic expected records test for demonstration, we will port CAT's expected record test later.
The test execution persists test artifact that can be useful for debugging test failures.
Recommended reading order
airbyte-ci/connectors/live-tests/src/live_tests/regression_tests/conftest.pyairbyte-ci/connectors/live-tests/src/live_tests/regression_tests/test_expected_records.pyTest output example