Skip to content

Python: update test_logical.py to use seed in random #17652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

allrob23
Copy link

@allrob23 allrob23 commented Oct 3, 2024

Pull Request checklist

All query authors

Internal query authors only

  • Autofixes generated based on these changes are valid, only needed if this PR makes significant changes to .ql, .qll, or .qhelp files. See the documentation (internal access required).
  • Changes are validated at scale (internal access required).
  • Adding a new query? Consider also adding the query to autofix.

Description

This PR introduces a global random seed, set immediately after the imports in the test file. By doing so, the tests will exhibit consistent random behavior across runs, making them more deterministic and significantly easier to debug.

Current Issues (Before this PR):

a. Flaky Tests: Test outcomes can vary, sometimes passing and other times failing due to unpredictable random values;
b. Difficult to Reproduce: When a test fails, it becomes challenging to recreate the exact conditions for debugging;
c. Inconsistent CI/CD Results: Different CI pipeline runs may yield inconsistent outcomes;
d. Wasted Debugging Time: Developers spend unnecessary time chasing down issues that are difficult to replicate.

Solution

To address these issues, this PR sets a global random seed at the top of the test file, right after the imports.

Benefits

a. Reproducibility: Each test run will generate the same random values, ensuring consistency;
b. Improved Debugging: Failures can be easily reproduced, aiding in quicker diagnosis and resolution;
c. Stable CI/CD Pipelines: CI runs will produce consistent results, reducing pipeline variability;
d. Explicit Control: The use of controlled randomness is immediately clear in the test structure;
e. Time Efficiency: Developers will spend less time debugging random-related failures, improving overall productivity.

@allrob23 allrob23 requested a review from a team as a code owner October 3, 2024 01:26
@github-actions github-actions bot added the Python label Oct 3, 2024
@allrob23 allrob23 changed the title tests: update test_logical.py to use seed in random Python: update test_logical.py to use seed in random Oct 3, 2024
@yoff
Copy link
Contributor

yoff commented Oct 3, 2024

The file you are patching is not actually supposed to be run (we run a static analysis of it), so all the issues and benefits you list are void.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants