Adding --recurse / --no-recurse option #283

mayuriesha · 2021-11-11T20:22:53Z

To help us get this pull request reviewed and merged quickly, please be sure to include the following items:

Tests (if applicable)
Documentation (if applicable)
Changelog entry
A full explanation here in the PR description of the work done

PR Type

What kind of change does this PR introduce?

Backward Compatibility

Is this change backward compatible with the most recently released version? Does it introduce changes which might change the user experience in any way? Does it alter the API in any way?

Yes (backward compatible)
No (breaking changes)

Issue Linking

Closes #268

What's new?

Introducing new flag --recurse/--no-recurse to scan entire directory or just the root directory. It is defaulted to recurse to maintain the current behavior of scan-folder.

rbailey-godaddy

If you don't mind a little more tweaking...

It would be more consistent with existing behavior to look at the value of global_options.recurse instead of extracting it in the caller and then passing recurse as a separate parameter. Additionally, you can then just check self.global_options.recurse within the class instead of needing to add self.recurse.

Also, the more I look at this, the more I can't see where the recurse parameter to scan_folder.main() is coming from. The crazy thing is that I play-tested it and it obviously works, so I guess I'm good with things as-is if nobody else has complaints -- I just can't understand what is happening!

mayuriesha · 2021-11-11T21:53:10Z

I believe self.global_options are for the flags that are defined in cli.py which are tartufo command line options. Where as the recurse is an option that goes with sub_command scan-folder. So I am not sure if we can use self.global_options.recurse.

About the parameter recurse for scan_folder->main, as far as I understand the click.command is handling the arguments/options that are defined and passes it to the method definition below.
https://github.com/pallets/click/blob/5d759e0ab383136f9869cdb5e14f00c78b38a9c9/docs/options.rst

sushantmimani · 2021-11-12T14:56:28Z

README.md

@@ -159,6 +159,7 @@ Commands:
  pre-commit        Scan staged changes in a pre-commit hook.
  scan-local-repo   Scan a repository already cloned to your local system.
  scan-remote-repo  Automatically clone and scan a remote git repository.
+  scan-folder       Scan a folder.


nit

Suggested change

scan-folder Scan a folder.

scan-folder Scan a folder on your local system.

tartufo/scanner.py

sushantmimani · 2021-11-12T15:12:00Z

tests/test_folder_scanner.py

+        self.assertEqual(2, len(issues))
        self.assertEqual("KQ0I97OBuPlGB9yPRxoSxnX52zE=", issues[0].matched_string)
        self.assertEqual(IssueType.Entropy, issues[0].issue_type)


Need a similar assertions for the new issue.

tartufo/scanner.py

tests/test_folder_scanner.py

tests/data/scan_folder/scan_sub_folder/sub_folder_test.txt

sushantmimani · 2021-11-12T15:28:37Z

CHANGELOG.md

@@ -20,6 +20,8 @@ Features:
  `--scan-filenames/--no-scan-filenames` flag which allows users to enable or disable file name scanning.
 * [#254](https://github.com/godaddy/tartufo/pull/260) - Changes the default value of
  `--regex/--no-regex` to True.
+* [#268](https://github.com/godaddy/tartufo/issues/268) - Adds a new
+  `--recurse / --no-recurse` flag which allows users scan entire directory or just the root directory


Suggested change

`--recurse / --no-recurse` flag which allows users scan entire directory or just the root directory

`--recurse / --no-recurse` flag which allows users to recursively scan the entire directory or just the root directory

sushantmimani

LGTM 🦅

tarkatronic

No hard blockers here; overall it looks great! Just a few suggestions to improve the tests with some Python-isms. 😄

Thank you!

tarkatronic · 2021-11-16T19:55:31Z

tests/test_folder_scanner.py

+        actual_issues = []
+        for issue in issues:
+            actual_issues.append(issue.matched_string)


Not a blocker, but cases like this are ideal for list comprehensions 😄

Suggested change

actual_issues = []

for issue in issues:

actual_issues.append(issue.matched_string)

actual_issues = [issue.matched_string for issue in issues]

tarkatronic · 2021-11-16T19:56:21Z

tests/test_folder_scanner.py


        with patch("pathlib.Path.open", side_effect=OSError()):
            with self.assertRaises(click.FileError):
                list(test_scanner.scan())

+    def test_scan_all_the_files_recursively(self):
+        folder_path = pathlib.Path(__file__).parent / "data/scan_folder"


All path separators should be outside the string, so that we can be absolutely certain they are OS-independent.

Suggested change

folder_path = pathlib.Path(__file__).parent / "data/scan_folder"

folder_path = pathlib.Path(__file__).parent / "data" / "scan_folder"

tarkatronic · 2021-11-16T19:56:59Z

tests/test_folder_scanner.py

+        actual_issues = []
+        for issue in issues:
+            actual_issues.append(issue.matched_string)


Same list comprehension trick can be applied here

Suggested change

actual_issues = []

for issue in issues:

actual_issues.append(issue.matched_string)

actual_issues = [issue.matched_string for issue in issues]

tarkatronic · 2021-11-16T19:57:26Z

tests/test_folder_scanner.py

+        )
+
+    def test_scan_only_root_level_files(self):
+        folder_path = pathlib.Path(__file__).parent / "data/scan_folder"


Suggested change

folder_path = pathlib.Path(__file__).parent / "data/scan_folder"

folder_path = pathlib.Path(__file__).parent / "data" / "scan_folder"

tarkatronic · 2021-11-16T19:57:47Z

tests/test_folder_scanner.py

+        actual_issues = []
+        for issue in issues:
+            actual_issues.append(issue.matched_string)


Suggested change

actual_issues = []

for issue in issues:

actual_issues.append(issue.matched_string)

actual_issues = [issue.matched_string for issue in issues]

mayuriesha requested review from tarkatronic, sushantmimani and rbailey-godaddy November 11, 2021 20:22

mayuriesha requested a review from a team as a code owner November 11, 2021 20:22

rbailey-godaddy approved these changes Nov 11, 2021

View reviewed changes

mayuriesha force-pushed the recurse_option branch from cf0e252 to 5bcbc50 Compare November 11, 2021 21:21

sushantmimani requested changes Nov 12, 2021

View reviewed changes

mayuriesha requested a review from sushantmimani November 12, 2021 21:20

mayuriesha force-pushed the recurse_option branch from 9adaee0 to 49d6adb Compare November 12, 2021 21:23

mayuriesha added 4 commits November 12, 2021 15:20

Adding --recurse / --no-recurse option

91592ca

Addressing review comments

4b5cc40

Updating test to check for new issue

892b367

Asserting on number of occurances

09c349b

mayuriesha force-pushed the recurse_option branch from 49d6adb to 09c349b Compare November 12, 2021 23:21

Merge remote-tracking branch 'upstream/v3.x' into recurse_option

eb5da67

sushantmimani approved these changes Nov 16, 2021

View reviewed changes

tarkatronic approved these changes Nov 16, 2021

View reviewed changes

Addressing review comments

7eaa7e1

mayuriesha merged commit 2daef6e into godaddy:v3.x Nov 16, 2021

mayuriesha linked an issue Nov 16, 2021 that may be closed by this pull request

Add a --recurse / --no-recurse flag to scan-folder #268

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Adding --recurse / --no-recurse option #283

Adding --recurse / --no-recurse option #283

Uh oh!

mayuriesha commented Nov 11, 2021

Uh oh!

rbailey-godaddy left a comment

Uh oh!

mayuriesha commented Nov 11, 2021

Uh oh!

sushantmimani Nov 12, 2021 •

edited

Loading

Uh oh!

Uh oh!

sushantmimani Nov 12, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sushantmimani Nov 12, 2021

Uh oh!

sushantmimani left a comment

Uh oh!

tarkatronic left a comment

Uh oh!

tarkatronic Nov 16, 2021

Uh oh!

tarkatronic Nov 16, 2021

Uh oh!

tarkatronic Nov 16, 2021

Uh oh!

tarkatronic Nov 16, 2021

Uh oh!

tarkatronic Nov 16, 2021

Uh oh!

Uh oh!

	scan-folder Scan a folder.
	scan-folder Scan a folder on your local system.

	`--recurse / --no-recurse` flag which allows users scan entire directory or just the root directory
	`--recurse / --no-recurse` flag which allows users to recursively scan the entire directory or just the root directory

	folder_path = pathlib.Path(__file__).parent / "data/scan_folder"
	folder_path = pathlib.Path(__file__).parent / "data" / "scan_folder"

Adding --recurse / --no-recurse option #283

Adding --recurse / --no-recurse option #283

Uh oh!

Conversation

mayuriesha commented Nov 11, 2021

PR Type

Backward Compatibility

Issue Linking

What's new?

Uh oh!

rbailey-godaddy left a comment

Choose a reason for hiding this comment

Uh oh!

mayuriesha commented Nov 11, 2021

Uh oh!

sushantmimani Nov 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sushantmimani Nov 12, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sushantmimani Nov 12, 2021

Choose a reason for hiding this comment

Uh oh!

sushantmimani left a comment

Choose a reason for hiding this comment

Uh oh!

tarkatronic left a comment

Choose a reason for hiding this comment

Uh oh!

tarkatronic Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

tarkatronic Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

tarkatronic Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

tarkatronic Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

tarkatronic Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sushantmimani Nov 12, 2021 •

edited

Loading