Parse .gitignore As Git Does

[Kurt-von-Laven]

Describe the bug
.gitignore files are parsed in a manner inconsistent with their specification.

To Reproduce
Steps to reproduce the behavior:

1. Setup a yarn 2 project using Plug'n'Play.
2. Create a .gitignore file as recommended by yarn, optionally replacing the .yarn/* with .yarn/**.
3. Set the "gitignore" and "blame" options to true in .jscpd.json.
4. Install some unplugged dependencies using yarn.
5. PnPify some SDKs using yarn pnpify --sdks.
6. Commit your work to git.
7. Run jscpd on the current working directory.
8. The .yarn/sdks/ and .yarn/unplugged/ directories are searched for duplicates despite being gitignored, resulting in a "fatal: no such path" crash from git blame since these directories are not tracked.

Expected behavior
I expected the files and directories within the .yarn/ directory that were not listed in .gitignore with a leading ! to be excluded.

Desktop (please complete the following information):

• OS: Ubuntu
• OS Version 21.04
• NodeJS Version 16.2.0
• jscpd version 3.3.26

Additional context
I have not fully investigated the issue, but I observed that gitignore-to-glob filters hidden files and directories out of the gitignore file, so I recommend simply running git-check-ignore and only supporting this feature when Git is installed. Alternatively, this issue may be resolved by switching to parse-gitignore@v0.5.1.

Workaround
Add "./.yarn/**" to the "ignore" array in .jscpd.json.

[borys-pc33]

I have problem that .gitignore is not consumed irrespective whether I have --gitignore command switcher or put "gitignore": true to the .jscpd.json

Workaround is the only what is works

[Kurt-von-Laven]

@borys-pc33, I am not able to reproduce your issue. I recommend posting your .jscpd.json on Stack Overflow and asking about it there.

[borys-pc33]

{
"threshold": 7,
"reporters": ["html", "console", "badge"],
"ignore": ["/node_modules/", "/bin/", "/obj/", ".git"],
"absolute": false,
"gitignore": true,
"output": "jscpd-report",
"minLines": 5,
"minTokens": 40,
"mode": "mild",
"formatsExts": {
"css": ["css"],
"liquid": ["liquid"],
"markdown": ["md"],
"yaml": ["yaml", "yml"],
"json": ["json"],
"csharp": ["cs"]
}
}

I think I can make a clone of the repo without irrelevant information

[Kurt-von-Laven]

Yeah, that would be helpful, in particular so your .gitignore can be seen. I don't see any issue with your .jscpd.json off hand. I don't think our issues are related. Also, see GitHub's guide to formatting code properly in Markdown.

[borysl]

Hello,

Sorry, was a little bit busy. Here is public repo with https://github.com/borysl/jscpd-test minimal necessary settings which lead to failure.
The previous commit borysl/jscpd-test@f024de1 will work

[Kurt-von-Laven]

Apologies, @borysl; somehow I completely missed your message. Your configuration issue was quite tricky and ultimately unrelated to the issue I originally reported. Here is a patch: borysl/jscpd-test#1. If we all agree that gitignore-to-glob fails silently with Windows line endings, I can file a bug upstream.

[soullivaneuh]

It looks like the issue is somewhat related to hidden paths.

Given the following .gitignore

/.cache/
/cache/

We have a difference of behavior we can quickly figure out simply by storing the npm cache on the root project directory:

# Produces no duplicate report from the `cache` directory
g clean -fdx && npm_config_cache=cache npx jscpd@3.5.3 --gitignore .
# Produces duplicate report from the `.cache` directory
g clean -fdx && npm_config_cache=.cache npx jscpd@3.5.3 --gitignore .

Looking at the related source code, we may assume it come from the pattern custom transformation or a bug with the used gitignore-to-glob package.

In case of second option, we may replace it by a more maintained one named ignore.

Dirty workaround that may feet some situation, using --ignore='./.cache/**/*' option alongside the --gitignore one.

[Kurt-von-Laven]

As mentioned in the original issue, the root cause is that gitignore-to-glob deliberately filters hidden files and directories out of the .gitignore. Using the ignore package instead seems like a great solution.

[FFdhorkin]

jscpd seems to not understand that .gitignore lines can be prefixed with a /, either.

For example, if I have /foo in my .gitignore, I still see dupes in the foo directory. However, if i remove the leading slash, jscpd ignores correctly (but now a file named foo or a subdirectory named foo will get ignored).