Skip to content

[Bug?]: Dependency considered optional but should not be #5827

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
1 task done
mhassan1 opened this issue Oct 23, 2023 · 5 comments · Fixed by #5840
Closed
1 task done

[Bug?]: Dependency considered optional but should not be #5827

mhassan1 opened this issue Oct 23, 2023 · 5 comments · Fixed by #5840
Labels
bug Something isn't working reproducible This issue can be successfully reproduced

Comments

@mhassan1
Copy link
Contributor

mhassan1 commented Oct 23, 2023

Self-service

  • I'd be willing to implement a fix

Describe the bug

I am seeing a particular configuration of dependencies and package extensions where an explicit workspace dependency is considered optional (even though it should not be considered optional because it is an explicit workspace dependency). See the reproduction, below.

In this case, the build of the explicit dependency may fail without failing the yarn install step, even though the yarn install step should have failed.

This behavior exists in both Yarn 3 and Yarn 4.

To reproduce

// force `node-gyp` to fail to build `iconv`
process.env.NODE_GYP_FORCE_PYTHON = 'i-dont-exist'

await packageJson({
  workspaces: ['packages/p1'],
  dependencies: {
    // `iconv` is a transitive, optional dependency of `lerna`
    lerna: '^7.1.1'
  }
})

await packageJson({
  dependencies: {
    // `iconv` is an explicit dependency of a workspace
    iconv: '^2.3.0'
  }
}, { cwd: 'packages/p1' })

await yarn('config', 'set', 'packageExtensions' , '--json', JSON.stringify({
  // `encoding` is missing a dependency on `iconv`
  'encoding@^0.1': {
    dependencies: {
      iconv: '^2'
    }
  }
}))

await expect(yarn('install')).rejects.toThrow('Failed with errors')

Environment

System:
  OS: macOS 13.6
  CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
Binaries:
  Node: 18.17.0 - /private/var/folders/82/cn2lb31s5hd572v7x5wkc1vc0000gq/T/xfs-f830d5e1/node
  Yarn: 4.0.0 - /private/var/folders/82/cn2lb31s5hd572v7x5wkc1vc0000gq/T/xfs-f830d5e1/yarn
  npm: 9.6.7 - ~/.nvm/versions/node/v18.17.0/bin/npm

Additional context

  • It seems like removing any of the pieces of the reproduction causes it to no longer reproduce
  • Adding iconv as a dependency in the root package.json causes it to no longer reproduce
@mhassan1 mhassan1 added the bug Something isn't working label Oct 23, 2023
@yarnbot yarnbot added the reproducible This issue can be successfully reproduced label Oct 23, 2023
@yarnbot

This comment has been minimized.

@arcanis
Copy link
Member

arcanis commented Oct 24, 2023

Will be fixed by #5840. Since it was already a bug in 3.x I may wait a bit (~a month?) before merging the PR, as I'm worried this bugfix could otherwise be seen as a breaking change or new bug preventing migrating to the 4.0 release.

@yarnbot
Copy link
Collaborator

yarnbot commented Oct 24, 2023

This issue reproduces on master:

Error: expect(received).rejects.toThrow()

Received promise resolved instead of rejected
Resolved to value: "➤ YN0000: · Yarn 4.0.0-dev
➤ YN0000: ┌ Resolution step
::group::Resolution step
➤ YN0085: │ + iconv@npm:2.3.5, lerna@npm:7.4.1, @babel/code-frame@npm:7.22.13, @babel/helper-validator-identifier@npm:7.22.20, @babel/highlight@npm:7.22.20, and 535 more.
::endgroup::
➤ YN0000: └ Completed in 4s 805ms
➤ YN0000: ┌ Fetch step
::group::Fetch step
➤ YN0013: │ 531 packages were added to the project (+ 153.42 MiB).
::endgroup::
➤ YN0000: └ Completed in 7s 410ms
➤ YN0000: ┌ Link step
::group::Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0007: │ iconv@npm:2.3.5 must be built because it never has been before or the last one failed
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp info it worked if it ends with ok
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp info using [email protected]
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp info using [email protected] | linux | x64
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python 
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python checking Python explicitly set from NODE_GYP_FORCE_PYTHON
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python - process.env.NODE_GYP_FORCE_PYTHON is \"i-dont-exist\"
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python - \"i-dont-exist\" is not in PATH or produced an error
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python 
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python **********************************************************
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python You need to install the latest version of Python.
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python you can try one of the following options:
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python - Use the switch --python=\"/path/to/pythonexecutable\"
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python   (accepted by both node-gyp and npm)
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python - Set the environment variable PYTHON
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python - Set the npm configuration variable python:
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python   npm config set python \"/path/to/pythonexecutable\"
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python For more information consult the documentation at:
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python **********************************************************
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! find Python 
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! configure error 
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! stack Error: Could not find any Python installation to use
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! stack     at PythonFinder.fail (/tmp/tmp-17cx8baoUcJbSU/.yarn/unplugged/node-gyp-npm-9.4.0-ebf5f5573e/node_modules/node-gyp/lib/find-python.js:330:47)
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! stack     at PythonFinder.runChecks (/tmp/tmp-17cx8baoUcJbSU/.yarn/unplugged/node-gyp-npm-9.4.0-ebf5f5573e/node_modules/node-gyp/lib/find-python.js:159:21)
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! stack     at PythonFinder.<anonymous> (/tmp/tmp-17cx8baoUcJbSU/.yarn/unplugged/node-gyp-npm-9.4.0-ebf5f5573e/node_modules/node-gyp/lib/find-python.js:202:16)
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! stack     at PythonFinder.execFileCallback (/tmp/tmp-17cx8baoUcJbSU/.yarn/unplugged/node-gyp-npm-9.4.0-ebf5f5573e/node_modules/node-gyp/lib/find-python.js:294:16)
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! stack     at exithandler (node:child_process:430:5)
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! stack     at ChildProcess.errorhandler (node:child_process:442:5)
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! stack     at ChildProcess.emit (node:events:517:28)
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:290:12)
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! stack     at onErrorNT (node:internal/child_process:477:16)
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! stack     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! System Linux 6.2.0-1015-azure
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! command \"/usr/bin/node\" \"/tmp/tmp-17cx8baoUcJbSU/.yarn/unplugged/node-gyp-npm-9.4.0-ebf5f5573e/node_modules/node-gyp/bin/node-gyp.js\" \"rebuild\"
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! cwd /tmp/tmp-17cx8baoUcJbSU/.yarn/unplugged/iconv-npm-2.3.5-37288146ac/node_modules/iconv
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! node -v v18.18.2
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! node-gyp -v v9.4.0
➤ YN0000: │ iconv@npm:2.3.5 STDERR gyp ERR! not ok 
➤ YN0009: │ iconv@npm:2.3.5 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-f22b048b/build.log)
➤ YN0007: │ @parcel/watcher@npm:2.0.4 must be built because it never has been before or the last one failed
➤ YN0007: │ nx@npm:16.10.0 [aedfc] must be built because it never has been before or the last one failed
::endgroup::
::group::/tmp/xfs-f22b048b/build.log
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | linux | x64
gyp ERR! find Python 
gyp ERR! find Python checking Python explicitly set from NODE_GYP_FORCE_PYTHON
gyp ERR! find Python - process.env.NODE_GYP_FORCE_PYTHON is \"i-dont-exist\"
gyp ERR! find Python - \"i-dont-exist\" is not in PATH or produced an error
gyp ERR! find Python 
gyp ERR! find Python **********************************************************
gyp ERR! find Python You need to install the latest version of Python.
gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
gyp ERR! find Python you can try one of the following options:
gyp ERR! find Python - Use the switch --python=\"/path/to/pythonexecutable\"
gyp ERR! find Python   (accepted by both node-gyp and npm)
gyp ERR! find Python - Set the environment variable PYTHON
gyp ERR! find Python - Set the npm configuration variable python:
gyp ERR! find Python   npm config set python \"/path/to/pythonexecutable\"
gyp ERR! find Python For more information consult the documentation at:
gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
gyp ERR! find Python **********************************************************
gyp ERR! find Python 
gyp ERR! configure error 
gyp ERR! stack Error: Could not find any Python installation to use
gyp ERR! stack     at PythonFinder.fail (/tmp/tmp-17cx8baoUcJbSU/.yarn/unplugged/node-gyp-npm-9.4.0-ebf5f5573e/node_modules/node-gyp/lib/find-python.js:330:47)
gyp ERR! stack     at PythonFinder.runChecks (/tmp/tmp-17cx8baoUcJbSU/.yarn/unplugged/node-gyp-npm-9.4.0-ebf5f5573e/node_modules/node-gyp/lib/find-python.js:159:21)
gyp ERR! stack     at PythonFinder.<anonymous> (/tmp/tmp-17cx8baoUcJbSU/.yarn/unplugged/node-gyp-npm-9.4.0-ebf5f5573e/node_modules/node-gyp/lib/find-python.js:202:16)
gyp ERR! stack     at PythonFinder.execFileCallback (/tmp/tmp-17cx8baoUcJbSU/.yarn/unplugged/node-gyp-npm-9.4.0-ebf5f5573e/node_modules/node-gyp/lib/find-python.js:294:16)
gyp ERR! stack     at exithandler (node:child_process:430:5)
gyp ERR! stack     at ChildProcess.errorhandler (node:child_process:442:5)
gyp ERR! stack     at ChildProcess.emit (node:events:517:28)
gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:290:12)
gyp ERR! stack     at onErrorNT (node:internal/child_process:477:16)
gyp ERR! stack     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
gyp ERR! System Linux 6.2.0-1015-azure
gyp ERR! command \"/usr/bin/node\" \"/tmp/tmp-17cx8baoUcJbSU/.yarn/unplugged/node-gyp-npm-9.4.0-ebf5f5573e/node_modules/node-gyp/bin/node-gyp.js\" \"rebuild\"
gyp ERR! cwd /tmp/tmp-17cx8baoUcJbSU/.yarn/unplugged/iconv-npm-2.3.5-37288146ac/node_modules/iconv
gyp ERR! node -v v18.18.2
gyp ERR! node-gyp -v v9.4.0
gyp ERR! not ok 
::endgroup::
➤ YN0000: └ Completed in 2s 641ms
➤ YN0000: · Done with warnings in 15s 19ms
"
    at expect (/github/workspace/.yarn/cache/expect-npm-24.8.0-8c7640c562-9b3e8262b0.zip/node_modules/expect/build/index.js:138:15)
    at module.exports (evalmachine.<anonymous>:29:7)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.3-558f52b79f-69609f7d06.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:57:13
    at async executeInTempDirectory (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.3-558f52b79f-69609f7d06.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:18:16)
    at async executeRepro (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.3-558f52b79f-69609f7d06.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:25:12)
    at async ExecCommand.execute (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.3-558f52b79f-69609f7d06.zip/node_modules/@arcanis/sherlock/lib/commands/exec.js:26:38)
    at async ExecCommand.validateAndExecute (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-312910c71e.zip/node_modules/clipanion/lib/advanced/Command.js:161:26)
    at async Cli.run (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-312910c71e.zip/node_modules/clipanion/lib/advanced/Cli.js:74:24)
    at async Cli.runExit (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-312910c71e.zip/node_modules/clipanion/lib/advanced/Cli.js:83:28)

@mhassan1
Copy link
Contributor Author

mhassan1 commented Jan 4, 2024

@arcanis Do we think it is safe to move ahead with the attached PR?

@jmpage
Copy link

jmpage commented Jan 19, 2024

@arcanis, re: potential breaking changes

I recently encountered a production issue where this bug allowed a build to continue even though unix-dgram failed to build because it was an optional dependency of hot-shots. We had explicitly required unix-dgram as a dependency as well with the understanding that this would cause yarn install to exit with a non-zero status if it failed to build. This caused a service to fail to report metrics.

I would consider a bugfix which causes a new version of yarn 3.x to exit with a non-zero status in this scenario to be a welcome change as this behavior is undocumented and it contravenes a user's expectations. If, upon bumping to a new version of yarn, a user encounters a change in behavior, they will learn that their project was previously built without a non-optional dependency and they can decide to modify it if this is intended. This may help people discover situations where their projects are not behaving as intended.

I understand if there are other concerns which you have regarding unwelcome disruptions caused by this bugfix.

arcanis added a commit that referenced this issue Jan 22, 2024
…#5840)

**What's the problem this PR addresses?**

Depending on the order we traversed the dependency tree, it could happen
that we'd see a package as optional before seeing the other cases where
it was not optional. Because marking the package as "not optional"
occurred after the "has this package been traversed before?" check, we
were never updating its status from "optional" to "not optional".

Fixes #5827

**How did you fix it?**

We now check for optionality regardless of whether the package has been
seen or not before. I added an order-dependent dragon test to try to
prevent regressions.

**Checklist**
<!--- Don't worry if you miss something, chores are automatically
tested. -->
<!--- This checklist exists to help you remember doing the chores when
you submit a PR. -->
<!--- Put an `x` in all the boxes that apply. -->
- [x] I have read the [Contributing
Guide](https://yarnpkg.com/advanced/contributing).

<!-- See
https://yarnpkg.com/advanced/contributing#preparing-your-pr-to-be-released
for more details. -->
<!-- Check with `yarn version check` and fix with `yarn version check
-i` -->
- [x] I have set the packages that need to be released for my changes to
be effective.

<!-- The "Testing chores" workflow validates that your PR follows our
guidelines. -->
<!-- If it doesn't pass, click on it to see details as to what your PR
might be missing. -->
- [x] I will check that all automated PR checks pass before the PR gets
reviewed.
merceyz pushed a commit that referenced this issue Jan 30, 2024
…#5840)

**What's the problem this PR addresses?**

Depending on the order we traversed the dependency tree, it could happen
that we'd see a package as optional before seeing the other cases where
it was not optional. Because marking the package as "not optional"
occurred after the "has this package been traversed before?" check, we
were never updating its status from "optional" to "not optional".

Fixes #5827

**How did you fix it?**

We now check for optionality regardless of whether the package has been
seen or not before. I added an order-dependent dragon test to try to
prevent regressions.

**Checklist**
<!--- Don't worry if you miss something, chores are automatically
tested. -->
<!--- This checklist exists to help you remember doing the chores when
you submit a PR. -->
<!--- Put an `x` in all the boxes that apply. -->
- [x] I have read the [Contributing
Guide](https://yarnpkg.com/advanced/contributing).

<!-- See
https://yarnpkg.com/advanced/contributing#preparing-your-pr-to-be-released
for more details. -->
<!-- Check with `yarn version check` and fix with `yarn version check
-i` -->
- [x] I have set the packages that need to be released for my changes to
be effective.

<!-- The "Testing chores" workflow validates that your PR follows our
guidelines. -->
<!-- If it doesn't pass, click on it to see details as to what your PR
might be missing. -->
- [x] I will check that all automated PR checks pass before the PR gets
reviewed.

(cherry picked from commit 7aa2359)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working reproducible This issue can be successfully reproduced
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants