worker.rollupOptions is ignored if set in a plugin

[Rich-Harris]

Describe the bug

If I set worker.rollupOptions in my vite.config.js I can control where workers are written to, but if it's set by a plugin, the configuration is discarded. This makes it hard for frameworks like SvelteKit to write them to a suitable location.

Reproduction

https://github.com/Rich-Harris/vite-workers-repro

System Info

System:
    OS: macOS 12.0.1
    CPU: (10) arm64 Apple M1 Max
    Memory: 198.39 MB / 32.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.15.1 - ~/.nvm/versions/node/v16.15.1/bin/node
    Yarn: 1.22.19 - ~/.nvm/versions/node/v16.15.1/bin/yarn
    npm: 8.11.0 - ~/.nvm/versions/node/v16.15.1/bin/npm
  Browsers:
    Chrome: 103.0.5060.114
    Chrome Canary: 105.0.5187.0
    Firefox: 102.0.1
    Safari: 15.1
  npmPackages:
    vite: ^3.0.2 => 3.0.2

Used Package Manager

npm

Logs

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.