Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(config): managerFilePatterns #34615

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 16 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/development/new-package-manager-template.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,14 @@ Explain how this package manager is different from existing ones.

### What kind of package files, and names, does this package manager use?

### Which [`fileMatch`](../usage/configuration-options.md#filematch) pattern(s) should Renovate use?
### Which [`managerFilePatterns`](../usage/configuration-options.md#managerFilePatterns) pattern(s) should Renovate use?

### Do many users need to extend the [`fileMatch`](../usage/configuration-options.md#filematch) pattern for custom file names?
### Do many users need to extend the [`managerFilePatterns`](../usage/configuration-options.md#managerFilePatterns) pattern for custom file names?

- [ ] Yes, provide details.
- [ ] No.

### Is the [`fileMatch`](../usage/configuration-options.md#filematch) pattern going to get many "false hits" for files that have nothing to do with package management?
### Is the [`managerFilePatterns`](../usage/configuration-options.md#managerFilePatterns) pattern going to get many "false hits" for files that have nothing to do with package management?

## Parsing and Extraction

Expand Down
94 changes: 52 additions & 42 deletions docs/usage/configuration-options.md
Original file line number Diff line number Diff line change
Expand Up @@ -718,9 +718,9 @@ Renovate has two custom managers:
| `regex` | Regular Expression, with named capture groups. |
| `jsonata` | JSONata query. |

To use a custom manager, you need give some information:
To use a custom manager, you must give Renovate this information:

1. `fileMatch`: name/pattern of the file to extract deps from
1. `managerFilePatterns`: regex/glob pattern of the file to extract deps from
1. `matchStrings`: `regex` patterns or `jsonata` queries used to process the file

The `matchStrings` must capture/extract the following three fields:
Expand Down Expand Up @@ -762,7 +762,7 @@ image: my.old.registry/aRepository/andImage:1.18-alpine
"customManagers": [
{
"customType": "regex",
"fileMatch": ["values.yaml$"],
"managerFilePatterns": ["/values.yaml$/"],
"matchStrings": [
"image:\\s+(?<depName>my\\.old\\.registry/aRepository/andImage):(?<currentValue>[^\\s]+)"
],
Expand Down Expand Up @@ -801,7 +801,7 @@ Example:
"customManagers": [
{
"customType": "regex",
"fileMatch": ["values.yaml$"],
"managerFilePatterns": ["/values.yaml$/"],
"matchStrings": [
"ENV .*?_VERSION=(?<currentValue>.*) # (?<datasource>.*?)/(?<depName>.*?)\\s"
]
Expand All @@ -816,7 +816,7 @@ Example:
{
"customType": "jsonata",
"fileFormat": "json",
"fileMatch": ["file.json"],
"managerFilePatterns": ["/file.json/"],
"matchStrings": [
"packages.{ \"depName\": package, \"currentValue\": version }"
]
Expand Down Expand Up @@ -863,7 +863,7 @@ Only the `json`, `toml` and `yaml` formats are supported.
{
"customType": "jsonata",
"fileFormat": "json",
"fileMatch": [".renovaterc"],
"managerFilePatterns": ["/.renovaterc/"],
"matchStrings": [
"packages.{ 'depName': package, 'currentValue': version }"
]
Expand All @@ -878,7 +878,7 @@ Only the `json`, `toml` and `yaml` formats are supported.
{
"customType": "jsonata",
"fileFormat": "yaml",
"fileMatch": ["file.yml"],
"managerFilePatterns": ["/file.yml/"],
"matchStrings": [
"packages.{ 'depName': package, 'currentValue': version }"
]
Expand All @@ -893,7 +893,7 @@ Only the `json`, `toml` and `yaml` formats are supported.
{
"customType": "jsonata",
"fileFormat": "toml",
"fileMatch": ["file.toml"],
"managerFilePatterns": ["/file.toml/"],
"matchStrings": [
"packages.{ 'depName': package, 'currentValue': version }"
]
Expand Down Expand Up @@ -952,7 +952,7 @@ As example the following configuration will update all three lines in the Docker
"customManagers": [
{
"customType": "regex",
"fileMatch": ["^Dockerfile$"],
"managerFilePatterns": ["/^Dockerfile$/"],
"matchStringsStrategy": "any",
"matchStrings": [
"ENV [A-Z]+_VERSION=(?<currentValue>.*) # (?<datasource>.*?)/(?<depName>.*?)(\\&versioning=(?<versioning>.*?))?\\s",
Expand Down Expand Up @@ -990,7 +990,7 @@ But the second custom manager will upgrade both definitions as its first `matchS
"customManagers": [
{
"customType": "regex",
"fileMatch": ["^example.json$"],
"managerFilePatterns": ["/^example.json$/"],
"matchStringsStrategy": "recursive",
"matchStrings": [
"\"backup\":\\s*{[^}]*}",
Expand All @@ -1000,7 +1000,7 @@ But the second custom manager will upgrade both definitions as its first `matchS
"datasourceTemplate": "docker"
},
{
"fileMatch": ["^example.json$"],
"managerFilePatterns": ["/^example.json$/"],
"matchStringsStrategy": "recursive",
"matchStrings": [
"\"test\":\\s*\\{[^}]*}",
Expand Down Expand Up @@ -1046,7 +1046,7 @@ Matched group values will be merged to form a single dependency.
"customManagers": [
{
"customType": "regex",
"fileMatch": ["^main.yml$"],
"managerFilePatterns": ["/^main.yml$/"],
"matchStringsStrategy": "combination",
"matchStrings": [
"prometheus_image:\\s*\"(?<depName>.*)\"\\s*//",
Expand All @@ -1055,7 +1055,7 @@ Matched group values will be merged to form a single dependency.
"datasourceTemplate": "docker"
},
{
"fileMatch": ["^main.yml$"],
"managerFilePatterns": ["/^main.yml$/"],
"matchStringsStrategy": "combination",
"matchStrings": [
"thanos_image:\\s*\"(?<depName>.*)\"\\s*//",
Expand Down Expand Up @@ -1501,35 +1501,6 @@ If you are running on any platform except `github.com`, you need to [configure a
We're planning improvements so that Renovate can show more changelogs.
Read [issue 14138 on GitHub](https://github.com/renovatebot/renovate/issues/14138) to get an overview of the planned work.

## fileMatch

`fileMatch` is used by Renovate to know which files in a repository to parse and extract.
`fileMatch` patterns in the user config are added to the default values and do not replace them.
The default `fileMatch` patterns cannot be removed, so if you need to include or exclude specific paths then use the `ignorePaths` or `includePaths` configuration options.

Some `fileMatch` patterns are short, like Renovate's default Go Modules `fileMatch` for example.
Here Renovate looks for _any_ `go.mod` file.
In this case you can probably keep using that default `fileMatch`.

At other times, the possible files is too vague for Renovate to have any default.
For default, Kubernetes manifests can exist in any `*.yaml` file and we don't want Renovate to parse every single YAML file in every repository just in case some of them have a Kubernetes manifest, so Renovate's default `fileMatch` for manager `kubernetes` is actually empty (`[]`) and needs the user to tell Renovate what directories/files to look in.

Finally, there are cases where Renovate's default `fileMatch` is good, but you may be using file patterns that a bot couldn't possibly guess about.
For example, Renovate's default `fileMatch` for `Dockerfile` is `['(^|/|\\.)([Dd]ocker|[Cc]ontainer)file$', '(^|/)([Dd]ocker|[Cc]ontainer)file[^/]*$']`.
This will catch files like `backend/Dockerfile`, `prefix.Dockerfile` or `Dockerfile-suffix`, but it will miss files like `ACTUALLY_A_DOCKERFILE.template`.
Because `fileMatch` is mergeable, you don't need to duplicate the defaults and could add the missing file like this:

```json
{
"dockerfile": {
"fileMatch": ["^ACTUALLY_A_DOCKERFILE\\.template$"]
}
}
```

If you configure `fileMatch` then it must be within a manager object (e.g. `dockerfile` in the above example).
The full list of supported managers can be found [here](modules/manager/index.md#supported-managers).

## filterUnavailableUsers

When this option is enabled PRs are not assigned to users that are unavailable.
Expand Down Expand Up @@ -2412,6 +2383,45 @@ Be careful with remapping `warn` or `error` messages to lower log levels, as it

Add to this object if you wish to define rules that apply only to major updates.

## managerFilePatterns

`managerFilePatterns` were formerly known as `fileMatch`, and regex-only.
`managerFilePatterns` instead supports regex or glob patterns, and any existing config containing `fileMatch` patterns will be automatically migrated.
Do not use the below guide for `fileMatch` if you are using an older version of Renovate.

`managerFilePatterns` tells Renovate which repository files to parse and extract.
`managerFilePatterns` patterns in the user config are _added_ to the default values, they do not replace the default values.

The default `managerFilePatterns` patterns can not be removed.
If you need to include, or exclude, specific paths then use the `ignorePaths` or `includePaths` configuration options.

Some `managerFilePatterns` patterns are short, like Renovate's default Go Modules `managerFilePatterns` for example.
Here Renovate looks for _any_ `go.mod` file.
In this case you can probably keep using that default `managerFilePatterns`.

At other times, the possible files is too vague for Renovate to have any default.
For example, Kubernetes manifests can exist in any `*.yaml` file.
We do not want Renovate to parse every YAML file in every repository, just in case _some_ of them have a Kubernetes manifest.
Therefore Renovate's default `managerFilePatterns` for the `kubernetes` manager is an empty array (`[]`).
Because the array is empty, you as user must tell Renovate which directories/files to check.

Finally, there are cases where Renovate's default `managerFilePatterns` is good, but you may be using file patterns that a bot couldn't possibly guess about.
For example, Renovate's default `managerFilePatterns` for `Dockerfile` is `['/(^|/|\\.)([Dd]ocker|[Cc]ontainer)file$/', '/(^|/)([Dd]ocker|[Cc]ontainer)file[^/]*$/']`.
This will catch files like `backend/Dockerfile`, `prefix.Dockerfile` or `Dockerfile-suffix`, but it will miss files like `ACTUALLY_A_DOCKERFILE.template`.
Because `managerFilePatterns` is "mergeable", you can add the missing file to the `filePattern` like this:

```json
{
"dockerfile": {
"managerFilePatterns": ["/^ACTUALLY_A_DOCKERFILE\\.template$/"]
}
}
```

You must configure `managerFilePatterns` _inside_ a manager object.
In the example above, the manager object is the `dockerfile`.
For reference, here is a [list of supported managers](modules/manager/index.md#supported-managers).

## milestone

If set to the number of an existing [GitHub milestone](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/about-milestones), Renovate will add that milestone to its PR.
Expand Down
2 changes: 1 addition & 1 deletion docs/usage/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Renovate supports upgrading dependencies in various types of Docker definition f

## How It Works

1. Renovate searches in each repository for any files matching each manager's configured `fileMatch` pattern(s)
1. Renovate searches in each repository for any files matching each manager's configured `managerFilePatterns`
1. Matching files are parsed, Renovate checks if the file(s) has any Docker image references (e.g. `FROM` lines in a `Dockerfile`)
1. If the image tag in use "looks" like a version (e.g. `myimage:1`, `myimage:1.1`, `myimage:1.1.0`, `myimage:1-onbuild`) then Renovate checks the Docker registry for upgrades (e.g. from `myimage:1.1.0` to `myimage:1.2.0`)

Expand Down
2 changes: 1 addition & 1 deletion docs/usage/getting-started/private-packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -489,7 +489,7 @@ private-package==1.2.3
```json
{
"pip-compile": {
"fileMatch": ["requirements.in"]
"managerFilePatterns": ["requirements.in"]
},
"hostRules": [
{
Expand Down
2 changes: 1 addition & 1 deletion docs/usage/java.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Renovate does not support:
- Catalog versions using `reject`, and `rejectAll` constraints
- Catalog versions using more than one of `require`, `strictly`, `prefer` in a single declaration
- Catalogs with custom names that do not end in `.toml`
- Catalogs outside the `gradle` folder whose names do not end in `.versions.toml` (unless overridden via [`fileMatch`](./configuration-options.md#filematch) configuration)
- Catalogs outside the `gradle` folder, whose names do _not_ end in `.versions.toml` (unless overridden via [`managerFilePatterns`](./configuration-options.md#managerFilePatterns) configuration)

### Gradle Plugin Support

Expand Down
28 changes: 16 additions & 12 deletions docs/usage/modules/manager/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,38 +18,42 @@ The goal of Renovate is to detect and maintain all third-party dependencies in y

### File Matching

Most managers have a default `fileMatch` array.
The `fileMatch` array has regular expression strings that match against the repository file list.
Most Renovate managers have a default `managerFilePatterns` array.
The `managerFilePatterns` array can hold a regular expression or glob pattern, that match against the repository file list.

#### Managers with no default fileMatch
#### Managers with no default managerFilePatterns

Some managers have no default `fileMatch` regular expression, because they have no filename convention that would let Renovate intelligently filter them.
In such a case, the manager will be disabled until you create a `fileMatch` regular expression, e.g. like the following:
Some managers have no default `managerFilePatterns`, because they have no filename convention that would let Renovate intelligently filter them.
If there is no default `filePattern`, the manager is disabled.
For the manager to work, you must create a `managerFilePatterns` regular expression, or glob pattern.
For example:

```json
{
"kubernetes": {
"fileMatch": ["^config/.*\\.yaml$"]
"managerFilePatterns": ["/^config/.*\\.yaml$/"]
}
}
```

#### Extending a manager's default fileMatch
#### Extending a manager's default managerFilePatterns

If the default `fileMatch` regular expression for a manager does not match against one of your relevant files, you can _extend_ the existing regular expression(s) by configuring a manager's `fileMatch` like in this example:
If the default `managerFilePatterns` for a manager does not match your file(s), you can _extend_ the pattern.
You extend the pattern by configuring the manager's `managerFilePatterns`.
For example:

```json
{
"dockerfile": {
"fileMatch": ["does-not-look-like-a-docker-file"]
"managerFilePatterns": ["does-not-look-like-a-docker-file"]
}
}
```

#### Ignoring files that match the default fileMatch
#### Ignoring files that match the default managerFilePatterns

Renovate will _extend_ the existing [`fileMatch`](../../configuration-options.md#filematch), meaning you don't need to include the default regular expressions like `Dockerfile` in your own array.
In other words, the regular expression are "additive".
Renovate will _extend_ the existing [`managerFilePatterns`](../../configuration-options.md#managerFilePatterns), meaning you don't need to include the default patterns like `Dockerfile` in your own array.
In other words, the patterns are "additive".
If a manager matches a file that you _don't_ want it to, ignore it using the [`ignorePaths`](../../configuration-options.md#ignorepaths) configuration option.
Also, if you ever find that Renovate is _not_ matching a file name that you're certain it should, check your preset config isn't the cause of it.
The `config:recommended` preset ignores common test and example directory names, for example.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ I can create a `renovate.json` configuration with the following custom manager c
"customManagers": [
{
"customType": "regex",
"fileMatch": ["(^|/)PKGBUILD$"],
"managerFilePatterns": ["/(^|/)PKGBUILD$/"],
"matchStrings": [
"pkgver=(?<currentValue>.*) # renovate: datasource=(?<datasource>.*) depName=(?<depName>.*)"
],
Expand All @@ -58,7 +58,7 @@ I can create a `renovate.json` configuration with the following custom manager c

Breaking that down:

- The `fileMatch` setting tells Renovate to look for any `PKGBUILD` files in a repository
- The `managerFilePatterns` setting tells Renovate to look for any `PKGBUILD` files in a repository
- The `matchStrings` is the regex format to extract the version, datasource, and dependency name from the `PKGBUILD`
- The `extractVersionTemplate` is to handle a “v” in front of any version number that is sometimes added to Git tags

Expand Down
16 changes: 8 additions & 8 deletions lib/config/__snapshots__/migration.spec.ts.snap
Original file line number Diff line number Diff line change
Expand Up @@ -338,19 +338,19 @@ exports[`config/migration > migrates customManagers 1`] = `
"customManagers": [
{
"customType": "regex",
"fileMatch": [
"(^|/|\\.)Dockerfile$",
"(^|/)Dockerfile[^/]*$",
"managerFilePatterns": [
"/(^|/|\\.)Dockerfile$/",
"/(^|/)Dockerfile[^/]*$/",
],
"matchStrings": [
"# renovate: datasource=(?<datasource>[a-z-]+?) depName=(?<depName>[^\\s]+?)(?: lookupName=(?<packageName>[^\\s]+?))?(?: versioning=(?<versioning>[a-z-0-9]+?))?\\s(?:ENV|ARG) .+?_VERSION="?(?<currentValue>.+?)"?\\s",
],
},
{
"customType": "regex",
"fileMatch": [
"(^|/|\\.)Dockerfile$",
"(^|/)Dockerfile[^/]*$",
"managerFilePatterns": [
"/(^|/|\\.)Dockerfile$/",
"/(^|/)Dockerfile[^/]*$/",
],
"matchStrings": [
"# renovate: datasource=(?<datasource>[a-z-]+?) depName=(?<depName>[^\\s]+?)(?: lookupName=(?<holder>[^\\s]+?))?(?: versioning=(?<versioning>[a-z-0-9]+?))?\\s(?:ENV|ARG) .+?_VERSION="?(?<currentValue>.+?)"?\\s",
Expand All @@ -365,8 +365,8 @@ exports[`config/migration > migrates gradle-lite 1`] = `
{
"gradle": {
"enabled": true,
"fileMatch": [
"foo",
"managerFilePatterns": [
"/foo/",
],
},
"packageRules": [
Expand Down
Loading
Loading