[question] How to properly use "core.sources:exclude_urls"?

[Nekto89]

What is your question?

Hi,

I currently try to switch to recommended way of backing up sources with conan.
I've added these lines to my global.conf

core.sources:download_urls=["https://myserver/artifactory/thirdparty_public-generic-dev-local/backup-sources/", "origin"]
core.sources:download_cache=D:/conan2_sources_backup
core.sources:exclude_urls=["https://myserver/artifactory/"]
core.sources:upload_url=https://myserver/artifactory/thirdparty_public-generic-dev-local/backup-sources/

Sources are cached locally and uploaded to server. But for some reason exclude doesn't work. I have recipes with 2 urls inside. For example, for xsimd I have additional URL from old way of mirroring:

  - "https://myserver/artifactory/thirdparty_public-generic-dev-local/xsimd/13.0.0.tar.gz"
  - "https://github.com/xtensor-stack/xsimd/archive/13.0.0.tar.gz"

I would expect that in this case conan won't mirror xsimd sources because first url starts with "https://myserver/artifactory/". But it still uploads sources. Am I missing something in syntax?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[AbrilRBS]

@Nekto89 hi, thanks for the report.

Currently, all URLs must match at least one of the exclusion URLs for the whole source not to be backed up, but it's true that for cases like yours, that helps little. I'll check with the team if we modifying this behaviour would be worth it, but meanwhile, one question:

If you're already manually mirroring the sources for this reference, why keep the original sources? In case a developer does not have access to the myserver one?

[Nekto89]

If you're already manually mirroring the sources for this reference, why keep the original sources? In case a developer does not have access to the myserver one?

I would like to get rid of this manual mirroring but there will be overlap for some time until old versions with conandata.yml customizations would be gone naturally. If I understood correctly, Artifactory is smart enough to optimize storage and doesn't create copy of file internally if hashes are the same. But still it would be "prettier" not to upload already existing file.
So do I understand correctly, that current implementation of "exclude_urls" is intended just for filtering out sources for internal packages that aren't mirrored from outside?