Support local plugins with is_umbrella settings

[jiahuili430]

Is it possible to read the {project_app_dirs, ["src/*"]} setting
and treat it as a proper umbrella project to use local plugins?

Issue: #2712

[ferd]

See #2697 for the feature to already work with umbrella applications.

I do not think it is a good idea to try and support it for non-umbrella projects because plugins used locally when the library is imported as a dependency are not going to resolve properly compared to external ones. As such, the decision not to support local plugins in non-umbrella projects is deliberate and not something I think we'll start doing.

[jiahuili430]

I'm trying to upgrade CouchDB from rebar to rebar3 https://github.com/apache/couchdb/pull/4089/files.
There are over 800 files and symlinks that need to be changed.
Is there a better way to handle it?

[ferd]

Oh, so specifically there, the project_app_dirs has been changed to a non-default value? What has src_dirs been changed to?

[jiahuili430]

I didn't change src_dirs.
To get rid of the dependency cycle error, I removed the following settings.

{deps_dir, "src"},
{deps, lists:map(MakeDep, DepDescs ++ OptionalDeps)},
{sub_dirs, SubDirs},
{lib_dirs, ["src"]},
{deps, []},

https://github.com/apache/couchdb/pull/4089/files#diff-a07a3d3b357a34c4acb80b2e178e9a19e6c2ed9d91cb0aa8d53cadabb1ae8bd0L190-R124

[ferd]

• deps_dir is used to specify what's the directory name once in _build/ and should have no impact here.
• libs_dir and sub_dirs don't exist or mean anything anymore.

There also are project_app_dirs and src_dirs. The former names where OTP applications are at the top-level, and the latter the name for directories containing source files within project_app_dirs. Setting the same value for project_app_dirs and src_dirs would be risky because src_dirs is for source files and is read recursively. This may cause problems where all the apps' source files may be seen as a single application? I don't know that this would be a good idea.

Specifically, in the code you're patching, the call to rebar_dir:lib_dirs() already is there to account for custom options. Making a special case that ignores non-default situations feels counter-productive and may be just papering over a weird behaviour due to your current configuration.

[jiahuili430]

Is there a better way to handle local plugins if we don't change src to apps in CouchDB?

[iilyak]

Let me chime in as well. Just for the reference here is the current PR for migrating CouchDB project into rebar3 https://github.com/apache/couchdb/pull/4089/files. It has grown too big to digest and maintain.

To reduce the change set we are looking for any advise on following issues:

• fetching dependencies in presence of cyclic definitions
  • in the proposed PR we had to go to extremes and download deps via configure
• avoid changing src to apps because it breaks lots of release related scripts

[tsloughter]

The problem with cycles isn't fetching, that is solved by rebar3 just picking the first it encounters, but we don't know what order to build apps in if there is a cycle.

[jiahuili430]

How about adding a new config option is_umbrella?
Demo: https://github.com/jiahuili430/local-plugins/tree/test

The main branch is using the apps folder without the is_umbrella setting, the ./rebar3 todo is still working.

[iilyak]

The docs for the feature in #2697 say that

plugins are installed before the discovery phase that decides
whether a project is an umbrella project or not, so we use an
imperfect heuristic.

Would it be possible to make it explicit via a new configuration option in rebar.config?

[ferd]

What should happen is is_umbrella is set to false and there is clearly an umbrella structure? What of the opposite where is_umbrella is true but the project is not an umbrella project?