[workspace] Packages within a workspace should not need to be in the workspace folder

[MBraedley-CN]

What is your suggestion?

Related to #18263

What

Using the conan workspace add/remove commands should allow relative paths above the current workspace folder, for instance I should be able to add a package whose folder is on the same level as my workspace by running conan workspace add ..\some_package or even a package whose folder is one level higher than my workspace by running conan workspace add ..\..\some_package.

Why

The current design path of workspaces suggests that an intended use case is to have a git repo for a fixed workspace which includes the packages for that workspace as submodules. This is good, I like the idea, but it doesn't cover common use cases for us, which include creating ad-hoc workspaces on a related subset of our (400+) repos, or quickly adding/removing a package to/from an existing workspace.

We already have tooling around creating virtual environments and adding git worktrees to those venvs, which also seems to be an intended use case, and in practice this works fine, but I wouldn't say it works great. It means the venv folder becomes the workspace folder, which tends to clutter things up. Being able to have a separate folder for the workspace within the venv would be preferable (and without having to manually edit the workspace files).

In a similar vein, having a single workspace for my day-to-day projects folder is not feasible, I currently have several (sometimes overlapping) Conan 1 workspaces for working on different areas of our codebase. Again, I believe I can do this with the current implementation in Conan 2, but it means editing the workspace files by hand or writing scripts to do that for us instead of using the built in commands, which would be preferable.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Thanks for the feedback @MBraedley-CN

If I understand correctly, what you are suggesting is enabling a possible layout of the workspace file such as:

|- mylib1
|       |- CMakeLists.txt
|       |- ...
|- mylib2
|       |- CMakeLists.txt
|       |- ...
|- mywsfolder
|       |- conanws.py
|       |- CMakeLists.txt (the super-build)?

I'd like to clarify one thing about the behavior of all Conan commands. When user executes a

cd mylib2
conan install .

Conan starts to scan the folders, going to the parent folder, looking for a conanws.py/conanws.yml file. If it is found, it means that the folder is in a workspace, that workspace is loaded, and it will define automatically on the fly editables such as mylib1/version and others.

If conan install cannot find the conanws.py because it is in a sibling folder, then conan install will know nothing about the current workspace, and will look for mylib1/version in the cache instead.

The alternative would be to add a --workspace-file=<path> to practically all Conan commands, but this seems a bit too non-ergonomic.
How do you envision the UX for such layout? Maybe adopting some strong convention over the mywsfolder folder name, so if we force it to be named something like conanws_xxx, then it would be possible to be found? But what if you have several of them as you say in parallel, which one should be used?

[MBraedley-CN]

That use case hasn't been a consideration for us, if we're using a workspace, then we want to be doing a conan install on the entire workspace, so conan workspace install. If we do a conan install on an individual package, we know we'd be overriding any workspace settings and using packages that are either in the cache or explicitly set as editable instead of the packages that are implicitly editable through the workspace, and that we might be messing up the toolchain files for a workspace super build.

Part of the difference in our visions for how this should work probably stems from how @AneekCN and I have been using workspaces in Conan 1 over the past few years, and that we have invested a not insignificant amount of time and tooling to further improve our workflows around using workspaces. The whole idea for us has been making it easy for us and our fellow developers to do a one button build of a set of plugins and each of the applications in our framework, and then run and debug that build. (Note that this currently requires us to generate and copy hard-links to our plugins based on if not self.in_local_cache, but that is a discussion for a later time I think.)

Ultimately, we would like to use the Conan 2 version of workspaces for the same purpose, even if it means we still have to add some tooling around it in order to create a CMakeLists.txt (or perhaps a cn_ws.cmake file) that can be used to open/edit/build the entire workspace within Visual Studio.

So, the workflow for a fresh workspace might be as follows:

conan workspace create my_ws  # create an empty workspace
cd my_ws
conan workspace add --packages liba, libb --product app1  # add packages to the workspace, ./ is assumed
conan workspace install  # performs a conan install for all workspace packages
                         # and potentially generates CMakeLists.txt for a super build
cmake --preset default  # optional for IDEs that have a CMake integration like Visual Studio

at which point editing, compiling, and debugging the entire workspace would be possible.

[memsharded]

Thanks very much for the feedback.

I think it can be a reasonable trade-off, as long as users understand that such a layout will not automatically activate the workspace unless they move to the folder or reference the folder, it will be good.

Lets try to make "sibling" workspace definitions possible.