Install via conda #2554

mvdbeek · 2016-06-26T14:38:19Z

This is @bwlang and my project for the gcc hackathon (also see bxlab#39).
The actual place where we're doing the install still needs to move,
we should somehow show the requirement resolution to the admin in the UI
and the API could use a little reworking.

mvdbeek · 2016-06-26T21:49:51Z

@bwlang might be a good idea to also include/review https://github.com/galaxyproject/galaxy/pull/2538/files

bwlang · 2016-06-27T01:32:18Z

I think this is now ready for some feedback.
In my hands, it installs tools via conda.

Todo: give the user some feedback about whether the installation was successful.
not sure if we should try to do that on the tool installation status page, or on a dedicated requirements page. I see advantages to both approaches.

bwlang · 2016-06-29T23:19:16Z

@mvdbeek what do you think...
ready to remove the WIP tag?

bgruening · 2016-06-30T17:16:52Z

templates/webapps/tool_shed/repository/common.mako

            </table>
        </td>
    </tr>
 </%def>

-<%def name="render_resolver_dependency_items( resolver_dependencies )">
+<%def name="render_resolver_dependencies    ( resolver_dependencies )">


Can we remove the spaces?

mvdbeek · 2016-07-02T09:57:33Z

There are still a few things to sort out:

When the dependency is resolved with the versionless resolver, there should be a warning symbol (it's green right now. We should probably also differentiate between requested requriement version and resolved requirement version)
- There is the "CondaDepenency" typo in lib/, which is carried on into the UI
- Toolshed packages are not shown as resolved (I think that may be in the nature of the toolshed resolver, not sure a "fix" would be simple)
- We may be breaking the (display) of other resolver modules, we should at least test that those are working
- A set of tests would probably be a good idea.
- Return the resolution status through the API (Install via conda #2554 (comment))

We also discussed an overview page of all requirements ("Manage tool requirements" ?) with @jmchilton , so that the admin can see all requirements of all tools, the resolver status of these and a list of tools/repositories that depend on these requirements, but I think that would be another PR.

nsoranzo · 2016-07-02T15:08:11Z

lib/galaxy/tools/__init__.py

+            for requirement in tool.requirements:
+                if requirement.type == 'package':
+                    reqs.append(json.dumps(requirement.to_dict()))
+        reqs = [json.loads(req) for req in list(set(reqs))]


No need to cast to list.

jmchilton · 2016-07-22T14:59:34Z

This is amazing - what is left to do? Is there some chance to get some of this into 15.07?

mvdbeek · 2016-07-22T15:16:00Z

Just need to fix the failing tests and rebase... Other than that it's pretty complete by now.

working: - fetching valid tool list - identifying unique requirements not working: File '/Users/langhorst/src/galaxy_hackathon/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 1138 in prepare_for_install [view.manager_dependency(req) for req in uniq_reqs.values()] TypeError: manager_dependency() takes exactly 1 argument (2 given)

@bwlang

(ping @bwlang)

nsoranzo · 2016-07-27T18:47:57Z

lib/galaxy/tools/deps/resolvers/unlinked_tool_shed_packages.py

@@ -81,9 +81,9 @@ def _find_possible_depenencies(self, name, version, type):
                            for revision in listdir(package_path):
                                revision_path = join(package_path, revision)
                                package = self._galaxy_package_dep(revision_path, version, True)


package = self._galaxy_package_dep(revision_path, version, name, True)

nsoranzo · 2016-07-27T18:49:51Z

lib/galaxy/tools/deps/resolvers/unlinked_tool_shed_packages.py


    # Finds all possible dependency to use
    # Should be extended as required
-    # Returns CandidateDepenency objects with data for preference picking
+    # Returns CandidateDependency objects with data for preference picking
    def _find_possible_depenencies(self, name, version, type):


s/depenencies/dependencies/

jmchilton · 2016-07-27T18:51:09Z

I'd like to rework some of the user-facing GUI aspects of this:

Currently the wording is something like:

Install resolver dependencies
Install repository dependencies

I'd like it to be something more like:

When available, install externally managed dependencies (e.g. conda) _New_
When available, install tool shed managed dependencies
Is this reasonable?
Should we consider this beta? If so should we move the conda stuff after the shed stuff? The argument for it being considered beta is that the logging of the install process I think is a bit worse than that of the shed dependencies right?
I want to get the conda checkbox to show up even if there is no tool_dependencies.xml file available. I'm going to take a crack at this.

martenson · 2016-07-27T19:09:56Z

The UI does seem a bit confusing, mostly because it has been confusing before.

Is there a reasonable scenario where you want to get both Conda and TS dependency installations? I feel like a radiobutton would be a better UI choice here. I also agree with @jmchilton that we should reword the text descriptions (like the 'un-check to skip' language)

Should we consider this beta? If so should we move the conda stuff after the shed stuff?

Yes. Yes.

jmchilton · 2016-07-27T19:13:53Z

@martenson I don't think the admin has a way of really knowing which they want - and for a large suite the answer is probably going to be both for a while.

I've been playing around with the biom stuff - where there is no tool_dependencies.xml and I do think with a small patch I have - things are more clear than they have even been.

…able. Also reword different install options, I hope this is more clear - I do think it exposes les of the internal language to the end user.

martenson · 2016-07-27T19:17:40Z

Also this builds on the older UI for tool installations. Would be nice to have this merged into the currently Beta API installation UI - #1392

The schedule of such is a bit tricky, I guess we continue with this PR as is and hope to port the changes to the new UI in 16.10?

nsoranzo · 2016-07-27T19:26:17Z

Thanks @mvdbeek for fixing all my nitpicking! I leave it to @martenson and @jmchilton for more review, testing and hopefully merging!

…ia_conda

jmchilton · 2016-07-27T19:29:44Z

Opened a PR with some of the changes @martenson and I mentioned here: bxlab#56.

UI Tweaks

jmchilton · 2016-07-27T19:35:07Z

I'm getting an itchy merge finger ... and other important requests before this merged? I think we can make clarifications after the merge as usability bug fixes.

mvdbeek · 2016-07-27T19:45:36Z

Awesome, thanks everyone!

mvdbeek · 2016-07-27T20:05:34Z

Also this builds on the older UI for tool installations. Would be nice to have this merged into the currently Beta API installation UI - #1392

This should work for new and old, we'll just have to change the wording to that of bxlab#56

Fix broken toolshed tests with #2554

Fix unit test broken in galaxyproject#2554

This option was introduced with galaxyproject#2554 and galaxyproject#2538, in reponse to a false positive conda install. This would have been caught by simply checking the exit code, which we are doing by default now. The additional verification is too strigent and not in line with the conda resolver. Activating this option will report perfectly well installed conda environments as having failed and will cause them to be uninstalled.

mvdbeek added kind/enhancement area/UI-UX area/toolshed area/tools area/admin labels Jun 26, 2016

mvdbeek added this to the 16.07 milestone Jun 26, 2016

galaxybot added the status/WIP label Jun 26, 2016

bgruening reviewed Jun 30, 2016
View reviewed changes

nsoranzo reviewed Jul 2, 2016
View reviewed changes

mvdbeek force-pushed the install_via_conda branch 2 times, most recently from 90acf71 to 7db313b Compare July 21, 2016 16:19

bwlang and others added 4 commits July 22, 2016 21:55

Need to unnpack kwargs (ping @bwlang)

dcb57a6

Move get_unique_requirements to shed_util_common

de6bbf1

(ping @bwlang)

Iterate over all tools in a repository

179d02c

nsoranzo reviewed Jul 27, 2016
View reviewed changes

Simplify conda install logic

2a5b043

nsoranzo reviewed Jul 27, 2016
View reviewed changes

nsoranzo added kind/feature and removed kind/enhancement labels Jul 27, 2016

Show conda install option even when no tool_dependencies.xml is avail…

8cca7be

…able. Also reword different install options, I hope this is more clear - I do think it exposes les of the internal language to the end user.

Address @nsoranzo's review

483b66b

jmchilton added 2 commits July 27, 2016 15:26

Mark conda install beta and move after TS options.

924a913

Merge remote-tracking branch 'bxlab/install_via_conda' into install_v…

88fac71

…ia_conda

Merge pull request #56 from jmchilton/install_via_conda

12209ab

UI Tweaks

blankenberg merged commit a731ce2 into galaxyproject:dev Jul 27, 2016

mvdbeek mentioned this pull request Jul 28, 2016

Fix an error with getting the size of an object from Azure. Attempte… #2678

Merged

dannon added a commit that referenced this pull request Jul 28, 2016

Merge pull request #2680 from mvdbeek/toolshed_test_fixes

fed3665

Fix broken toolshed tests with #2554

nsoranzo added a commit to nsoranzo/galaxy that referenced this pull request Jul 29, 2016

Update INDETERMINATE_DEPENDENCY to NullDependency

f06c895

Fix unit test broken in galaxyproject#2554

mvdbeek mentioned this pull request Jul 30, 2016

Add install_resolver_dependencies keyword to install_repository_revision galaxyproject/bioblend#190

Merged

nsoranzo mentioned this pull request Aug 3, 2016

Trinity dependencies broken in the main toolshed galaxyproject/tools-iuc#905

Closed

mvdbeek mentioned this pull request Aug 9, 2016

New release (with 16.07?) galaxyproject/bioblend#193

Closed

afgane mentioned this pull request Aug 22, 2016

[16.07] curated release notes for 16.07 #2821

Merged

mvdbeek mentioned this pull request Nov 4, 2016

[16.07] Remove verbose_install_check option #3127

Merged

Install via conda #2554

Install via conda #2554

Uh oh!

Conversation

mvdbeek commented Jun 26, 2016

Uh oh!

mvdbeek commented Jun 26, 2016

Uh oh!

bwlang commented Jun 27, 2016

Uh oh!

bwlang commented Jun 29, 2016

Uh oh!

bgruening Jun 30, 2016

Choose a reason for hiding this comment

Uh oh!

mvdbeek commented Jul 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nsoranzo Jul 2, 2016

Choose a reason for hiding this comment

Uh oh!

jmchilton commented Jul 22, 2016

Uh oh!

mvdbeek commented Jul 22, 2016

Uh oh!

nsoranzo Jul 27, 2016

Choose a reason for hiding this comment

Uh oh!

nsoranzo Jul 27, 2016

Choose a reason for hiding this comment

Uh oh!

jmchilton commented Jul 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martenson commented Jul 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmchilton commented Jul 27, 2016

Uh oh!

martenson commented Jul 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nsoranzo commented Jul 27, 2016

Uh oh!

jmchilton commented Jul 27, 2016

Uh oh!

jmchilton commented Jul 27, 2016

Uh oh!

mvdbeek commented Jul 27, 2016

Uh oh!

mvdbeek commented Jul 27, 2016

Uh oh!

Uh oh!

mvdbeek commented Jul 2, 2016 •

edited

Loading

jmchilton commented Jul 27, 2016 •

edited

Loading

martenson commented Jul 27, 2016 •

edited

Loading

martenson commented Jul 27, 2016 •

edited

Loading