Tool UI Unusably and Unreasonably Slow

[blankenberg]

Have History with only 68 History Items (has Collections that are large [8x626], but have been copied from other histories without copying the hidden/underlying datasets). Clicking rerun on a tool (Select/Grep, but doesn't much matter which tool) takes 1.2 Minutes to display tool interface. This should be nearly instantaneous.

Similar speeds when loading tool without rerun:

Most likely this is due to inefficiencies within the dataset/collection matcher logic.

[mvdbeek]

I think @bgruening had a similar problem recently, that resulted in #3624 and #3625. Maybe the slow query logging would be a good starting point here.

[bgruening]

Oh yes, I had this as well. But mine was also for every history entry not just collections.
Slow query logging is so useful, please enable it and see which query takes so long.

[jmchilton]

This should be nearly instantaneous.

Not copying the original datasets doesn't mean that Galaxy isn't going to try to check permissions on all of the datasets before showing them to you. You are checking thousands of dataset permissions in this operation. I think we can optimize that down from over a minute - but it is going to take a few seconds unless we change the security model.

Update: In fact given the way we cache datasets roles and such for datasets in the history - I wouldn't be surprised if you are killing your performance by making Galaxy fetch datasets from other histories.

[blankenberg]

Should it even be checking the dataset collections before I click the widget to switch over to a collection mode? At this point in the analysis I'm not actually interested in those collections, as I am working on a single file table (that was distilled down from the collection).

[jmchilton]

Should it even be checking the dataset collections before I click the widget to switch over to a collection mode?

Definitely not - this form needs to be more dynamic for sure.

[guerler]

The list should be provided on the first request, however in either case we need to approach the underlying problem of resolving the available collection ids. Dataset permissions are not checked when matching regular datasets since #2766 until the job has been executed. This improved the tool form load performance by about 60%. Can we do something similar for collections?

[jmchilton]

@guerler I can look into the permission thing - but it will be a constant slow down per dataset the same way since the datatype still needs to be inspected for all the datasets - the operation is still O(n) on the number of datasets in collections. It would be nice to be able to load the form quickly and then progressively enhance it as more information becomes available - the current setup seems inherently unscalable - it isn't just a problem with large collections but large histories too.

[guerler]

Sounds good, please do that or let know me if I can help. My suggestion is to provide the list of collections without or only minimal checks when parsing them to the front-end on initial form load and then evaluate full compatibility and permissions on job execution. How about just checking a subset of datatypes in the collection? As a follow-up, I can work on revising the front-end by lazy-loading datasets through multiple requests to further improve the performance but if few collections can cause such a delay lazy-loading will not help.

[jmchilton]

I have WIP here https://github.com/jmchilton/galaxy/tree/collection_performance. I'm not seeing any immediate significant performance benefit of disabling security - but I'm going to try with larger collections and deeper collections.

I will consider disabling some checks on collections but I'm skeptical (security is fine - but datatype or job state). I don't want people to see collections with the wrong types or collections with a few failed datasets in the drop down. That seems like a poor user experience - but so is one minute load times. I have some performance testing work in that branch - I'll see if I can replicate the poor performance @blankenberg is seeing first and then provide numbers - if we can determine the source of the slow down it should provide a guide for what to optimize.