OBO Dashboard stopped working within reasonable time and memory constraints

[matentzn]

We have a bit of a problem; for some reason, we cannot run the OBO dashboard anymore; NCIT and friends of similar size stopped being processable. Before ROBOT 1.9.0, it was still possible to run OBO Dashboard with 48GB or memory, and a timeout of 2 hours per ontology. Now, we cannot run it at all anymore (not even with 55 GB memory and 4 hours timeout).

INFO:root:Creating dashboard for ncit...
INFO:root:RUNNING: make  ROBOT_JAR=/tools/robot.jar ROBOT=robot  dashboard/ncit/dashboard.html (Timeout: 14400)
Terminating process

Its the timeout that kicks in, not an out of memory error; but 4 hours is already a lot for an ontology.

I will try now with 56 GB memory which is the absolute maximum I have on my machine, and an 8 hour timeout.

Any ideas of how we can deal with this is very appreciated.

PS: my guess is the problem is ROBOT report, but it could also be ROBOT reason - I cannot see it in the log unfortunately.

[cthoyt]

I suggest skipping the big ones in the meantime. NCIT, NCBITaxon, and ChEBI aren't curated/created in a normal way anyway

[erik-whiting]

@matentzn is this reproducible by running make ncit or something similar locally?

[erik-whiting]

by the way, what's the impact? http://dashboard.obofoundry.org/dashboard/index.html looks fine?

[erik-whiting]

Also, ROBOT 1.9.1 added the code in this PR, some of the comments mention a performance hit. I wonder if that's why you're experiencing problems

[matentzn]

Yea, it failed again, with the increased memory and time constraints.

[matentzn]

@erik-whiting we cannot re-run it at the moment (update)