RAxML Threading Blog #67
Conversation
weshinsley
requested review from
muppi1993 and
a team
and removed request for
muppi1993
There was a problem hiding this comment.
Looks good.
The only thing I might add, which you circle around, is that people's first cut should surely always be to run as many independent jobs as possible and then start using threading where you have more cores than jobs. So if you're going to run 10,000 RaXML jobs and can only get 100 jobs on the cluster it's impossible to imagine any thread count greater than 1 being the most efficient.
content/blog/raxml_threading.md
Outdated
|
|
||
| One of the threads (if you use more than one) acts as an | ||
| administrator, which somewhat explains the lack of gain from 1 to 2 threads; | ||
| after that, from 2 to 6 threads, we're around 90% efficient. Beyond that, the |
There was a problem hiding this comment.
you might want a footnote explaining exactly what you mean here by efficient
There was a problem hiding this comment.
My maths is a bit shoddy here - will rewrite...
content/blog/raxml_threading.md
Outdated
|
|
||
| The results are a bit confusing here and there; the 10-core is surprisingly | ||
| erratic, and needs some deeper investigation. The overhead of stacking up 4 and 8 core | ||
| jobs is a bit more than we might; perhaps those jobs are using more of the node than |
There was a problem hiding this comment.
"than we might" => "than we might want"?
There was a problem hiding this comment.
Or "than we might expect"?
There was a problem hiding this comment.
thanks - "expect" was what I first had in mind - but have gone for "expect or want"
| --- | ||
| author: Wes Hinsley | ||
| date: 2022-08-23 | ||
| title: The risk of over-threading |
content/blog/raxml_threading.md
Outdated
| {{< figure src="/img/raxml_032.png" alt="RAxML with over 8 cores" >}} | ||
|
|
||
| .. and we actually start to make things slower - using 32 cores performs | ||
| comparably to using 8. There just isn't enough parallel work for all the |
There was a problem hiding this comment.
There was a problem hiding this comment.
I think looking at task manager it's still in a parallel section, but just performing much worse - it could be there's a sequential bit in there, or some kind of barrier sync... I'll mention Amdahl as well.
There was a problem hiding this comment.
(But I think Amdahl would only limit how good you can get with increasing threads - I don't think that would explain a u-turn and the overall time getting slower would it?)
There was a problem hiding this comment.
Looks good.
The only thing I might add, which you circle around, is that people's first cut should surely always be to run as many independent jobs as possible and then start using threading where you have more cores than jobs. So if you're going to run 10,000 RaXML jobs and can only get 100 jobs on the cluster it's impossible to imagine any thread count greater than 1 being the most efficient.
I probably need one more test to be sure of this - which is how 32 cores stack on a single node...
| the graph ended. Now we have more cores, so let's throw them all at the | ||
| problem... | ||
|
|
||
| {{< figure src="/img/raxml_032.png" alt="RAxML with over 8 cores" >}} |
There was a problem hiding this comment.
on graphs with time, adding "(lower is better)" to the y axis might help the reader
Co-authored-by: muppi1993 <[email protected]>
Co-authored-by: muppi1993 <[email protected]>
Co-authored-by: Rich FitzJohn <[email protected]>
content/blog/raxml_threading.md
Outdated
|
|
||
| The results are a bit confusing here and there; the 10-core is surprisingly | ||
| erratic, and needs some deeper investigation. The overhead of stacking up 4 and 8 core | ||
| jobs is a bit more than we might; perhaps those jobs are using more of the node than |
There was a problem hiding this comment.
Or "than we might expect"?
Co-authored-by: EmmaLRussell <[email protected]>
No description provided.