Clustering performance improvements

[KristinnVikar]

Proposed changes

Reduce clustering from O(n^2) where n is the number of total templates loaded, to O(n).

On my machine, this reduces total nuclei startup time from 19s -> 8.5s (2.23x faster!) (Edit: re-did timing tests)

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[GeorginaReeder]

Thanks so much for your contribution @KristinnVikar , we appreciate it!

We also have a Discord server, which you’re more than welcome to join. It's a great place to connect with fellow contributors and stay updated with the latest developments!

[KristinnVikar]

Using a HashMap based clustering reduces clustering time even further!

However I'm getting a slight regression with the change, where I get "Templates clustered: 1544" but without I get "Templates clustered: 1550", some 6 templates that no longer want to cluster together.

I did some further measurements, originally clustering took 14000ms (single threaded bottleneck), now it takes ~18ms, about 777x faster

[tarunKoyalwar]

lgtm !

nice work @KristinnVikar , this reduced clustering time by lot , by adding time.Now() to benchmark that particular function on my setup i found

Before

[FTL] Clustering took 3.30753675s: total: 6675

After

[FTL] Clustering took 8.236709ms: total: 6679

that's almost 402x faster !!

Note

we can see there are more clusters compared to dev/latest but that's ok and better and i just added more clustering conditions to ssl to avoid issues