[BUG] Creating a detector hangs. The detector is then silently created many hours later

[engechas]

What is the bug?
This one's a 2-parter. The first part is that clusters under high load will hang when creating a detector. The networking tab of dev tools eventually shows a 504 GW timeout for the request.

In one occurrence of this, two detectors that were hung during creation did not appear until nearly a day later. Once they appeared, they began running normally.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Generate a high indexing workload against a cluster
2. Attempt to create a detector against the index being ingested to
3. Verify detector creation hangs

What is the expected behavior?
Detector creation should not hang. Detectors should not be created under the hood many hours later if the detector creation failed from the user perspective.

What is your host/environment?

• OS: [e.g. iOS]
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

[praveensameneni]

Believe this PR addresses the issue. Feel free to re-open if it's still present