Implement more sophisticated scheduler

[amscanne]

Today, the scheduler is essentially the Go runtime scheduler. In some cases however, this can lead to poor interactions between application threads and extra cycles burnt (e.g. #2033, #1942).

This issue serves as the parent issue for scheduler exploration. There are effectively two possible paths that we will pursue:

1. Reduce negative interactions with the Go scheduler by binding task goroutines directly to host threads. This may require some modifications to the Go runtime that allow for a larger number of Ps, as many of the poor interactions come at the entersyscall/exitsyscall time (and may involve the sysmon thread). This has advantage that it's likely to map closely to host performance if done correctly. It has the disadvantage that the number of threads is under user control, and the failure to create a new host thread can take place at any point in the runtime (likely taking down the sandbox).
2. Take more control over interactions with the Go runtime and implement some custom scheduling functionality. This will likely require more work, and it's unclear to the extent we can implement anything reasonably performance. However, this does not suffer from the host thread limit failures discussed in (1).

Today we effectively have some of the worst of both (1) and (2). We should continue to explore the appropriate avenue, come up with a design and improve our scheduler scalability.

[fly3366]

Any change?

[kscooo]

Any change?

[milantracy]

Any change?

@kscooo which change are you looking for specifically?

[kscooo]

@milantracy Implement high-performance Go scheduler

[milantracy]

@milantracy Implement high-performance Go scheduler

understand

@kscooo would you mind sharing your use case, and the bottleneck that you think could be mitigated by having the feature?

there might be other workaround that could be helpful

[kscooo]

@milantracy I actually don't have extensive experience using gvisor, I'm just curious about the overall details of the documentation and code implementation of the proposed improvement plan.