Fix possibly failing to reserve slots due to stale issued number

[Sushisource]

What was changed

Fix a bug where resource based slot supplier was looking at a stale issued slots number while trying to reserve

Why?

Could cause it to be stuck failing to reserve

Checklist

1. Closes ResourceBasedSlotSupplier.ReleaseSlot not implemented #1868

2. How was this tested:
   Added test

3. Any docs updates needed?

[Quinn-With-Two-Ns]

Can we document that NumIssuedSlots is the current number of slots issues and will get updated over time? I can see users getting confused thinking this is the number of reserved slots at the time of the call.

[Sushisource]

Done 👍

[Quinn-With-Two-Ns]

What is the reason for using the SlotInfo count instead of having ResourceBasedSlotSupplier keep track? It should be fine but it is hard to reason about since the SlotInfo count is updated outside the mutex of the ResourceBasedSlotSupplier?

[Quinn-With-Two-Ns]

Also is there a reason this isn't implemented if max slots is provided? https://github.com/temporalio/sdk-go/blob/master/contrib/resourcetuner/resourcetuner.go#L205

[Sushisource]

What is the reason for using the SlotInfo count instead of having ResourceBasedSlotSupplier keep track? It should be fine but it is hard to reason about since the SlotInfo count is updated outside the mutex of the ResourceBasedSlotSupplier?

Because internally we always need to know the issued count, so the tracking supplier does that so it's not duplicated in all implementations. The mutex in the resource based supplier is purely for the last issued time / PID decisions.

[Sushisource]

Also is there a reason this isn't implemented if max slots is provided? https://github.com/temporalio/sdk-go/blob/master/contrib/resourcetuner/resourcetuner.go#L205

It's debatable. I left it unimplemented because it maps to the "available slots" metric, but, you can easily set a max here that isn't actually attainable in any real sense.

[Quinn-With-Two-Ns]

Because internally we always need to know the issued count

To confrim we only update our internal count after the slot supplier ReleaseSlot returns right?

[Sushisource]

Because internally we always need to know the issued count

To confrim we only update our internal count after the slot supplier ReleaseSlot returns right?

Yup, exactly