Fixes: #121608
Backport of #121887

The ThreadPool in rare cases allows a scenario when an enqueued workitem
is not guaranteed to be executed unless more workitems are enqueued. In
some scenarios execution of a particular workitem may be necessary
before more work is enqueued, thus leading to a deadlock.
This is a subtle regression introduced by a change in enqueuer/worker
handshake algorithm.

The same pattern is used in 2 other ThreadPool-like internal features in
addition to the ThreadPool.

## Customer Impact

- [ ] Customer reported
- [x] Found internally

## Regression

- [x] Yes
- [ ] No

## Testing

Standard test pass for deterministic regressions.
A targeted stress application that demonstrates the issue. 
(no fix: hangs within 1-2 minutes, with the fix: runs for 20+ min. on
the same system)

## Risk
Low. This is a revert to the preexisting algorithm for the
enqueuer/worker handshake. (in all 3 places)

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Andy Gocke <angocke@microsoft.com>

…122362)

Fixes: #121608
Backport of #121887

The ThreadPool in rare cases allows a scenario when an enqueued workitem
is not guaranteed to be executed unless more workitems are enqueued. In
some scenarios execution of a particular workitem may be necessary
before more work is enqueued, thus leading to a deadlock.
This is a subtle regression introduced by a change in enqueuer/worker
handshake algorithm.

The same pattern is used in 2 other ThreadPool-like internal features in
addition to the ThreadPool.

## Customer Impact

- [ ] Customer reported
- [x] Found internally

## Regression

- [x] Yes
- [ ] No

## Testing

Standard test pass for deterministic regressions.
A targeted stress application that demonstrates the issue. 
(no fix: hangs within 1-2 minutes, with the fix: runs for 20+ min. on
the same system)

## Risk
Low. This is a revert to the preexisting algorithm for the
enqueuer/worker handshake. (in all 3 places)

This is a follow up on recommendations from Scalability Experiments done
some time ago.

The Scalability Experiments resulted in many suggestions. In this part
we look at overheads of submitting and executing a workitem to the
threadpool from the thread scheduling point of view. In particular -
this PR tries to minimize changes to the workqueue to scope the changes.
The workqueue related recommendations will be addressed separately. 

The threadpool parts are very interconnected though, and sometimes
removing one bottleneck results in another one to show up, so some
workqueue changes had to be done, just to avoid regressions.

There are also a few "low hanging fruit" fixes for per-workitem
overheads like unnecessary fences or too frequent modifications of
shared state.
Hopefully this will negate some of the regressions from
#121887 (as was reported in
#122186)


In this change:
- fewer operations per work item where possible. 
such as fewer/weaker fences where possible, reporting heartbeat once per
dispatch quantum vs. per each workitem, etc..

- avoid spurious wakes of worker threads. (except, unavoidably, when
thread goal is changed - by HillClimb and such).
only one thread is requested at a time. requesting another thread is
conditioned on evidence of work present in the queue (basically the
minimum required for correctness).
  as a result a thread that becomes active typically finds work.
in particular this avoids a cascade of spurious wakes when pool is
running out of workitems.

- stop tracking spinners count in LIFO semaphore. 
we can keep track of spinners, but informational value of knowing the
extremely transient count is close to zero, so we should not.

- no Sleep in LIFO semaphore.
using spin-Sleep is questionable in a synchronization feature that can
block and ask OS to wake a thread deterministically.

- shortening spinning in the LIFO semaphore to a more affordable value. 
since the LIFO semaphore can perform a blocking wait until condition it
wants to see happens, once spinning gets into the range of wait/wake
latency, it makes no sense to spin for much longer.
it is also not uncommon that the work is introduced by non-pool threads,
thus the pool threads may have to block just to allow for more work to
be scheduled.

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>