https://issues.apache.org/jira/browse/ARROW-16498

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

CC @michalursa

TaskScheduler, StressTwo should reproduce this but it can still be a bit of a pain on a fast system. Adding a print statement between:

  const auto& tasks = PickTasks(num_new_tasks);

and

  if (static_cast<int>(tasks.size()) < num_new_tasks) {

adds enough of a delay to trigger this fairly reliably (you will also have to comment out the fix).

This fix should be a pretty light-touch but it does add one compare_and_swap in the ScheduleMore method. Another potential fix would be to turn the following lock into a full on spinlock:

  for (;;) {
    int expected = num_tasks_to_schedule_.value.load();
    if (num_tasks_to_schedule_.value.compare_exchange_strong(expected, 0)) {
      num_new_tasks += expected;
      break;
    }
  }

changing it to:

  for (;;) {
    int expected = num_tasks_to_schedule_.value.load();
    if (expected == 0) {
      continue;
    }
    if (num_tasks_to_schedule_.value.compare_exchange_strong(expected, 0)) {
      num_new_tasks += expected;
      break;
    }
  }

Let me know if you want to try anything different.

I'm going to merge this on green (just tying up Windows export issues since some new unit tests require new exports).

Benchmark runs are scheduled for baseline = 7341dbc and contender = 07135b2. 07135b2 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.23% ⬆️0.04%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.36%] ursa-i9-9960x
[Finished ⬇️0.28% ⬆️0.08%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 07135b25 ec2-t3-xlarge-us-east-2
[Finished] 07135b25 test-mac-arm
[Finished] 07135b25 ursa-i9-9960x
[Finished] 07135b25 ursa-thinkcentre-m75q
[Finished] 7341dbcd ec2-t3-xlarge-us-east-2
[Finished] 7341dbcd test-mac-arm
[Finished] 7341dbcd ursa-i9-9960x
[Finished] 7341dbcd ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java