@edoakes This PR changes the behavior of (non-asyncio) concurrent actor calls. Previously, there will be only one thread pool of size N. Now, we will create a new thread pool of size N for each caller. This leads to creating too many threads.
I read your PR description. It seems that this change is unintentional?

I think we should still put the pool and the fiber state in CoreWorkerDirectTaskReceiver. We can address the original issue by only calling SetMaxActorConcurrency and SetActorAsAsync in actor creation tasks.

@raulchen I don't fully understand -- why is there a new thread pool for each caller? We only initialize pool_ once.

@edoakes Each SchedulingQueue has a thread pool. And in CoreWorkerDirectTaskReceiver, we will create a new SchedulingQueue for each caller.

  /// Queue of pending requests per actor handle.
  /// TODO(ekl) GC these queues once the handle is no longer active.
  std::unordered_map<TaskID, std::unique_ptr<SchedulingQueue>> scheduling_queue_;

  auto it = scheduling_queue_.find(task_spec.CallerId());
  if (it == scheduling_queue_.end()) {
    auto result = scheduling_queue_.emplace(
        task_spec.CallerId(), std::unique_ptr<SchedulingQueue>(new SchedulingQueue(
                                  task_main_io_service_, *waiter_, worker_context_)));
    it = result.first;
  }

Filed an issue #14894
will fix it shortly.

Ah yeah I see your point; good find, we should definitely only be creating this many threads per actor