On a system using the kernel argument isolcpu, or when using taskset to limit the CPUs available to the process OpenCV continues to allocate threads for all cores on the system even if they are not available to it. This causes OpenCV to create far more threads then is optimal for the target system.

To obtain the actual number of cores available to the process one should use sched_getaffinity as such:

#include <sched.h>

int getCPUs()
{
  cpu_set_t cpu_set;
  sched_getaffinity(0, sizeof(cpu_set), &cpu_set);
  return CPU_COUNT(&cpu_set);
}

To workaround this bad behavior one can do the following:

cpu_set_t cpu_set;
sched_getaffinity(0, sizeof(cpu_set_t), &cpu_set);
cv::setNumThreads(std::max(1, CPU_COUNT(&cpu_set)-1));