What is the problem?

While developing #11379, I found out the performance was really slow sometimes, and found out only 2 IO workers were utilized although I specified 10 IO workers.

Reproduction (REQUIRED)

Please provide a script that can be run to reproduce the issue. The script should have no external library dependencies (i.e., use fake or mock data / environments):

If we cannot run your script, we cannot fix your issue.

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.