I'd like to give it a go as well as I was dealing with a lot of issues around our concurrency and locks. I commented some of the stuff out because of lack of time and because I encountered some race conditions which I didn't have time to fix at the time.

I'd like to give it a go as well as I was dealing with a lot of issues around our concurrency and locks. I commented some of the stuff out because of lack of time and because I encountered some race conditions which I didn't have time to fix at the time.

💯 I understand as this is a core change. I think the challenge is that we can't predict how many table.resolve jobs we have as when we have a relation we resolve it for each parent resource that was fetched. Hence we can only enforce the number of Go routines at the top level.
The only way I could think of of enforcing the max Go routines is instead of resolving the relations in the same Go routine as the parent, send it as a new job back to the worker channel/queue. The problem with that approach is that we need to keep in memory all the relation jobs until those are scheduled. For example if a top level table fetches 10,000 items and it has 4 relations, we'll queue 40,000 new jobs.

I deleted the comments as code in comments usually gets stale and leads to future confusion.

I'd like to give it a go as well as I was dealing with a lot of issues around our concurrency and locks. I commented some of the stuff out because of lack of time and because I encountered some race conditions which I didn't have time to fix at the time.

💯 I understand as this is a core change. I think the challenge is that we can't predict how many table.resolve jobs we have as when we have a relation we resolve it for each parent resource that was fetched. Hence we can only enforce the number of Go routines at the top level. The only way I could think of of enforcing the max Go routines is instead of resolving the relations in the same Go routine as the parent, send it as a new job back to the worker channel/queue. The problem with that approach is that we need to keep in memory all the relation jobs until those are scheduled. For example if a top level table fetches 10,000 items and it has 4 relations, we'll queue 40,000 new jobs.

I deleted the comments as code in comments usually gets stale and leads to future confusion.

I think this still wont solve it and will always cause deadlock. The only way I see is possible to solve it is with X parameters:

concurrency_relations_1, concurrency_relations_2, and so on. a limit on number of jobs wont solve as child relations can always cause deadlock for parent, doesn't matter if it's queue, worker channel, etc...

concurrency_relations_1, concurrency_relations_2

Yeah I thought about that too, but that's a dynamic number based on the depth of relations and also very hard to expose as a config.

a limit on number of jobs wont solve as child relations can always cause deadlock for parent, doesn't matter if it's queue, worker channel, etc...

I think if you resolve only 1 level each time and each relation is a new job it won't deadlock as the parent already finished running (if we do a non blocking send or use a very big channel). But again that leads to memory issues as the number of jobs grows very high.

Hence this PR only implements a limit at the top level tables and avoids the term Go routines.
If we don't have a limit at all the only way to control concurrency is to use multiple configuration files with a different set of resources per configuration.

On another note Azure has 55 top level tables, and if someone has 4 subscriptions the current code will create 220 Go routines. I'm actually not worried about the number of Go routines, but on the number of API calls and rate limiting.
⬆️ Might not be a problem yet, but it would be nice to be able to control it.

Closing in favor of #129