This PR has the following changes to source plugin(s) tables:

• Table aws_iam_group_last_accessed_details was added
• Table aws_iam_policy_last_accessed_details was added
• Table aws_iam_role_last_accessed_details was added
• Table aws_iam_user_last_accessed_details was added

How much concurrency does AWS support?

How much concurrency does AWS support?

@bbernays can you explain your comment? Do you mean API limits? Concurrent report jobs limit?

I couldn't find anything about this in the AWS docs

How much concurrency does AWS support?

@bbernays can you explain your comment? Do you mean API limits? Concurrent report jobs limit?

I couldn't find anything about this in the AWS docs

Yeah I mean about concurrent reports jobs limit. I know the original PR had all of the jobs being created sequentially, so I wonder if there was a technical reason for that

Yeah I mean about concurrent reports jobs limit. I know the original PR had all of the jobs being created sequentially, so I wonder if there was a technical reason for that

I did not hit any limits on our playground account 😄 nor could find any docs about it

I'd like to check my understanding of this code:

1. for every group / policy / role/ user, a last_accessed_jobs resolver will get called, because of the parent-child relationship with the corresponding iam table
2. this child resolver creates a new job and writes the ARN and job ID to the corresponding last_accessed_jobs table. Then,
3. a last_accessed_details resolver kicks off (again because of parent-child relationship), sleeping until the job either fails or succeeds. It then writes the details to the last_accessed_details table.

Is my understanding correct? This is a clever solution! I think it subtly changes the semantics of our tables though - especially the last_accessed_jobs table. Usually, a table is expected to fetch all available data for the particular resource. In the case of last_accessed_jobs though, it's more or less being employed as a queue--users won't get a full list of all "last accessed jobs" that have ever been created on their accounts, they will only get the last accessed jobs that have been generated by this particular sync. I think that's maybe not ideal? Is it necessary to have the jobs table at all? Why not have only a details table, where we both generate the job and wait for its results?

Is my understanding correct? This is a clever solution! I think it subtly changes the semantics of our tables though - especially the last_accessed_jobs table. Usually, a table is expected to fetch all available data for the particular resource. In the case of last_accessed_jobs though, it's more or less being employed as a queue--users won't get a full list of all "last accessed jobs" that have ever been created on their accounts, they will only get the last accessed jobs that have been generated by this particular sync. I think that's maybe not ideal? Is it necessary to have the jobs table at all? Why not have only a details table, where we both generate the job and wait for its results?

You correct! And you have a very good point. I think we can drop the jobs table as there's a 1-1 mapping between job and details. There will be a slight behavioral change (depending on the scheduler) as the details tables will take a bit longer and also get better concurrency.
The idea was to have as little API calls as possible per table, so the SDK can manage the concurrency of the resolvers.
Another thing to note is that a job ID is only valid within the same session:

so I didn't see a point in saving all of them.

TDLR: I'll drop the jobs tables

@hermanschaaf cleaned up the job tables in d3ed6b9