Motivation

• Assuming a role and using a client with the returned credentials is a pattern we often need across the code base, especially for the use of service roles or service-linked roles. The new method provides a simple interface to do this in one call.
• To enforce IAM between some services, we need to use the new aws client, and also handle the exceptions (it should fail with a warning, but should not crash anything important if a request fails).

Changes

connect_to.with_assumed_role

A new method for the aws client factories.

Usage:

logs_client = connect_to
    .with_assumed_role(role_arn="arn:aws:iam::000000000000:role/test-role", service_principal="lambda")
    .logs

This call will assume the given role as the given service, and return a service level client factory, where various clients can be created using the returned credentials.

The assume role call will be done immediately, the other clients instantiated lazily as they are requested.

Client migrations

Several important inter-service calls between lambda, sqs and sns are migrated to the new clients.
This migration is ongoing, and not complete yet.

During this, I silently killed SYNCHRONOUS_SNS_EVENTS, as it is scheduled to remove with 2.0 anyway.

Fix check if installed check outside of installer lock

This one is a tricky one.
Basically, we are circumventing the lock if the is_installed() method already returns true. But, the default is just checking if a directory is created, which can be true during the installation as well. Moving this entirely behind the lock avoids one thread installing while the other one is checking, and therefore avoids starting incompletely installed dependencies.

This seems to be the root cause for the failing test_kinesis_firehose_elasticsearch_s3_backup test, which is so annoying. It is basically trying to start elasticsearch while we are still installing plugins -> class loading failure.