Summary

The new AWS API client stack with support for sending additional metadata for internal calls.

We want to gradually move all functionality and use of aws_stack to this and remove it.

Supported Scenarios

Our new AWS client factory needs to take different scenarios of usage into account.

Scenarios

Internal Hidden

Request which either AWS does not do, or does not document in any way. Used for LS workarounds or completely hidden calls.

❌ CloudTrail
❌ IAM Enforcement
✔️ In Memory Possible

Internal as Service

Call of a service to another service as that service principal. Usually has a sourcearn of the resource on which behalf the request was made.

✔️ CloudTrail
✔️ IAM Enforcement
✔️ In memory possible

Internal as Role

Call of a service which assumed a provided role (assumption is enforced via “Internal as Service` itself) beforehand, as that role.

✔️ CloudTrail
✔️ IAM Enforcement
✔️ In memory possible

Internal as calling User

According to @dominikschubert this is the case for some CloudFormation functionality. Basically, CloudFormation does requests on behalf of the user which created the stack.

✔️ CloudTrail
✔️ IAM Enforcement
✔️ In memory possible

External for third-party providers

Used for communication to provider backends outside of our control. For example, dynamodblocal or kinesis-mock.

❌ CloudTrail
❌ IAM Enforcement
❌ In memory possible

External for testing

Used by testing clients to test our resources using boto3.

✔️ CloudTrail
✔️ IAM Enforcement
❌ In memory possible

Factories

This PR provides two subtypes of client factories.

Scenario 1-4 will be taken care of by the InternalClientFactory

Scenario 5-6 will be taken care of by the ExternalClientFactory

Sample usage

Calls made from outside of LocalStack (eg. in integration tests etc.)

from localstack.aws.connect import connect_to

client = connect_to().s3

client.list_objects(BucketName='foo')

Calls made internally across services

from localstack.aws.connect import connect_to

client = connect_to().s3

client.publish(
  TopicArn='arn:aws:sns:ap-south-1:000000000000:lorem', \
  Message='blah', \
  _ServicePrincipal='s3',
)

Performance

New client clocks faster for equivalent aws_stack functionality:

$ AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1 ipython
Python 3.10.8 (main, Nov 10 2022, 18:37:57) [GCC 12.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.11.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from localstack.aws.connect import connect_to, connect_externally_to
   ...:
   ...: def test():
   ...:     c = connect_to(region_name='eu-central-1').ec2
   ...:     d = connect_to(region_name='eu-central-1').ec2
   ...:     e = connect_to(region_name='ap-south-1').ec2
   ...:     f = connect_to(region_name='ap-south-1').ec2
   ...:     i = connect_externally_to(region_name=None, aws_access_key_id=None, aws_secret_access_key=None).dynamodb
   ...:     j = connect_externally_to(region_name=None, aws_access_key_id=None, aws_secret_access_key=None).dynamodb
   ...:     k = connect_externally_to(region_name='ap-south-1', aws_access_key_id='test', aws_secret_access_key='test').dynamodb
   ...: %timeit test()
   ...:
38.8 µs ± 225 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In [2]: from localstack.utils.aws.aws_stack import connect_to_service, create_external_boto_client
   ...: def test_old():
   ...:     c = connect_to_service('ec2')
   ...:     d = connect_to_service('ec2')
   ...:     e = connect_to_service('ec2', region_name='ap-south-1')
   ...:     f = connect_to_service('ec2', region_name='ap-south-1')
   ...:     i = create_external_boto_client('dynamodb')
   ...:     j = create_external_boto_client('dynamodb')
   ...:     k = create_external_boto_client('dynamodb', region_name='ap-south-1', aws_access_key_id='test', aws_secret_access_key='test')
   ...: %timeit test_old()
   ...:
58.2 µs ± 251 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

Other notes

Created by a mob programming session by @alexrashed @dominikschubert @simonrw @thrau @viren-nadkarni @dfangl

Sample use in DynamoDB: aws-client...aws-client-dynamodb

Tests

Unit tested, against a fake Gateway with the real handlers.
The unit tests aim to test the above mentioned scenarios, even though there is some duplication.
Some of the unit tests will in the next iteration be used to check some "convenience utilities", like creating a client directly for a to-be-assumed role, etc.

Potential future use for test clients:

# this is only a demonstration how a fixture could look like, and that the approach matches our requirements
class TestFactoryTestUsage:
    @pytest.fixture(scope="module")
    def test_client_factory(self):
        return ExternalClientFactory()

    @pytest.fixture(scope="class")
    def clients(self, test_client_factory):
        """Clients fixture which will allow all kind of clients to be initialized"""
        botocore_config = botocore.config.Config()

        # can't set the timeouts to 0 like in the AWS CLI because the underlying http client requires values > 0
        if os.environ.get("TEST_DISABLE_RETRIES_AND_TIMEOUTS"):
            botocore_config = botocore_config.merge(
                botocore.config.Config(
                    connect_timeout=1_000, read_timeout=1_000, retries={"total_max_attempts": 1}
                )
            )

        if os.environ.get("TEST_TARGET") == "AWS_CLOUD":
            return test_client_factory(config=botocore_config)

        return test_client_factory(
            region_name="us-east-1",
            aws_access_key_id="test",
            aws_secret_access_key="test",
            endpoint_url=config.get_edge_url(),
            config=botocore_config,
        )

    @pytest.mark.skip
    def test_something_with_boto_clients(self, clients):
        functions = clients.awslambda.list_functions()
        print(functions)

Future work

• Add convenience method for assume a role and creating a client directly
• Add convenience method proxy for type-safe adding of SourceArn and SourcePrincipal