Skip to content

Add initial support for CFn Fn:Transform intrinsic functions#7711

Merged
whummer merged 8 commits intomasterfrom
cfn-fn-transform
Feb 22, 2023
Merged

Add initial support for CFn Fn:Transform intrinsic functions#7711
whummer merged 8 commits intomasterfrom
cfn-fn-transform

Conversation

@whummer
Copy link
Member

@whummer whummer commented Feb 17, 2023

Add initial support for CFn Fn:Transform intrinsic functions. This requirement came up while playing around with a more complex AWS sample app using a SAM / CloudFormation template: https://workshop.serverlesscoffee.com

For reference - part of the SAM template in the sample is a construct like this:

  RESTApConfigService:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      DefinitionBody:
        'Fn::Transform':
          Name: 'AWS::Include'
          Parameters:
            Location: '../backends/2-config-service/RestAPIs/api.yaml'

(the Location parameter is replaced with an s3://... URL by SAM upon deployment)

@whummer whummer temporarily deployed to localstack-ext-tests February 17, 2023 13:48 — with GitHub Actions Inactive
@github-actions
Copy link

github-actions bot commented Feb 17, 2023

LocalStack integration with Pro

       1 files   -     2         1 suites   - 2   1h 1m 31s ⏱️ - 30m 50s
1 718 tests  -   15  1 358 ✔️  -   26  359 💤 +  10  1 +1 
1 718 runs   - 733  1 358 ✔️  - 402  359 💤  - 332  1 +1 

For more details on these failures, see this check.

Results for commit a7e3a12. ± Comparison against base commit 87feed2.

♻️ This comment has been updated with latest results.

@whummer whummer temporarily deployed to localstack-ext-tests February 17, 2023 15:16 — with GitHub Actions Inactive
@whummer whummer temporarily deployed to localstack-ext-tests February 17, 2023 15:46 — with GitHub Actions Inactive
@whummer whummer requested a review from pinzon February 17, 2023 15:58
@coveralls
Copy link

coveralls commented Feb 17, 2023

Coverage Status

Coverage: 85.015% (+0.04%) from 84.975% when pulling a7e3a12 on cfn-fn-transform into 87feed2 on master.

@whummer whummer temporarily deployed to localstack-ext-tests February 17, 2023 17:51 — with GitHub Actions Inactive
@whummer whummer temporarily deployed to localstack-ext-tests February 17, 2023 21:00 — with GitHub Actions Inactive
@whummer
Copy link
Member Author

whummer commented Feb 18, 2023

Test error in integration-tests-against-pro (test TestRedshift::test_create_clusters) seems unrelated to these changes:

...
2023-02-17T21:26:54.1775824Z   File "/home/runner/work/localstack/localstack/localstack/localstack/packages/api.py", line 89, in install
2023-02-17T21:26:54.1776199Z     self._prepare_installation(target)
2023-02-17T21:26:54.1776843Z   File "/home/runner/work/localstack/localstack/localstack-ext/localstack_ext/packages/core.py", line 76, in _prepare_installation
2023-02-17T21:26:54.1777222Z     self._os_switch(
2023-02-17T21:26:54.1777812Z   File "/home/runner/work/localstack/localstack/localstack-ext/localstack_ext/packages/core.py", line 62, in _os_switch
2023-02-17T21:26:54.1778459Z     raise SystemNotSupportedException(
2023-02-17T21:26:54.1779221Z localstack.packages.core.SystemNotSupportedException: OS level packages are only installed within docker containers.
2023-02-17T21:26:54.1787826Z 2023-02-17T21:26:54.178  INFO --- [   asgi_gw_5] localstack.request.aws     : AWS redshift.CreateCluster => 500 (InternalError)
2023-02-17T21:26:54.2252216Z 2023-02-17T21:26:54.224  INFO --- [   asgi_gw_7] localstack.request.aws     : AWS dynamodb.Scan => 200
2023-02-17T21:26:54.3094671Z 2023-02-17T21:26:54.309  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS redshift.CreateCluster => 400 (ClusterAlreadyExists)
2023-02-17T21:26:54.3376208Z RERUN
2023-02-17T21:26:54.3474312Z ../localstack/tests/integration/test_redshift.py::TestRedshift::test_create_clusters 

@whummer whummer temporarily deployed to localstack-ext-tests February 20, 2023 16:33 — with GitHub Actions Inactive
@whummer whummer temporarily deployed to localstack-ext-tests February 20, 2023 18:05 — with GitHub Actions Inactive
Copy link
Member

@pinzon pinzon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

Copy link
Member

@dominikschubert dominikschubert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly small nits that don't block the merge, otherwise LGTM. I like the intrinsic function abstraction. It kinda resembles what I've seen so far in related tools and how I'd like us to approach the rest as well 👍

Request for change affects the recorded snapshot since it won't pass against AWS when re-running.

from localstack.utils.aws import aws_stack
from localstack.utils.strings import str_to_bool

TAG_KEY_LOGICAL_ID = "aws:cloudformation:logical-id"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there something that explicitly breaks on this missing? Because I'm not sure about only implementing it here when it should actually be included nearly everywhere 🤔

Copy link
Member Author

@whummer whummer Feb 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a great catch. This change actually hints at a general limitation we have in our approach (as you're well aware) - the way how we detect the resource state and the fact that we're allowing templates to be deployed on top of existing resources.

With the current implementation of (some of) our resource classes, in combination with the change above, the fetch_state method can sometimes return false positives - i.e., report that the resource already exists when in fact it doesn't exist yet.

This issue in particular occurs if we have multiple instances of resource types whose deployment state cannot easily be determined based on the resource properties, e.g., consider the example below (we have a similar example in our tests):

  routeTable1:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: vpc-12345
  routeTable2:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: vpc-12345

I've simplified the implementation - now the state lookup detects whether PhysicalResourceId has already been set (using the result_handler hook) and only then returns the resource details. This is quite a bit cleaner, and also should give us better parity with AWS (no additional tags). 👌

This also plays into the general discussion of resource state handling and the deployment loop (as we already discussed) - over time, we can refactor the approach and loosen the assumption that existing resources are "imported" into the stack idempotently if they already exist. This will give us even better parity with AWS over time. 👍

@whummer whummer temporarily deployed to localstack-ext-tests February 20, 2023 23:08 — with GitHub Actions Inactive
@whummer whummer temporarily deployed to localstack-ext-tests February 21, 2023 09:43 — with GitHub Actions Inactive
@whummer whummer temporarily deployed to localstack-ext-tests February 21, 2023 11:06 — with GitHub Actions Inactive
@whummer whummer temporarily deployed to localstack-ext-tests February 21, 2023 11:06 — with GitHub Actions Inactive
@whummer whummer temporarily deployed to localstack-ext-tests February 21, 2023 13:57 — with GitHub Actions Inactive
@whummer
Copy link
Member Author

whummer commented Feb 21, 2023

Tests against pro failed with TestRedshift::test_create_clusters (seems unrelated to this PR), and then timed out. Should be good for a re-review @dominikschubert 👍

2023-02-21T15:01:29.4233622Z =========================== short test summary info ============================
2023-02-21T15:01:29.4234226Z FAILED ../localstack/tests/integration/test_redshift.py::TestRedshift::test_create_clusters
2023-02-21T15:01:29.4234908Z = 1 failed, 1346 passed, 292 skipped, 67 xfailed, 12 xpassed, 4877 warnings, 2 rerun in 3691.38s (1:01:31) =

Copy link
Member

@dominikschubert dominikschubert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now 🚀

Thanks for addressing all the raised nits/concerns 👍

transformers: Dict[str, Type] = {"AWS::Include": AwsIncludeTransformer}


def apply_transform_intrinsic_functions(template: dict, stack=None) -> dict:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I understand why we currently need it, I'm not super happy about the stack class being in here again 😬

@whummer whummer merged commit 499e7bd into master Feb 22, 2023
@whummer whummer deleted the cfn-fn-transform branch February 22, 2023 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants