This PR addresses issues with the terraform test pipeline discovered while working on #6853.
The route53resolver tests are currently timing out, but the pipeline does not fail.

This PR contains the following changes:

• Ensures that the pipeline fails if any terraform tests fail (proper error management in run.sh).
• Implements a strict collection of the allowlisted terraform tests (use strict regexes in get-tf-tests.py).
• Fixes for the failing tests:
  • s3:
    • TestAccS3Bucket_Duplicate_UsEast1AltAccount (removed from the test allowlist)

      2022-09-12T09:50:52.4625278Z === CONT  TestAccS3Bucket_Duplicate_UsEast1AltAccount
      2022-09-12T09:51:01.1596683Z     bucket_test.go:400: Step 1/1, expected an error with pattern, no match on: Error running apply: exit status 1
      2022-09-12T09:51:01.1597193Z         
      2022-09-12T09:51:01.1598165Z         Error: creating Amazon S3 (Simple Storage) Bucket (tf-test-bucket-583879316476979402): bucket already exists
      2022-09-12T09:51:01.1598545Z         
      2022-09-12T09:51:01.1598878Z           with aws_s3_bucket.test,
      2022-09-12T09:51:01.1599417Z           on terraform_plugin_test.tf line 12, in resource "aws_s3_bucket" "test":
      2022-09-12T09:51:01.1599899Z           12: resource "aws_s3_bucket" "test" {
      

  • route53resolver:
    • TestAccRoute53ResolverQueryLogConfig_disappears (fixed by accessing the store properly)
    • TestAccRoute53ResolverQueryLogConfig_tags (fixed by accessing the store properly)
    • TestAccRoute53ResolverQueryLogConfig_basic (fixed by accessing the store properly)
    • TestAccRoute53ResolverQueryLogConfigAssociation_disappears (fixed by accessing the store properly)
    • TestAccRoute53ResolverFirewallDomainList_domains (fixed by accessing the store properly)

      2022-09-12T09:53:47.8795642Z     query_log_config_test.go:54: Step 1/1 error: Error running apply: exit status 1
      2022-09-12T09:53:47.8796007Z
      2022-09-12T09:53:47.8798233Z         Error: error creating Route53 Resolver Query Log Config: InternalError: exception while calling route53resolver.CreateResolverQueryLogConfig: 'AccountSpecificBackend' object has no attribute 'tagger'
      2022-09-12T09:53:47.8799214Z         	status code: 500, request id: 9KM3SHS5UN2724213D63LIN6FQGTJQKC206DG8E4R598AB5ZVYF7
      2022-09-12T09:53:47.8799591Z
      2022-09-12T09:53:47.8800164Z           with aws_route53_resolver_query_log_config.test,
      2022-09-12T09:53:47.8800960Z           on terraform_plugin_test.tf line 7, in resource "aws_route53_resolver_query_log_config" "test":
      2022-09-12T09:53:47.8801678Z            7: resource "aws_route53_resolver_query_log_config" "test" {
      2022-09-12T09:53:47.8801947Z
      2022-09-12T09:53:49.5246826Z --- FAIL: TestAccRoute53ResolverQueryLogConfig_disappears (2810.72s)
      2022-09-12T09:58:41.5540977Z === CONT  TestAccRoute53ResolverQueryLogConfig_tags
      2022-09-12T09:58:41.5544064Z     query_log_config_test.go:78: Step 1/4 error: Error running apply: exit status 1
      2022-09-12T09:58:41.5544629Z
      2022-09-12T09:58:41.5546143Z         Error: error creating Route53 Resolver Query Log Config: InternalError: exception while calling route53resolver.CreateResolverQueryLogConfig: 'AccountSpecificBackend' object has no attribute 'tagger'
      2022-09-12T09:58:41.5547319Z         	status code: 500, request id: CW9PP08LGG2OB3EVENR8F5GHKE20F9VW0XRHYGC8U6AJL48CH5RT
      2022-09-12T09:58:41.5547645Z
      2022-09-12T09:58:41.5548191Z           with aws_route53_resolver_query_log_config.test,
      2022-09-12T09:58:41.5548821Z           on terraform_plugin_test.tf line 6, in resource "aws_route53_resolver_query_log_config" "test":
      2022-09-12T09:58:41.5549390Z            6: resource "aws_route53_resolver_query_log_config" "test" {
      2022-09-12T09:58:41.5549661Z
      2022-09-12T09:58:43.0838174Z --- FAIL: TestAccRoute53ResolverQueryLogConfig_tags (3104.28s)
      2022-09-12T09:58:57.4419492Z === CONT  TestAccRoute53ResolverQueryLogConfig_basic
      2022-09-12T09:58:57.4420249Z     query_log_config_test.go:23: Step 1/2 error: Error running apply: exit status 1
      2022-09-12T09:58:57.4420641Z
      2022-09-12T09:58:57.4422182Z         Error: error creating Route53 Resolver Query Log Config: InternalError: exception while calling route53resolver.CreateResolverQueryLogConfig: 'AccountSpecificBackend' object has no attribute 'tagger'
      2022-09-12T09:58:57.4423075Z         	status code: 500, request id: IFCWL3RL0OH78VFWYY5DCOAM7VCQU1J1ATYMPDJ8IB4SRPKVLQC8
      2022-09-12T09:58:57.4423487Z
      2022-09-12T09:58:57.4423954Z           with aws_route53_resolver_query_log_config.test,
      2022-09-12T09:58:57.4424649Z           on terraform_plugin_test.tf line 7, in resource "aws_route53_resolver_query_log_config" "test":
      2022-09-12T09:58:57.4425292Z            7: resource "aws_route53_resolver_query_log_config" "test" {
      2022-09-12T09:58:57.4425646Z
      2022-09-12T09:59:00.2395038Z --- FAIL: TestAccRoute53ResolverQueryLogConfig_basic (3098.44s)
      2022-09-12T09:59:26.9404733Z === CONT  TestAccRoute53ResolverQueryLogConfigAssociation_disappears
      2022-09-12T09:59:26.9405659Z     query_log_config_association_test.go:52: Step 1/1 error: Error running apply: exit status 1
      2022-09-12T09:59:26.9406182Z
      2022-09-12T09:59:26.9408505Z         Error: error creating Route53 Resolver Query Log Config: InternalError: exception while calling route53resolver.CreateResolverQueryLogConfig: 'AccountSpecificBackend' object has no attribute 'tagger'
      2022-09-12T09:59:26.9409555Z         	status code: 500, request id: K4UB9MWFJM5H9QFGI93Z1624PFDFI6RSL897YSCZPPD102IAH3ZD
      2022-09-12T09:59:26.9410362Z
      2022-09-12T09:59:26.9411072Z           with aws_route53_resolver_query_log_config.test,
      2022-09-12T09:59:26.9412077Z           on terraform_plugin_test.tf line 14, in resource "aws_route53_resolver_query_log_config" "test":
      2022-09-12T09:59:26.9412974Z           14: resource "aws_route53_resolver_query_log_config" "test" {
      2022-09-12T09:59:26.9413535Z
      2022-09-12T09:59:30.7844206Z --- FAIL: TestAccRoute53ResolverQueryLogConfigAssociation_disappears (3099.95s)
      2022-09-12T10:00:52.0020687Z === CONT  TestAccRoute53ResolverFirewallDomainList_domains
      2022-09-12T10:00:52.0023109Z     firewall_domain_list_test.go:53: Step 4/4 error: After applying this test step and performing a `terraform refresh`, the plan was not empty.
      2022-09-12T10:00:52.0051294Z         stdout
      2022-09-12T10:00:52.0051847Z
      2022-09-12T10:00:52.0052826Z
      2022-09-12T10:00:52.0053577Z         Terraform used the selected providers to generate the following execution
      2022-09-12T10:00:52.0055342Z         plan. Resource actions are indicated with the following symbols:
      2022-09-12T10:00:52.0056183Z           ~ update in-place
      2022-09-12T10:00:52.0056603Z
      2022-09-12T10:00:52.0198315Z         Terraform will perform the following actions:
      2022-09-12T10:00:52.0198637Z
      2022-09-12T10:00:52.0201092Z           # aws_route53_resolver_firewall_domain_list.test will be updated in-place
      2022-09-12T10:00:52.0203740Z           ~ resource "aws_route53_resolver_firewall_domain_list" "test" {
      2022-09-12T10:00:52.0206149Z               ~ domains  = [
      2022-09-12T10:00:52.0208862Z                   - "66tguovt.test.",
      2022-09-12T10:00:52.0211570Z                 ]
      2022-09-12T10:00:52.0214240Z                 id       = "rslvr-fdl-e166b265c167fdb5b"
      2022-09-12T10:00:52.0217093Z                 name     = "tf-acc-test-8485330124294055099"
      2022-09-12T10:00:52.0219938Z                 tags     = {}
      2022-09-12T10:00:52.0222213Z                 # (2 unchanged attributes hidden)
      2022-09-12T10:00:52.0224318Z             }
      2022-09-12T10:00:52.0226361Z
      2022-09-12T10:00:52.0228536Z         Plan: 0 to add, 1 to change, 0 to destroy.
      2022-09-12T10:00:56.0390280Z --- FAIL: TestAccRoute53ResolverFirewallDomainList_domains (61.30s)
      

  • route53:
    • TestAccRoute53ZoneAssociation_crossAccount (removed from the allowlist)
    • TestAccRoute53VPCAssociationAuthorization_disappears (removed from the allowlist)
    • TestAccRoute53VPCAssociationAuthorization_basic (removed from the allowlist)

      2022-09-12T09:15:17.5808405Z     zone_association_test.go:118: Step 1/2 error: Error running apply: exit status 1
      2022-09-12T09:15:17.5808906Z
      2022-09-12T09:15:17.5810873Z         Error: Error creating Route53 VPC Association Authorization: InternalFailure: API action 'CreateVPCAssociationAuthorization' for service 'route53' not yet implemented or pro feature - check https://docs.localstack.cloud/aws/feature-coverage for further information
      2022-09-12T09:15:17.5812009Z         	status code: 501, request id: 6MJPGBSLVP4LFB2OCV4YW9CMQOQ5VR8R5A77IZ2QQ0SVBIUY37OR
      2022-09-12T09:15:17.5812438Z
      2022-09-12T09:15:17.5812954Z           with aws_route53_vpc_association_authorization.test,
      2022-09-12T09:15:17.5813765Z           on terraform_plugin_test.tf line 36, in resource "aws_route53_vpc_association_authorization" "test":
      2022-09-12T09:15:17.5814539Z           36: resource "aws_route53_vpc_association_authorization" "test" {
      2022-09-12T09:15:17.5814892Z
      2022-09-12T09:15:20.7815499Z --- FAIL: TestAccRoute53ZoneAssociation_crossAccount (37.82s)
      2022-09-12T09:18:21.8024190Z === CONT  TestAccRoute53VPCAssociationAuthorization_disappears
      2022-09-12T09:18:21.8025713Z     vpc_association_authorization_test.go:48: Step 1/1 error: Error running apply: exit status 1
      2022-09-12T09:18:21.8026384Z
      2022-09-12T09:18:21.8028598Z         Error: Error creating Route53 VPC Association Authorization: InternalFailure: API action 'CreateVPCAssociationAuthorization' for service 'route53' not yet implemented or pro feature - check https://docs.localstack.cloud/aws/feature-coverage for further information
      2022-09-12T09:18:21.8185472Z         	status code: 501, request id: QQY9JQ4ZV7Y1TH1FW4UCMPWOGMEVWLMV5ZBRB5MWV37RC37DRNRK
      2022-09-12T09:18:21.8186139Z
      2022-09-12T09:18:21.8186667Z           with aws_route53_vpc_association_authorization.test,
      2022-09-12T09:18:21.8187484Z           on terraform_plugin_test.tf line 29, in resource "aws_route53_vpc_association_authorization" "test":
      2022-09-12T09:18:21.8188203Z           29: resource "aws_route53_vpc_association_authorization" "test" {
      2022-09-12T09:18:21.8188556Z
      2022-09-12T09:18:26.5223056Z --- FAIL: TestAccRoute53VPCAssociationAuthorization_disappears (50.24s)
      2022-09-12T09:18:50.3039511Z === CONT  TestAccRoute53VPCAssociationAuthorization_basic
      2022-09-12T09:18:50.3040676Z     vpc_association_authorization_test.go:20: Step 1/2 error: Error running apply: exit status 1
      2022-09-12T09:18:50.3041349Z
      2022-09-12T09:18:50.3043512Z         Error: Error creating Route53 VPC Association Authorization: InternalFailure: API action 'CreateVPCAssociationAuthorization' for service 'route53' not yet implemented or pro feature - check https://docs.localstack.cloud/aws/feature-coverage for further information
      2022-09-12T09:18:50.3044875Z         	status code: 501, request id: D0MPVX16I1Y4EFBB6PZIDGTLUJQSJVF35NV90MKPHIVG6V3GASBI
      2022-09-12T09:18:50.3045562Z
      2022-09-12T09:18:50.3046236Z           with aws_route53_vpc_association_authorization.test,
      2022-09-12T09:18:50.3047227Z           on terraform_plugin_test.tf line 29, in resource "aws_route53_vpc_association_authorization" "test":
      2022-09-12T09:18:50.3048187Z           29: resource "aws_route53_vpc_association_authorization" "test" {
      2022-09-12T09:18:50.3048784Z
      2022-09-12T09:18:53.4851715Z --- FAIL: TestAccRoute53VPCAssociationAuthorization_basic (35.34s)
      

/cc @dominikschubert @macnev2013