Skip to content

chore(individual-pkg-gen): remove L1s from alpha modules#16354

Closed
madeline-k wants to merge 7 commits intoaws:v2-mainfrom
madeline-k:v2/remove-l1s-from-alpha-modules
Closed

chore(individual-pkg-gen): remove L1s from alpha modules#16354
madeline-k wants to merge 7 commits intoaws:v2-mainfrom
madeline-k:v2/remove-l1s-from-alpha-modules

Conversation

@madeline-k
Copy link
Copy Markdown
Contributor

@madeline-k madeline-k commented Sep 2, 2021

The PR contains the following changes:

  1. Skip copying the generated L1 files into the alpha modules
  2. Re-write imports in the alpha modules that reference L1s to reference aws-cdk-lib/aws-
    These imports come in the following formats: ./<service>.generated, ../<service>.generated, ../lib/<service>.generated, ./<service>-canned-metrics.generated. All of these formats get converted to aws-cdk-lib/aws-<service>
  3. Don't export generated L1s in the index.ts files of the alpha modules.

Closes #15587


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@madeline-k madeline-k requested review from a team and njlynch September 2, 2021 22:47
@gitpod-io
Copy link
Copy Markdown

gitpod-io bot commented Sep 2, 2021

@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Sep 2, 2021
Copy link
Copy Markdown
Contributor

@njlynch njlynch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unfortunate the individual-pkg-gen package doesn't currently have tests, but can you add a test for the rewrite changes in aws-cdk-migration?

Comment on lines +118 to +119
// The following line takes the 2nd to last item in modulePathSplit, which will always be the basename of the module e.g. api-gatewayv2.
return `aws-cdk-lib/aws-${modulePathSplit[modulePathSplit.length-2].replace('-canned-metrics', '')}`;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit too hacky for my taste. I think I'm leaning towards Nick's suggestion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed the .replace('-canned-metrics','') part now based on Nick's suggestion. I agree with you this was hacky, and it still is a little bit. Open to any other suggestions you have to improve it!

readonly customModules?: { [moduleName: string]: string };

/**
* Optional flag to set for rewriting imports in alpha packages. When true, this will rewrite imports of generated L1s to reference aws-cdk-lib.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to provide a tiny example in the tsdoc.

/**
* Optional flag to set for rewriting imports in alpha packages. When true, this will rewrite imports of generated L1s to reference aws-cdk-lib.
*/
readonly isInAlphaPackage?: boolean;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Name feels leaky. Not sure aws-cdk-migration should be aware of alpha packages.
A more direct name for what this feature does would be better.

Something like useCfnConstructs or useBaseCfnConstructs?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opted for rewriteCfnImports to keep it as specific as possible. Let me know what you think

@madeline-k
Copy link
Copy Markdown
Contributor Author

I am running into an issue now with imports like this:

https://github.com/aws/aws-cdk/blob/master/packages/@aws-cdk/aws-redshift/test/cluster.test.ts#L6

import { CfnCluster, Cluster, ClusterParameterGroup, ClusterSubnetGroup, ClusterType } from '../lib';

This one is currently failing the PR Build, and there are probably more.

I see two solutions here:

  1. enforce in v1 that Cfn imports are separate, and actually directly reference the generated file, not just lib. That would rewrite the above to the following on master, and have a linter rule that requires Cfn imports to be separate. With this option, the logic I've written here won't have to change.
import { Cluster, ClusterParameterGroup, ClusterSubnetGroup, ClusterType } from '../lib';
import { CfnCluster } from '../lib/redshift.generated';
  1. In rewrite.ts and copy-files-removing-deps.ts, write logic to directly transform to the below when running transform.sh.
import { Cluster, ClusterParameterGroup, ClusterSubnetGroup, ClusterType } from '../lib';
import { CfnCluster } from 'aws-cdk-lib/aws-redshift';

I think option 2 would be ideal, but I am concerned that changing one line into two in the v2 rewrite tools will get messy.

@njlynch
Copy link
Copy Markdown
Contributor

njlynch commented Sep 7, 2021

  1. enforce in v1 that Cfn imports are separate, and actually directly reference the generated file, not just lib. That would rewrite the above to the following on master, and have a linter rule that requires Cfn imports to be separate. With this option, the logic I've written here won't have to change.

It appears there are 11 usages of loading Cfn* types from something that's not @aws-cdk/core or *.generated:

% git grep "import .*\bCfn.*" | grep -v generated | grep -v '@aws-cdk/core'
packages/@aws-cdk/aws-appmesh/lib/header-match.ts:import { CfnRoute } from './index';
packages/@aws-cdk/aws-athena/test/athena.test.ts:import { CfnWorkGroup } from '../lib';
packages/@aws-cdk/aws-athena/test/integ.workgroup.ts:import { CfnWorkGroup } from '../lib';
packages/@aws-cdk/aws-cloudfront/test/test-origin.ts:import { CfnDistribution, IOrigin, OriginBase, OriginBindConfig, OriginBindOptions, OriginProps, OriginProtocolPolicy } from '../lib';
packages/@aws-cdk/aws-iam/test/policy.test.ts:import { AnyPrincipal, CfnPolicy, Group, Policy, PolicyDocument, PolicyStatement, Role, ServicePrincipal, User } from '../lib';
packages/@aws-cdk/aws-msk/lib/cluster.ts:import { CfnCluster, KafkaVersion } from './';
packages/@aws-cdk/aws-redshift/test/cluster.test.ts:import { CfnCluster, Cluster, ClusterParameterGroup, ClusterSubnetGroup, ClusterType } from '../lib';
packages/@aws-cdk/aws-sam/test/application.test.ts:import { CfnApplication } from '../lib';
packages/@aws-cdk/cfnspec/lib/index.ts:import { CfnLintFileSchema } from './_private_schema/cfn-lint';
packages/awslint/lib/rules/module.ts:import { CfnResourceReflection } from './cfn-resource';
packages/awslint/lib/rules/resource.ts:import { CfnResourceReflection } from './cfn-resource';

Of those, 7 are only importing the L1s, so there's only 4 statements that would conflict with the proposed rule (as long as we exclude imports from @aws-cdk/core).

However, I worry a bit about us dodging this problem for named imports and missing the complexity of module imports:

import * as foo from '@aws-cdk/aws-foo';
/// ...
new foo.Flubber(...);
new foo.CfnFizzBuzz(...);
///

The above case will also require us to create the new import statements. If we're going to have to do it for that case, we might just want to handle it for both cases.

So Option 2 is probably preferable in the long run.

@aws-cdk-automation
Copy link
Copy Markdown
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
  • Commit ID: 1c03976
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@madeline-k madeline-k changed the base branch from v2-main to master September 13, 2021 22:42
@madeline-k madeline-k changed the base branch from master to v2-main September 13, 2021 22:51
@madeline-k
Copy link
Copy Markdown
Contributor Author

After a lot of investigation into the options here, I think the best thing is to require that the L1 imports that we support rewriting in this tool are in a "normal" format. The three options are:

  1. enforce in v1 that Cfn imports are separate, and actually directly reference the generated file, not just lib. That would rewrite the above to the following on master, and have a linter rule that requires Cfn imports to be separate. With this option, the logic I've written here won't have to change. Implemented in this PR: chore(individual-pkg-gen): remove L1s from alpha modules #16481

  2. In rewrite.ts and copy-files-removing-deps.ts, write logic to directly transform Cfn imports using text manipulation. I implemented this one in another PR: chore(individual-pkg-gen): remove L1s from alpha modules, and support… #16482. It is super hacky, and I don't really trust it to not cause headaches for us in the future. Open to suggestions on how to make this one better.

  3. In rewrite.ts and copy-files-removing-deps.ts, write logic to directly transform the Cfn imports (and all imports that need updating) using ts.updateXX() apis to manipulate the AST nodes. I explored this option in this PR: chore(individual-pkg-gen): remove L1s from alpha modules, and rewrite… #16483 It is not really working fully yet. But I have hit a roadblock, more description in the PR.

I think we should go with option 1. @njlynch, @rix0rrr, @nija-at, thoughts?

Closing this PR in favor of one of the above 3 :)

@nija-at
Copy link
Copy Markdown
Contributor

nija-at commented Sep 14, 2021

Option 1 seems the best to me. Trying to manipulate typescript text or nodes is just going to make this harder. Option 3 if it is simple and clean. Option 2 just sounds bad.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contribution/core This is a PR that came from AWS.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants