Describe the issue

Below is a long-winded way of saying:

Short version

What did you try?

Read CONTRIBUTING.md and follow its instructions for packaging a module in aws-cdk

What happened?

I was eventually able to run yarn run package but did not find any python packages being produced

What behavior did you expect:

To understand how to locally work in the repo then produce packages that I can use for local testing/development and (most importantly to me) understand how to do this for v2 of this lib.

Details

I'm trying to get ready to contribute some updates to this project but, before I dive in, I wanted to get familiar and get an idea of how I can try out local changes and package them to see if they will do what I need in a Python based app. From reading CONTRIBUTING.md, my understanding of how best to go from zero to python-package would be more or less (after installing node/yarn/.NET/python/docker:

$ git clone https://github.com/{your-account}/aws-cdk.git
$ cd aws-cdk
$ yarn install
$ cd packages/@aws-cdk/aws-ec2
$ ../../../scripts/buildup  # Unclear to me if this is necessary but it doesn't hurt
$ cd <root-of-cdk-repo>
$ docker run --rm --net=host -it -v $PWD:$PWD -w $PWD jsii/superchain
docker$ cd packages/@aws-cdk/aws-ec2
docker$ ../../../scripts/foreach.sh --up yarn run package

That last step is where things go wonky. I spent a few days off and on trying to get that to work and here is the short version of what I found:
Running things as described reliably produces an error for me:

aws-cdk/packages/aws-cdk: yarn run package  (12 remaining)
yarn run v1.22.5
$ cdk-package
Validating circular imports
Error: spawnSync .../aws-cdk/node_modules/esbuild-linux-64/bin/esbuild ENOBUFS
Package failed. Total time (2.5s)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error: last command failed. fix problem and resume by executing: .../dev/aws-cdk/packages/@aws-cdk/aws-ec2/../../../scripts/foreach.sh
directory:    .../aws-cdk/packages/aws-cdk

After a little digging, I found that the :latest version of the jsii docker referenced in CONTRIBUTING.md uses node10, while the docs say that >=14.15.0 is required. If I instead use :1-buster-slim-node14, that gets me further, but another change between versions is that :latest runs as root, while the latter does not, so it runs into a number of file permissions issues trying to operate on folders/files that the docker user does not own. The most reliable way I was able to get past that was to force the user back to root with the --user root arg for docker run (not ideal, obviously). So, changing the package part of the process, gets me a successful package run:

$ docker run --rm --net=host -it --user root -v $PWD:$PWD -w $PWD jsii/superchain:1-buster-slim-node14
docker$ cd packages/@aws-cdk/aws-ec2
docker$ ../../../scripts/foreach.sh --up yarn run package

But wait, there's more!

Looking at the command output, I was excited to go find my python packages that I could theoretically go test in a demo app, but to my surprise, there was js, dotnet, and java output in all the relevant dist/ folders, but nothing for python. Monitoring the /tmp/foreach.stdio file during another attempt showed me that it did 'finish' the python packaging, though I can find no artifacts of that having happened 😕 .

So that leaves me with two problems:

1. I can't seem to produce a package for python in the module used as an example for CONTRIBUTING.md
2. Once I manage that, I'm not sure how to go about doing the same for cdk v2

Links

CONTRIBUTING.md