chore: remodel the aws-cdk repo to support v2 development only (#24376)

Restructures the repository to reflect the current state of package publishing. Combines all of the stable packages within @aws-cdk/* namespace into aws-cdk-lib, and separates the unstable portions of the alpha modules into their own packages depending on `aws-cdk-lib` for their L1s. Changes how code generation works to run during the build of `aws-cdk-lib` and new cloudformation resources are added automatically when the spec is updated.

Removed all of the integration tests and put them into their own package to avoid cyclic dependencies. Integ tests for alpha packages remain with their package source.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

Author: madeline-k

CONTRIBUTING.md

@@ -88,35 +88,35 @@ The CDK uses [jsii](https://github.com/aws/jsii/) as its primary build system. j
 typescript-compliant source code and produce polyglot libraries, such as, in Java, .NET, Python and Go.
 
 The repo contains `packages/` directory that contains the CDK public modules. The source code for the IAM module in the
-CDK can be found at the location `packages/@aws-cdk/aws-iam`.
+CDK can be found at the location `packages/aws-cdk-lib/aws-iam`.
 The repo also contains the `tools/` directory that holds custom build tooling (modeled as private npm packages)
 specific to the CDK.
 
 ### Build
 
-The full build of the CDK takes a long time to complete; 1-2 hours depending on the performance of the build machine.
-However, most first time contributions will require changing only one CDK module, sometimes two. A full build of the
-CDK is not required in these cases.
-
-If you want to work on the `@aws-cdk/aws-ec2` module, the following command will build just the EC2 module and any
-necessary dependencies.
+The full build of all of the packages within the repository can take a few minutes, about 20  when all tests are run.
+Most contributions only require working on a single package, usually `aws-cdk-lib`. To build this package for the first
+time, you can execute the following to build it and it's dependencies.
 
 ```console
-$ cd packages/@aws-cdk/aws-ec2
+$ cd packages/aws-cdk-lib
 $ ../../../scripts/buildup
 ```
 
 Note: The `buildup` command is resumable. If your build fails, you can fix the issue and run `buildup --resume` to
 resume.
 
-At this point, you can run build and test the `aws-ec2` module by running
+At this point, you can run build and test the `aws-cdk-lib` module by running
 
 ```console
-$ cd packages/@aws-cdk/aws-ec2
+$ cd packages/aws-cdk-lib
 $ yarn build
 $ yarn test
 ```
 
+To cut down on iteration time as you develop, you can run `yarn watch` within the `aws-cdk-lib` directory to keep
+some of the build state in memory and incrementally rebuild as you make changes.
+
 However, if you wish to build the entire repository, the following command will achieve this.
 
 ```console
@@ -137,12 +137,12 @@ Packing involves generating CDK code in the various target languages and packagi
 respective package managers. Once in a while, these will need to be generated either to test the experience of a new
 feature, or reproduce a packaging failure.
 
-To package a specific module, say the `@aws-cdk/aws-ec2` module:
+To package a specific module, say the `aws-cdk-lib` module:
 
 ```console
 $ cd <root-of-cdk-repo>
 $ docker run --rm --net=host -it -v $PWD:$PWD -w $PWD jsii/superchain:1-buster-slim
-docker$ cd packages/@aws-cdk/aws-ec2
+docker$ cd packages/aws-cdk-lib
 docker$ ../../../scripts/foreach.sh --up yarn run package
 docker$ exit
 ```
@@ -317,12 +317,12 @@ CDK integration tests.
 We've added a watch feature to the CDK that builds your code as you type it. Start this by running `yarn watch` for
 each module that you are modifying.
 
-For example, watch the EC2 and IAM modules in a second terminal session:
+For example, watch the aws-cdk-lib and aws-cdk modules in a second terminal session:
 
 ```console
-$ cd packages/@aws-cdk/aws-ec2
+$ cd packages/aws-cdk-lib
 $ yarn watch & # runs in the background
-$ cd packages/@aws-cdk/aws-iam
+$ cd packages/aws-cdk
 $ yarn watch & # runs in the background
 ```
 
@@ -637,10 +637,10 @@ The README file contains code snippets written as typescript code. Code snippets
 (such as `` ```ts ``) will be automatically extracted, compiled and translated to other languages when the
 during the [pack](#pack) step. We call this feature 'rosetta'.
 
-You can run rosetta on the EC2 module (or any other module) by running:
+You can run rosetta on the aws-cdk-lib module (or any other module) by running:
 
 ```console
-$ cd packages/@aws-cdk/aws-ec2
+$ cd packages/aws-cdk-lib
 $ yarn rosetta:extract --strict
 ```
 
@@ -696,14 +696,14 @@ cases where some of those do not apply - good judgement is to be applied):
 - Types from the documented module should be **un-qualified**:
 
   ```ts
-  // An example in the @aws-cdk/core library, which defines Duration
+  // An example in the aws-cdk-lib library, which defines Duration
   Duration.minutes(15);
   ```
 
 - Types from other modules should be **qualified**:
 
   ```ts
-  // An example in the @aws-cdk/core library, using something from @aws-cdk/aws-s3
+  // An example in the aws-cdk-lib library, using something from aws-cdk-lib/aws-s3
   const bucket = new s3.Bucket(this, 'Bucket');
   // ...rest of the example...
   ```
@@ -712,7 +712,7 @@ cases where some of those do not apply - good judgement is to be applied):
   necessary for compilation but unimportant to the example:
 
   ```ts
-  // An example about adding a stage to a pipeline in the @aws-cdk/pipelines library
+  // An example about adding a stage to a pipeline in the aws-cdk-lib/pipelines library
   declare const pipeline: pipelines.CodePipeline;
   declare const myStage: Stage;
   pipeline.addStage(myStage);
@@ -766,7 +766,7 @@ Consequently, there are two useful scripts that are built on top of `foreach.sh`
 All linters are executed automatically as part of the build script, `yarn build`.
 
 They can also be executed independently of the build script. From the root of a specific package (e.g.
-`packages/@aws-cdk/aws-ec2`), run the following command to execute all the linters on that package -
+`packages/aws-cdk-lib`), run the following command to execute all the linters on that package -
 
 ```bash
 yarn lint
@@ -878,88 +878,24 @@ $ cdk -a some.app.js synth | $awscdk/scripts/template-deps-to-dot | dot -Tpng >
 
 You can use `find-cycles` to print a list of internal dependency cycles:
 
-```shell
-$ scripts/find-cycles.sh
-Cycle: @aws-cdk/aws-iam => @aws-cdk/assert => aws-cdk => @aws-cdk/aws-s3 => @aws-cdk/aws-kms => @aws-cdk/aws-iam
-Cycle: @aws-cdk/assert => aws-cdk => @aws-cdk/aws-s3 => @aws-cdk/aws-kms => @aws-cdk/assert
-Cycle: @aws-cdk/aws-iam => @aws-cdk/assert => aws-cdk => @aws-cdk/aws-s3 => @aws-cdk/aws-iam
-Cycle: @aws-cdk/assert => aws-cdk => @aws-cdk/aws-s3 => @aws-cdk/assert
-Cycle: @aws-cdk/assert => aws-cdk => @aws-cdk/aws-cloudformation => @aws-cdk/assert
-Cycle: @aws-cdk/aws-iam => @aws-cdk/assert => aws-cdk => @aws-cdk/util => @aws-cdk/aws-iam
-Cycle: @aws-cdk/aws-sns => @aws-cdk/aws-lambda => @aws-cdk/aws-codecommit => @aws-cdk/aws-sns
-Cycle: @aws-cdk/aws-sns => @aws-cdk/aws-lambda => @aws-cdk/aws-codecommit => @aws-cdk/aws-codepipeline => @aws-cdk/aws-sns
-```
-
 ## Running CLI integration tests
 
 The CLI package (`packages/aws-cdk`) has some integration tests that aren't
 run as part of the regular build, since they have some particular requirements.
 See the [CLI CONTRIBUTING.md file](packages/aws-cdk/CONTRIBUTING.md) for
 more information on running those tests.
 
-## Building aws-cdk-lib
-
-In AWS CDK v2, all stable libraries are packaged into a single monolithic
-package and published as `aws-cdk-lib`. In most cases, you can iterate on a
-single module's directory as previously described in this document (e.g.
-`packages/@aws-cdk/aws-s3`). In some cases, you might need to build
-`aws-cdk-lib`:
-
-```
-# Generate all of the L1s first. If you have already done a full build in the repository, you can skip this.
-cd <CDK repo root>/
-./scripts/gen.sh
-
-# Generate and build `aws-cdk-lib`
-cd packages/aws-cdk-lib
-yarn build
-```
-
-The commands above perform the following steps:
-1. Run `yarn install` to install all dependencies
-2. Generate `.generated.ts` files in each `packages/@aws-cdk/aws-<service>`
-   directory. These files contain TypeScript source code for all of the L1 (Cfn)
-   Constructs, and are generated from the [CloudFormation Resource
-   Specification](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html).
-3. Copy the `.ts` source code files from each `packages/@aws-cdk/aws-<service>`
-   directory to the corresponding `packages/aws-cdk-lib/aws-<service>`
-   directory.
-4. Compile `aws-cdk-lib`.
-
-Running unit tests and integration tests still has to be performed in each
-module's `packages/@aws-cdk` directory.
-
 ## Building and testing v2 -alpha packages
 
-In AWS CDK v2, all experimental libraries are published separately with an
--alpha suffix. In most cases, you can iterate on a single module's directory as
-already described in this document (e.g. `packages/@aws-cdk/aws-amplify`). If
-you need to generate and iterate on the alpha package, here are the steps. The
-main differences between the alpha package is naming of the package, and import
-statements.
-
-First, make sure the following packages are built:
-  - packages/@aws-cdk/assert
-  - packages/aws-cdk-lib
-  - tools/individual-pkg-gen
-
-The following command will create all of the alpha packages by copying files
-from their source directories under `packages/@aws-cdk/aws-<service>`, and it
-will build and run unit tests for all of them. This is sometimes too much for a
-developer machine or laptop.
+Modules that are not stable are vended separately from `aws-cdk-lib`. These packages are found in the
+`packages/@aws-cdk` directory and are marked `stability: 'experimental'` in their package.json files.
+This means they will be given the `alpha` version from the `version.v2.json` when published and they
+cannot be taken as dependencies by `aws-cdk-lib`
 
-```
-<CDK repo root>/scripts/transform.sh
-```
-
-To only copy and transform the source files, and then build and test one
-alpha package at a time, use the following:
+Experimental packages are used to develop new constructs and experiment with their APIs before marking
+them as stable and including them within `aws-cdk-lib`. Once they are included in `aws-cdk-lib`, no
+more breaking api changes can be made.
 
-```
-<CDK repo root>/scripts/transform.sh --skip-build
-cd packages/individual-packages/aws-<service>
-yarn build+test
-```
 ## Changing Cloud Assembly Schema
 
 If you plan on making changes to the `cloud-assembly-schema` package, make sure you familiarize yourself with

buildspec-pr.yaml

@@ -25,7 +25,6 @@ phases:
   build:
     commands:
       - /bin/bash ./build.sh
-      - /bin/bash ./scripts/transform.sh
       # After compilation, run Rosetta (using the cache if available).
       # This will print errors, and fail the build if there are compilation errors in any packages marked as 'strict'.
       - /bin/bash ./scripts/run-rosetta.sh

buildspec-remodel.yaml

@@ -0,0 +1,41 @@
+version: 0.2
+
+env:
+  secrets-manager:
+    TOKEN: $TOKEN_SECRET_ARN
+  variables:
+    NODE_OPTIONS: --max_old_space_size=8192
+
+phases:
+  install:
+    commands:
+      # CodeBuild always runs as root, allow npm to operate as such
+      - npm config set unsafe-perm true
+      - mkdir ${HOME}/aws-cdk && cd ${HOME}/aws-cdk
+      # Configure git and git credentials
+      - git config --global user.email "${COMMIT_EMAIL}"
+      - git config --global user.name "${COMMIT_USERNAME}"
+      - echo "Retrieving gh token from secretsmanager"
+      - git clone https://[email protected]/aws/aws-cdk -b feat/repo-restructure .
+      - git checkout ${CODEBUILD_RESOLVED_SOURCE_VERSION}
+      # Install yarn if it wasn't already present in the image
+      - yarn --version || npm -g install yarn
+      - yarn install --frozen-lockfile
+      - /sbin/sysctl -w vm.max_map_count=2251954
+  build:
+    commands:
+      - cd ${HOME}/aws-cdk
+      - npx lerna run build --scope @aws-cdk/remodel --include-dependencies
+      - npx remodel ${PWD} --tmp-dir ${HOME}/remodel --no-clean --dry-run
+  post_build:
+    commands:
+      - cd ${HOME}/remodel
+      - git add -A
+      - git commit -m "Execute remodel"
+      - git remote add aws-cdk https://[email protected]/aws/aws-cdk
+      - git fetch aws-cdk
+      - git push aws-cdk +feat/repo-restructure:feat/remodel
+artifacts:
+  files:
+    - "**/*"
+  base-directory: ${HOME}/remodel

buildspec.yaml

@@ -27,7 +27,6 @@ phases:
       - 'if ${BUMP_CANDIDATE:-false}; then /bin/bash ./scripts/bump-candidate.sh; fi'
       - /bin/bash ./scripts/align-version.sh
       - /bin/bash ./build.sh
-      - /bin/bash ./scripts/transform.sh
   post_build:
     commands:
       # Short-circuit: Don't run pack if the above build failed.

packages/@aws-cdk-testing/framework-integ/.eslintrc.js

@@ -0,0 +1,13 @@
+const baseConfig = require('../../../tools/@aws-cdk/cdk-build-tools/config/eslintrc');
+baseConfig.parserOptions.project = __dirname + '/tsconfig.json';
+module.exports = {
+  ...baseConfig,
+  ignorePatterns: [
+    ...baseConfig.ignorePatterns,
+    '**/*.snapshot/**/*'
+  ],
+  rules: {
+    ...baseConfig.rules,
+    'import/order': 'off',
+  }
+};

packages/@aws-cdk-testing/framework-integ/.gitignore

@@ -0,0 +1,8 @@
+
+.LAST_BUILD
+*.snk
+junit.xml
+!.eslintrc.js
+.nyc_output
+coverage
+nyc.config.js

packages/@aws-cdk-testing/framework-integ/.npmignore

@@ -0,0 +1,7 @@
+
+.LAST_BUILD
+*.snk
+junit.xml
+.eslintrc.js
+# exclude cdk artifacts
+**/cdk.out

packages/@aws-cdk/alexa-ask/LICENSE renamed to packages/@aws-cdk-testing/framework-integ/LICENSE

@@ -0,0 +1,16 @@
+AWS Cloud Development Kit (AWS CDK)
+Copyright 2018-2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+Third party attributions of this package can be found in the THIRD_PARTY_LICENSES file

packages/@aws-cdk-testing/framework-integ/NOTICE

@@ -0,0 +1,63 @@
+{
+  "name": "@aws-cdk-testing/framework-integ",
+  "description": "Integration tests for aws-cdk-lib",
+  "private": true,
+  "version": "0.0.0",
+  "scripts": {
+    "build": "cdk-build",
+    "watch": "cdk-watch",
+    "lint": "cdk-lint",
+    "pkglint": "pkglint -f",
+    "test": "cdk-test",
+    "package": "cdk-package",
+    "build+test": "yarn build && yarn test",
+    "build+extract": "yarn build",
+    "build+test+package": "yarn build+test && yarn package",
+    "build+test+extract": "yarn build+test"
+  },
+  "pkglint": {
+    "exclude": [
+      "package-info/repository",
+      "dependencies/cdk-point-dependencies"
+    ]
+  },
+  "author": {
+    "name": "Amazon Web Services",
+    "url": "https://aws.amazon.com",
+    "organization": true
+  },
+  "license": "Apache-2.0",
+  "devDependencies": {
+    "@aws-cdk/cdk-build-tools": "0.0.0",
+    "delay": "5.0.0",
+    "@aws-cdk/pkglint": "0.0.0"
+  },
+  "dependencies": {
+    "@aws-cdk/lambda-layer-kubectl-v24": "^2.0.100",
+    "aws-cdk-lib": "0.0.0",
+    "aws-sdk": "^2.1317.0",
+    "aws-sdk-mock": "5.6.0",
+    "cdk8s": "^2.7.15",
+    "cdk8s-plus-24": "2.4.40",
+    "constructs": "^10.0.0",
+    "@aws-cdk/integ-tests-alpha": "0.0.0"
+  },
+  "repository": {
+    "url": "https://github.com/aws/aws-cdk.git",
+    "type": "git",
+    "directory": "packages/@aws-cdk-testing/framework-integ"
+  },
+  "keywords": [
+    "aws",
+    "cdk"
+  ],
+  "homepage": "https://github.com/aws/aws-cdk",
+  "engines": {
+    "node": ">= 14.15.0"
+  },
+  "stability": "experimental",
+  "maturity": "experimental",
+  "publishConfig": {
+    "tag": "latest"
+  }
+}