.gitignore

@@ -7,4 +7,5 @@ example.com.key
 postgres.example.com.csr
 test.pem
 secret.tfvars
+ubuntu.pem
 

README.md

@@ -1,27 +1,25 @@
-[[_TOC_]]
+# Database Lab Terraform Module
 
-# How to setup Database Lab using Terraform in AWS
-
-This [Terraform Module](https://www.terraform.io/docs/language/modules/index.html) is responsible for deploying the [Database Lab Engine](https://gitlab.com/postgres-ai/database-lab) to cloud hosting providers.
+This [Terraform Module](https://www.terraform.io/docs/language/modules/index.html) can be used as a template for deploying the [Database Lab Engine](https://gitlab.com/postgres-ai/database-lab) to cloud hosting providers. Please feel free to tailor it to meet your requirements. 
 
 Your source PostgreSQL database can be located anywhere, but DLE with other components will be created on an EC2 instance under your AWS account. Currently, only "logical" mode of data retrieval (dump/restore) is supported – the only available method for managed PostgreSQL cloud services such as RDS Postgres, RDS Aurora Postgres, Azure Postgres, or Heroku. "Physical" mode is not yet supported, but it will be in the future. More about various data retrieval options for DLE: https://postgres.ai/docs/how-to-guides/administration/data.
 
-## Supported Cloud Platforms:
+## Supported Cloud Platforms
 - AWS
 
 ## Prerequisites
 - [AWS Account](https://aws.amazon.com)
 - [Terraform Installed](https://learn.hashicorp.com/tutorials/terraform/install-cli) (minimal version: 1.0.0)
 - AWS [Route 53](https://aws.amazon.com/route53/) Hosted Zone (For setting up TLS) for a domain or sub-domain you control
 - You must have AWS Access Keys and a default region in your Terraform environment (See section on required IAM Permissions)
-- The DLE runs on an EC2 instance which can be accessed using a selected set of SSH keys uploaded to EC2. Use the Terraform parameter `aws_keypair` to specify which EC2 Keypair to use
+- The DLE runs on an EC2 instance which can be accessed using a selected set of SSH keys uploaded to EC2.
 - Required IAM Permissions: to successfully run this Terraform module, the IAM User/Role must have the following permissions:
     * Read/Write permissions on EC2
     * Read/Write permissions on Route53
     * Read/Write permissions on Cloudwatch
 
-## Configuration overview
-- :construction: Currently, it is supposed that you run `terraform` commands on a Linux machine. MacOS and Windows support is not yet implemented (but planned).
+## How to use
+- :construction: Currently, it is supposed that you run `terraform` commands on a Linux machine or MacOS. Windows support is not yet implemented (but planned).
 - It is recommended to clone this Git repository and adjust for your needs. Below we provide the detailed step-by-step instructions for quick start (see "Quick start") for a PoC setup
 - To configure parameters used by Terraform (and the Database Lab Engine itself), you will need to modify `terraform.tfvars` and create a file with secrets (`secret.tfvars`)
 - This Terraform module can be run independently or combined with any other standard Terraform module. You can learn more about using Terraform and the Terraform CLI [here](https://www.terraform.io/docs/cli/commands/index.html)
@@ -31,7 +29,7 @@ Your source PostgreSQL database can be located anywhere, but DLE with other comp
     - values passed on the command line
 - All variables starting with `postgres_` represent the source database connection information for the data (from that database) to be fetched by the DLE. That database must be accessible from the instance hosting the DLE (that one created by Terraform)
 
-## How-to guide: using this Terraform module to set up DLE and its components
+## Quick start
 The following steps were tested on Ubuntu 20.04 but supposed to be valid for other Linux distributions without significant modification.
 
 1. SSH to any machine with internet access, it will be used as deployment machine
@@ -51,53 +49,45 @@ The following steps were tested on Ubuntu 20.04 but supposed to be valid for oth
     ```
 1. Edit `terraform.tfvars` file. In our example, we will use Heroku demo database as a source:
     ```config
-    dle_version_full = "2.4.1"
+    dle_version = "v3.0.1"
 
     aws_ami_name = "DBLABserver*"
-    aws_keypair = "YOUR_AWS_KEYPAIR"
 
     aws_deploy_region = "us-east-1"
     aws_deploy_ebs_availability_zone = "us-east-1a"
-    aws_deploy_ec2_instance_type = "t2.large"
+    aws_deploy_ec2_instance_type = "c5.large"
     aws_deploy_ec2_instance_tag_name = "DBLABserver-ec2instance"
-    aws_deploy_ebs_size = "40"
+    aws_deploy_ebs_size = "10"
+    aws_deploy_ec2_volumes_count = "2"
     aws_deploy_ebs_type = "gp2"
     aws_deploy_allow_ssh_from_cidrs = ["0.0.0.0/0"]
     aws_deploy_dns_api_subdomain = "tf-test" # subdomain in aws.postgres.ai, fqdn will be ${dns_api_subdomain}-engine.aws.postgres
 
-    # Source – two options. Choose one of two:
-    #    - direct connection to source DB
-    #    - dump stored on AWS S3
-
-    # option 1 – direct PG connection
-    source_type = "postgres" # source is working dome postgres database
     source_postgres_version = "13"
-    source_postgres_host = "ec2-3-215-57-87.compute-1.amazonaws.com" # an example DB at Heroku
+    source_postgres_host = "ec2-3-215-57-87.compute-1.amazonaws.com"
     source_postgres_port = "5432"
-    source_postgres_dbname = "d3dljqkrnopdvg" # an example DB at Heroku
-    source_postgres_username = "bfxuriuhcfpftt" # an example DB at Heroku
-    
-    # option 2 – dump on S3. Important: your AWS user has to be able to create IAM roles to work with S3 buckets in your AWS account
-    # source_type = 's3' # source is dump stored on demo s3 bucket
-    # source_pgdump_s3_bucket = "tf-demo-dump" # This is an example public bucket
-    # source_pgdump_path_on_s3_bucket = "heroku.dmp" # This is an example dump from demo database
-    
+    source_postgres_dbname = "d3dljqkrnopdvg" # this is an existing DB (Heroku example DB)
+    source_postgres_username = "bfxuriuhcfpftt" # in secret.tfvars, use:   source_postgres_password = "dfe01cbd809a71efbaecafec5311a36b439460ace161627e5973e278dfe960b7"
     dle_debug_mode = "true"
     dle_retrieval_refresh_timetable = "0 0 * * 0"
     postgres_config_shared_preload_libraries = "pg_stat_statements,logerrors" # DB Migration Checker requires logerrors extension
 
     platform_project_name = "aws_test_tf"
+
+    # list of ssh public keys stored in files
+    ssh_public_keys_files_list = ["~/.ssh/id_rsa.pub"]
+    # or provided inline
+    ssh_public_keys_list = [
+      "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDhbblazDXCFEc21DtFzprWC8DiqidnVRROzp6J6BeJR9+XydPUtl0Rt2mcNvxL5ro5bI9u5JRW8aDd6s+Orpr66hEDdwQTbT1wp5nyduFQcT3rR   +aeDSilQvAHjr4/z/GZ6IgZ5MICSIh5hJJagHoxAVqeS9dCA27tv/n2T2XrxIUeBhywH1EmfwrnEw97tHM8F+yegayFDI1nVOUWUIxFMaygMygix8uKbQ2fl4rkkxG2oEx7uyAFMXHt4bewNbZuAp8b/b5ODL6tGHuHhcwfbWGriCO+l7UOf1K9maTx00o4wkzAPyd+qs70y/1iMX2YOOLYaYYdptEnFal2DVoD example@example.com"
+      ]
     ```
 1. Create `secret.tfvars` containing `source_postgres_password`, `platform_access_token`, and `vcs_github_secret_token`. An example:
     ```config
-    source_postgres_password = "dfe01cbd809a71efbaecafec5311a36b439460ace161627e5973e278dfe960b7" # an example DB at Heroku
+    source_postgres_password = "YOUR_DB_PASSWORD" # todo: put pwd for heroku example DB here
     platform_access_token = "YOUR_ACCESS_TOKEN"   # to generate, open https://console.postgres.ai/, choose your organization,
                                                 # then "Access tokens" in the left menu
-    vcs_github_secret_token = "vcs_secret_token"  # generate a personal access token with scope of "repo"
+    vcs_github_secret_token = "vcs_secret_token"  # to generate, open https://github.com/settings/tokens/new
     ```
-    To generate a personal GitHub access token with the scope of "repo", open the [guide on GitHub Docs](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token) and follow the instructions.
-
-    Note that the "repo" scope essentially gives full access to all user-controlled repositories. Should you have any concerns about which repositories the DLE can have access to, consider using a separate GitHub account that has access to the reduced number of repositories.
 1. Initialize
     ```shell
     terraform init
@@ -108,21 +98,32 @@ The following steps were tested on Ubuntu 20.04 but supposed to be valid for oth
     export AWS_SECRET_ACCESS_KEY = "accesskey"
     ```
 1. Deploy:
-    ```shell
-    terraform  apply -var-file="secret.tfvars" -auto-approve
+    ```
+    terraform apply -var-file="secret.tfvars" -auto-approve && terraform output -raw next_steps
     ```
 1. If everything goes well, you should get an output like this:
     ```config
-    vcs_db_migration_checker_verification_token = "gsio7KmgaxECfJ80kUx2tUeIf4kEXZex"
-    dle_verification_token = "zXPodd13LyQaKgVXGmSCeB8TUtnGNnIa"
-    ec2_public_dns = "ec2-11-111-111-11.us-east-2.compute.amazonaws.com"
-    ec2instance = "i-0000000000000"
-    ip = "11.111.111.11"
-    platform_joe_signing_secret = "lG23qZbUh2kq0ULIBfW6TRwKzqGZu1aP"
-    public_dns_name = "demo-api-engine.aws.postgres.ai"  # todo: this should be URL, not hostname – further we'll need URL, with protocol – `https://`
-    ```
 
-1. To verify result and check the progress, you might want to connect to the just-created EC2 machine using IP address or hostname from the Terraform output. In our example, it can be done using this one-liner (you can find more about DLE logs and configuration on this page: https://postgres.ai/docs/how-to-guides/administration/engine-manage):
+    #####################################################################
+
+    Congratulations! Database Lab Engine installed.
+    Data initialization may take time, depending on the database size.
+
+    You should be able to work with all DLE interfaces already:
+    - [RECOMMENDED] UI: https://tf-test.aws.postgres.ai:446
+    - CLI: dblab init --url=https://tf-test.aws.postgres.ai --token=sDTPu17pzXhW9DkhcSGpAMj72KgiIJxG --environment="i-0687b060f45314be5" --insecure
+    - API: https://tf-test.aws.postgres.ai
+    - SSH connection for troubleshooting: ssh ubuntu@3.92.133.178 -i dmitry-DBLABserver-ec2instance.pem
+
+    (Use verification token: sDTPu17pzXhW9DkhcSGpAMj72KgiIJxG)
+
+    For support, go to https://postgres.ai/contact.
+
+    #####################################################################
+
+   ```
+
+1. To verify result and check the progress, you might want to connect to the just-created EC2 machine using IP address or hostname from the Terraform output and ssh key from ssh_public_keys_files_list and/or ssh_public_keys_list variables. In our example, it can be done using this one-liner (you can find more about DLE logs and configuration on this page: https://postgres.ai/docs/how-to-guides/administration/engine-manage):
     ```shell
     echo "sudo docker logs dblab_server -f" | ssh ubuntu@18.118.126.25 -i postgres_ext_test.pem
     ```
@@ -183,10 +184,14 @@ The following steps were tested on Ubuntu 20.04 but supposed to be valid for oth
     }
     ```
 
+This is it!
+
+If you need to remove everything created by this Terraform module, you can run `terraform destroy -var-file="secret.tfvars" -auto-approve`. Do not forget to do it if you're just experimenting. Otherwise, if you leave infrastructure blocks running, they might significantly affect your AWS bill (depending on the EC2 instance family you've chosen, the disk type, and size).
+
 ## Important Note
 When the DLE creates new database clones, it makes them available on incremental ports in the 6000 range (e.g. 6000, 6001, ...). The DLE CLI will also report that the clone is available on a port in the 6000 range.  However, please note that these are the ports when accessing the DLE from `localhost`.  This Terraform module deploys [Envoy](https://www.envoyproxy.io/) to handle SSL termination and port forwarding to connect to DLE generated clones.
 
-Bottom Line: When connecting to clones, add `3000` to the port number reported by the DLE CLI to connect to the clone. for example, if the CLI reports that a new clone is available at port `6001` connect that clone at port `9001`.
+Follow the [how-to guide](https://postgres.ai/docs/how-to-guides/administration/install-database-lab-with-terraform) to install Database Lab with Terraform on AWS
 
 ## Known Issues
 ### Certificate Authority Authorization (CAA) for your Hosted Zone