Deployment of a Private Communication in a MultiCloud Environment (AWS and GCP) 100% Automated Using Terraform

9 min readSep 28, 2024

--

Deployment of a Private Communication in a MultiCloud Environment (AWS and GCP) 100% Automated Using Terraform

Project Description

In this real-world scenario, I acted as a Cloud Specialist tasked with interconnecting AWS and Google Cloud Platform (GCP) services into a single, private, and secure architecture. The company had existing architectures in both AWS and GCP. Still, they needed these architectures to communicate securely and privately through services like Virtual Private Gateway, Customer Gateway, Cloud Routers, Cloud VPN, and more.

To meet the strict one-week deadline for production readiness, I designed the solution architecture and opted for a fully automated deployment using Terraform. The automation process ensured that complex, multi-cloud infrastructure could be deployed rapidly, with consistent results and minimal manual intervention.

This hands-on project was a significant opportunity to implement real-world multi-cloud solutions, using the power of Infrastructure as Code (IaC) with Terraform.

Solution Architecture — Implementing Private Multi-Cloud Communication Between AWS and GCP Using Terraform for Infrastructure as Code (IaC) Automation

Key Steps Performed:

Set up AWS and GCP accounts.
Created a GCP project and enabled necessary services.
Configured Cloud Shell and downloaded project files.
Generated and configured GCP and AWS credentials.
Installed Terraform and set up project configurations.
Created and imported SSH key pairs for secure access.
Initialized and validated Terraform configurations.
Deployed the infrastructure using Terraform.
Tested private communication between AWS and GCP instances.

Step 1: Creating a GCP Project

Project Name: tcb-gcp-aws

Step 2: Enable Billing

Ensure billing is enabled for the created GCP project.

Step 3: Enabling the Compute Engine API

Step 4: Authorize GCP Cloud Shell

Step 5) Project Automation Folder Files

Run the [chmod +x *.sh] to grant execute permission to all [.sh] script files within the directory, making them executable.

All .sh script files have been modifed with permissiom set for executable

Step 6 Creating credentials in the GCP

[ ./gcp_set_credentials.sh ~/file.json ]

To create credentials in GCP, start by making a “service account key” in JSON format, which acts as a special key for secure access to your Google Cloud resources.

Download this JSON file to your computer, then upload it to your Cloud Shell environment. Once uploaded, run the command to let your Cloud Shell use the key, so you can access and control your Google Cloud resources securely.

The purpose of this script is to automate and simplify the process of setting up the GCP credentials for Terraform, ensuring your configurations are properly backed up and all variables are set up for seamless cloud resource management.

Check for Key File: The script verifies whether a JSON key file for the GCP service account has been provided. If not found, it returns an error.
Backup Existing Credentials: Before using the new key, any existing credentials are backed up to ensure the previous setup is preserved.
Create New Credentials File: The (.json) key file is copied to the appropriate location (~/ .config/gcloud) to set up secure access for GCP.
Set Up Terraform Variables: A ( terraform.tfvars ) file is created if it doesn’t already exist, storing important information for Terraform to use the credentials.
Add Key Path to Variables File: The path to the (.json) key file is added to ( terraform.tfvars ) so that Terraform can find and use the correct credentials.
Point Terraform to the Credentials: Terraform’s configuration is updated to use the new credentials, enabling secure interactions with GCP resources.

Step 7 Creating Credentials in AWS (Region: Oregon — us-west-2)

AWS console | IAM | Users | Add user | Programmatic access
Attach policy: AdministratorAccess
Download the .csv (credentials file)
Renaming the .csv file to accessKeys.csv
Uploading the file (key) .csv to the Cloud Shell.
Running the command: ./aws_set_credentials.sh ~/accessKeys.csv

The script reads the accessKeys.csv file to have granted the permissions to set up AWS credentials on your system, allowing secure access to AWS resources. This process typically configures your environment so that any tools or services (like AWS CLI) can authenticate with AWS using the credentials provided in the file.

Download the .csv (credentials file) / rename the .csv file to accessKeys.csv

Creating Credentials terraform IAM | Users | Add user | Programmatic access

This file contains access keys, which include your AWS Access Key ID and Secret Access Key. These keys are necessary for programmatic access to AWS services.

Here is a high-level summary of what the ./aws_set_credentials.sh script accomplishes:

Checks for Key File: Verifies if you have provided an accessKeys.csv file containing AWS access keys. If missing, it shows an error message.
Backs Up Credentials: Before making any changes, it creates a backup of the existing AWS credentials to ensure that the previous settings are preserved.
Creates Credentials File: It initializes a new AWS credentials file, copying over the access keys from the accessKeys.csv file. The new file is placed in the ~/.aws directory.
Adds AWS Secrets: Extracts the Access Key ID and Secret Access Key from the accessKeys.csv and adds them to the credentials file under a [default] profile.
Creates Terraform Variables File: It sets up a terraform.tfvars file (if it doesn't exist), which will store the path to the AWS credentials for Terraform to use.
Updates Terraform Variables: Adds a key-value pair to the terraform.tfvars file to let Terraform know where to find the AWS credentials.

Step 8 Getting the Terraform ready

Change directory folder to hands-on-tcb-bmc-gcp
Run the command: ./get_terraform.sh

The script ( ./get_terraform.sh ) performs the following tasks:

Check for Project Argument: The script checks if a GCP project ID is provided as an argument. If missing, it throws an error.
Update Project ID in Config File: If the autonetdeploy_config.yaml file exists, it updates the project_id with the given GCP project ID.
Create Terraform Variables File: A terraform.tfvars file is created if it doesn’t already exist, initializing it for Terraform.
Add Project ID to Terraform Variables: The script adds the GCP project ID as a key-value pair in the terraform.tfvars file, which is used by Terraform.
Apply Project Configurations: The script applies these configurations across relevant files, ensuring that the GCP project ID is set properly for both Deployment Manager and Terraform settings.

Step 9 Setting up the GCP Project

From Cloud Shell, run the command:
gcloud config set project [PROJECT-ID]
gcloud config set project tcb-gcp-aws-0922

Run the script to set the project:
./gcp_set_project.sh
Check for GCP Project Argument: Verify if the GCP project ID is provided as an argument. If not, the script throws an error.
Update Project ID in Config File: If the autonetdeploy_config.yaml file exists, updates its project_id field to match the provided GCP project ID.
Create Terraform Variables File: Creates a terraform.tfvars file (if it doesn’t already exist) to store key information for Terraform.
Add Project ID to Terraform Variables: Adds the GCP project ID as a key-value pair in the terraform.tfvars file, making it accessible for Terraform.
Apply Project Configurations: The script checks for the relevant files and directories, then adds or updates the project configuration as needed for both Deployment Manager and Terraform.
Check if the project ID was inserted into the Terraform variables file:
cat /home/[USERNAME]/hands-on-tcb-bmc-gcp/terraform/terraform.tfvars

Check if the **project ID** was inserted into the Terraform variables file:

Step 10 Generating a Key Pairs

Get Your Username: Run whoami to find out your current system username.
Generate SSH Key: Use ssh-keygen -t rsa -f ~/.ssh/vm-ssh-key -C [YOUR-USERNAME] to create an RSA SSH key pair, replacing [YOUR-USERNAME] with your actual username.
Set Private Key Permissions: Run chmod 400 ~/.ssh/vm-ssh-key to set secure permissions for the private key, ensuring it's only accessible by you.

**Generate SSH Key**: Use `ssh-keygen -t rsa -f ~/.ssh/vm-ssh-key -C`

Set Private Key Permissions: Run chmod 400 ~/.ssh/vm-ssh-key to set

**List the priavte and public keys** ( **ls -ltr** )

Step 11 Importing the public key to GCP

Run SSH Configuration Command: Use gcloud compute config-ssh --ssh-key-file=~/.ssh/vm-ssh-key from Cloud Shell to configure SSH access with your key.
Check SSH Key Import: Go to the metadata page to confirm that your SSH key was successfully imported.
Verify SSH Key: Click on the SSH Keys tab in the metadata to view your imported SSH key. 😎

**Check SSH Key Import**: **Go to the metadata page to confirm that your SSH key was successfully imported.**

Step 12 Importing the public key to GCP

Step 13 Importing the public key to AWS

Download the Public Key: Download the public key file generated in Cloud Shell from the path /home/[YOUR-USERNAME]/.ssh/vm-ssh-key.pub.
Go to AWS EC2 Console: Navigate to AWS Console > EC2 > Network & Security > Key Pairs.
Name the Key Pair: Use the name vm-ssh-key for the key pair.
Upload the Public Key: Browse for the file vm-ssh-key.pub, select it, and upload it to associate with the key pair in AWS.

**Import AWS Console** > **EC2** > **Network & Security** > **Key Pairs**.

**Verify SSH Key**: Click on the **SSH Keys** tab in the metadata to view your imported SSH key.

Step 14 It’s Terraform Time

Navigate to the Hands-On Directory: From Cloud Shell, ensure you’re in the hands-on-tcb-bmc-gcp folder.

**Navigate to the Hands-On Directory**: From **Cloud Shell**, ensure you’re in the `hands-on-tcb-bmc-gcp` folder.

Move to Terraform Folder: Use cd terraform to access the Terraform directory.
Initialize Terraform: Run terraform init to initialize the Terraform environment.

**Move to Terraform Folder**: Use `cd terraform` to access the Terraform directory.

Validate Terraform Configuration: Use terraform validate to check for any configuration errors.

**Validate Terraform Configuration**: Use `terraform validate` to check for any configuration errors.

Check Terraform Deployment Plan: Execute terraform plan to review the plan for deployment.

**Check Terraform Deployment Plan**: Execute `terraform plan` to review the plan for deployment.

Deploy Infrastructure with Terraform: Run terraform apply and type yes when prompted to confirm and start deployment.

**Deploy Infrastructure with Terraform**: Run `terraform apply` and type **yes** when prompted to confirm and start deployment.

(Optional) View Deployment Variables: Use terraform output to display any output variables after deployment.

**) View Deployment Variables**: Use `terraform output` to display any output variables after deployment.

Verify Terraform GCP deployment — check Instances & Resources

**Verify Terraform GCP IaC deployment —** check Instances & Resources

**Verify Terraform GCP deployment —** check Instances & Resources

Verify Terraform AWS deployment — check Instances & Resources

**Verify Terraform AWS deployment —** check Instances & Resources

Test Connectivity Between Instances: Follow the video instructions to test connectivity between instances created in both GCP and AWS.

(Optional) Complete Additional Testing Steps: Continue as per additional instructions or testing steps provided after the initial connectivity test.

Conclusion:

This project was an excellent real-world use case of interconnecting AWS and GCP in a private, secure, and automated manner.

Using Terraform ensured the entire Infrastructure as Code (IaC) deployment process was streamlined and repeatable, saving time and reducing errors.

Multi-cloud architectures are becoming more prevalent, and the ability to automate these deployments is essential for organizations seeking to leverage the strengths of both platforms.