Skip to content

Create a simple redshift cluster using terraform to access S3 buckets

Notifications You must be signed in to change notification settings

nialloriordan/terraform-redshift-cluster

Repository files navigation

Redshift Cluster via Terraform

Table of Contents

Overview

The purpose of this repo is to provide an easy method to create, update and destroy redshift clusters via Terraform.

The redshift cluster will also be able to access AWS S3 buckets.

Terraform

Quick Start

Show/Hide Details

Ensure you have terraform cli installed and setup for local development. More information here

Rename terraform.tfvars.example as terraform.tfvars. Update as appropriate as this will contain the value for all relevant variables.

Run terraform init to initialise the terraform directory.

Run terraform plan to view the execution plan.

Run terraform apply to create the redshift cluster. Make sure to enter yes to confirm the execution.

Finally to delete the cluster run terraform destroy. Make sure to enter yes to confim the deletion and to destroy any other related objects.

Details

When you run terraform apply the following resources are being created:

  1. aws_iam_role: This is an IAM role that will be provided to Redshift to allow it to access other AWS resources
  2. aws_iam_role_policy_attachment: this will attach an s3 read access policy to the newly created IAM role. This will enable the redshift cluster to access S3 buckets.
  3. aws_redshift_cluster: The Amazon Redshift Cluster
  4. aws_vpc: An Amazon Virtual Private Cloud (VPC)
  5. aws_default_security_group: Opens a TCP connection within the VPC at the specified port. This will enable you to query the Redshift database at your provided IP address

Boto3

Show/Hide Details

For reference I have provided python scripts to create a redshift cluster via boto3.

In your environment of choice install the following packages using python 3.6.10:

  • boto3==1.18.9
  • psycopg2==2.9.1

Rename dwh.cfg.example as dwh.cfg. Update as appropriate as this will contain the value for all relevant variables.

Within the scripts/ folder run: python create_redshift_cluster False to create a redshift cluster. The first argument passed referes to an option to delete the redshift cluster.

Finally to delete the cluster run python create_redshift_cluster True to delete the redshift cluster.

Architecture

Terraform is an IaC tool that allows engineers / developers to build, change and destroy infrastructure conveniently and efficiently.

Amazon Redshift is a column oriented data warehouse with a Postgres compatible querying layer. With Redshift it is possible to scale from 1 to N clusters depending on your storage and query performance needs.

Structure

Show/Hide Details

About

Create a simple redshift cluster using terraform to access S3 buckets

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published