How to Increase Go-To-Market Speed with AWS CloudFormation

As an Amazon Web Services (AWS) Consulting Partner, we focus on giving our clients a competitive edge with business agility. Considering the vast number of services AWS provides, this requires an expansive understanding of the nuances of each service, like AWS CloudFormation, and how to integrate them seamlessly within a client’s unique technology ecosystem.

Recently, we worked with a client who sought to transform legacy business architecture into a new SaaS tool but was unsure where to start. After gathering requirements, which included reviewing their business processes and customer goals, we determined that utilizing AWS CloudFormation would be a crucial step in delivering a built-for-purpose business application quickly.

AWS Service Delivery Designation

Soliant has helped many clients reduce time to market using CloudFormation. In fact, we are in the midst of applying for the AWS CloudFormation Service Delivery Program and hope to attain that designation soon.

AWS CloudFormation: Streamlining Development with Automated Tooling

CloudFormation provides the means to automate the deployment of important infrastructure and application components. With it, you can:

• Take advantage of pre-built open-source stacks to reduce development effort, time, and costs
• Deliver a Minimum Viable Cloud (MVC) product quickly
• Scale fast
• Automate development with Infrastructure-as-Code (IaC) via customizable text file templates following JSON or YAML standards
• Take advantage of CloudFormation’s automatic provisioning to automate the deployment of each component within your templates
• Prevent mistakes when bringing up a new environment and/or revising an existing stack
• Share repeatable, predictable deployment stacks across your development teams

Possibilities with CloudFormation

Here are a few examples of infrastructure and application stacks we’ve automated via CloudFormation:

• Authentication, authorization, and OIDC infrastructure including OryFM, an OAuth2 identity provider that we leverage in
• Claris FileMaker applications
• Amazon S3 configurations
• AWS Lambda services
• Amazon CloudFront
• Node APIs
• React business applications
• Amazon RDS configurations
• Amazon Redshift

How to Set Up AWS CloudFormation

Three Things to Consider Before You Get Started:

1. The best practice is to start with a predefined template if possible. AWS provides many such templates on their Github here. You can also check out Soliant’s CloudFormation templates on Github here.
2. To modify an existing template or build one from scratch, you can use either your favorite text editor tool or the AWS CloudFormation Designer. The Designer tool allows you to author your templates using a graphical user interface, greatly simplifying and speeding up development.
3. AWS provides many services for free through its AWS Free Tier during a 12 month trial period. However, you may incur costs any time you configure AWS services.

How to Deploy the Amazon Redshift CloudFormation

As organizations across the world become ever more inundated with data, my own interest in leveraging that data via artificial intelligence (AI) continues to grow. In order to analyze data, we, of course, must be able to store it and process it. A good place to start with your AI journey then is with building a data warehouse. With that in mind, we will walk through setting up Amazon Redshift, a cloud data warehouse at petabyte scale. We’ll begin with a single-node Redshift cluster. Please note, the following process requires a moderate level of understanding of AWS. If you haven’t already, you may want to check out training resources from AWS here. Or Qwiklabs also has some very useful free (and paid) training here.

Your QuickStart Template:

1. To get started, log into your AWS Management Console.
2. For the Redshift CloudFormation Quick Start deployment, you’ll need to have the following set up:
   1. An EC2 Key Pair in the Region in which you plan to deploy. If you’ve never set up an EC2 Key Pair, follow the instructions here. Write down the Key Pair Alias as you will need it in number 6 below.
   2. A user role with Identity Access Management (IAM) permissions. Learn more here.
   3. Consider any additional requirements specific to your organization and environment as outlined in the Amazon Redshift Quick Start Guide.
3. To deploy into a new VPC, click this link to be taken to the CloudFormation creation dialog, as shown in Figure 1.

4. Confirm you have the appropriate Region selected in the upper right-hand corner.
5. Choose Next.
6. To keep things simple, in the Specify Stack Details dialog, we’ll keep all of the default parameters. However, there are five parameters that require input:
   1. Stack Name
   2. Availability Zones: Specify two Availability Zones to use for the subnets in the VPC
   3. Allowed External Access CIDR: For testing purposes, you can use: 10.0.0.0/0. Since this block will be accessible via external SSH, in a production environment, you will need to follow appropriate security measures with any live data that you manage in the cluster.
   4. Redshift master user password: This will be associated with the master account for the cluster. Save this in your password manager for later reference.
   5. Under Linux Bastion Configuration/Key Pair Name: Use the Key Pair Name referenced in number 2 above.
      Under SNS Notification Email, specify your email – this is set to a default address that you will want to change.
7. Choose Next on the Options page.</
8. Review your configuration settings and acknowledge that CloudFormation might:
   1. Create IAM resources with custom names.
   2. Require the following capability: CAPABILITY_AUTO_EXPAND
9. Choose Create Stack.Confirm that the stack reads as CREATE_COMPLETE. (You will need to press the Refresh button in the upper right corner of the events tab as shown in Figure 2 below.)
10. That’s it! You’ve deployed Amazon Redshift.

Monitoring for Errors

You do need to watch the events closely as they roll out in case an error occurs. See the Troubleshooting section below if you run into any issues – like I did. From what I can tell, the Redshift Quick Start is currently using a version of Python that is no longer supported – I’ve submitted a bug on Github and provided a workaround below. Hopefully, by the time you read this, that issue will be addressed. If not, see the Linux Bastion Host Troubleshooting section below.

11. If all stacks complete successfully, you are ready to test out your deployment. Make sure you can connect to the Redshift database, run queries, etc. The Quick Start Guide provides specific testing procedures.

Take a moment to click through the different stacks set up with this CloudFormation deployment. Then click on the Resources tab for each one; you’ll notice that about 45 different AWS components have been set up. Now, imagine what it would take to properly set up all of those components individually! As you can see, CloudFormation has the capability of saving enormous amounts of time while also supporting the implementation of best practices.

Troubleshooting

In the event your deployment encountered an error, it should automatically roll back. If that is the case, not all events will remain visible after the rollback. Thus, you may need to rerun the stack with automatic rollback turned off in order to find the exact issue causing the error. To rerun the stack, take the following steps:

1. In the AWS Management Console CloudFormation/Services, click on the stack’s name to view the details.
2. Choose the Template tab.

3. Then select the View in Designer button.
4. Once in the Designer, select the cloud icon with an upward arrow on the left-hand side of the screen.

Linux Bastion Host Troubleshooting

If your deployment gets stuck on the Linux Bastion Auto Scaling Group like mine did, you will need to deploy the Redshift Quick Start and the Linux Bastion Host Quick Start separately. To do so, take the following steps:

1. Delete the failed Redshift stack.
2. Then launch the Linux Bastion Host Quick Start into a new VPC first. (After that completes successfully, you will then launch the Redshift Quick Start into the VPC created with the template used in this step.)
3. After launching the Linux Bastion Quick Start, you will land in the Create Stack process, Step 1: Specify Template, confirm the proper template is specified, and the proper region is selected. Then select Next.
4. Specify the appropriate details and launch.
5. Once that template deploys successfully, you will need to note the ID of the VPC.
6. Select the Linux-bastion stack.
7. Select Resources.
8. Click on the Physical ID of the VPCStack Resource.
9. Select Resources.
10. Search for VPC.
11. Copy the Physical ID of the VPC Resource into your clipboard to use when launching the Redshift Stack.
12. Now, launch the Redshift Quick Start into an Existing VPC. You’ll notice this looks a bit different because we are launching into an existing VPC.
13. Confirm the ID of the VPC from step 10 above lands in the VPC Network Configuration fields and complete the rest of the information in the VPC section. See the Redshift Quick Start Deployment Guide for more details.
14. Launch the Redshift template and confirm no errors occurred.

How CloudFormation Works with Other Services

As you can see, CloudFormation, like nearly every modern technology service, is not a standalone tool. Beyond using it to stand up other AWS Services like Redshift and the other examples provided above, we also often pair it with the Serverless Framework to spin up complex components even faster.

Given that Soliant has a deep history with rapidly building business applications using low-code environments, we’ve also developed Claris FileMaker specific CloudFormation templates, available on Github here. If you’d like to partner with us on using CloudFormation in your next project, you can find out more about our CloudFormation services here.

Now, back to that client who wanted to build a new SaaS tool, learn how we leveraged CloudFormation to build it for them – fast.

What’s Next with Redshift and the Journey of AI?

If you’re interested in generating value for your business with AI, stay tuned for my follow-up posts on how to load data into RedShift, then join, filter, and extract it into various AWS cloud AI services. We’ll also discuss what AI use cases can bring the most business value. Given that Gartner recognized AWS as a Leader in their first-ever Magic Quadrant for Cloud AI Developer Services Report released in February 2020, we have a broad range of tools to work with that offer both pre-trained AI services and more advanced ML functions. Or, contact our team to learn how we can help your business through our consulting services.