Setting Up an AMI Development Environment

Warning

Below are instructions for building a custom AMI, potentially with custom version of the cookbook recipes to build CfnCluster AMIs. This is an advanced method of customizing CfnCluster, with many hard to debug pitfalls. The CfnCluster team highly recommends using Custom Bootstrap Actions scripts for customization, as post install hooks are generally easier to debug and more portable across releases of CfnCluster.

You will need to repeat the steps used to create your custom AMI with each new CfnCluster release.

Steps

This guide is written assuming your OS is Ubuntu 14.04. If you don’t have an Ubuntu machine you can easily get an EC2 instance running Ubuntu.

1. sudo apt-get -y install build-essential git

2. Go to https://downloads.chef.io/chef-dk, grab the latest version for your OS and install.

   For example:

   wget https://packages.chef.io/stable/ubuntu/12.04/chefdk_0.17.17-1_amd64.deb
   sudo dpkg -i chefdk_0.17.17-1_amd64.deb
   

3. git clone https://github.com/awslabs/cfncluster-cookbook

4. Grab the latest go-lang link from https://golang.org/dl/

5. Run the following:

   wget https://storage.googleapis.com/golang/go1.7.linux-amd64.tar.gz
   cd /usr/local
   sudo tar xf ~/go1.7.linux-amd64.tar.gz
   echo 'export GOPATH=~/work' >> ~/.bashrc
   echo 'export PATH=$GOPATH/bin:/usr/local/go/bin:$PATH' >> ~/.bashrc
   . ~/.bashrc
   

6. Install packer from source

   go get github.com/mitchellh/packer
   

The next part of setting up your environment involves setting a lot of environment variables, you can either set them as I explain what they are or use the script provided at the bottom.

1. Set your aws key pair name and path, if you don’t have a key pair create one.

   export AWS_KEYPAIR_NAME=your-aws-keypair                # Name of your key pair
   export EC2_SSH_KEY_PATH=~/.ssh/your-aws-keypair # Path to your key pair
   

2. Set the AWS instance type you’d like to launch.

   export AWS_FLAVOR_ID=c3.4xlarge
   

3. Set the availability zone and region:

   export AWS_AVAILABILITY_ZONE=us-east-1c
   export AWS_DEFAULT_REGION=us-east-1
   

4. Create a AWS VPC in that region:

   export AWS_VPC_ID=vpc-XXXXXXXXX
   

5. Create a subnet in that region and set it below:

   export AWS_SUBNET_ID=subnet-XXXXXXXX
   

6. Create a security group and set it:

   export AWS_SECURITY_GROUP_ID=sg-XXXXXXXX
   

7. Create an IAM Profile from the template here.

   export AWS_IAM_PROFILE=CfnClusterEC2IAMRole             # IAM Role name
   

8. Set the path to your kitchen yaml file. Note that this comes in CfnClusterCookbook.

   export KITCHEN_LOCAL_YAML=.kitchen.cloud.yml
   

9. Create a 10G ebs backed volume in the same availability zone:

   export CFN_VOLUME=vol-XXXXXXXX  # create 10G EBS volume in same AZ
   

10. Set the stack name.

    export AWS_STACK_NAME=cfncluster-test-kitchen
    

11. Create an sqs queue:

    export CFN_SQS_QUEUE=cfncluster-chef                    # create an SQS queue
    

12. Create a dynamoDB table with hash key instanceId type String and name it cfncluster-chef then export the following:

    export CFN_DDB_TABLE=cfncluster-chef  # setup table as cfncluster-chef
    

13. You should now be able to run the following:

    kitchen list
    

14. If something isn’t working you can run:

    kitchen diagnose all
    

Here’s a script to do all of the above, just fill out and the fields and source it like: . ~/path/to/script

export AWS_KEYPAIR_NAME=your-aws-keypair                # Name of your key pair
export EC2_SSH_KEY_PATH=~/.ssh/your-aws-keypair.pem     # Path to your key pair
export AWS_FLAVOR_ID=c3.4xlarge
export AWS_DEFAULT_REGION=us-east-1
export AWS_AVAILABILITY_ZONE=us-east-1c
export AWS_VPC_ID=vpc-XXXXXXXX
export AWS_SUBNET_ID=subnet-XXXXXXXX
export AWS_SECURITY_GROUP_ID=sg-XXXXXXXX
export AWS_IAM_PROFILE=CfnClusterEC2IAMRole     # create role using IAM docs for CfnCluster
export KITCHEN_LOCAL_YAML=.kitchen.cloud.yml
export CFN_VOLUME=vol-XXXXXXXX                                  # create 10G EBS volume in same AZ
export AWS_STACK_NAME=cfncluster-test-kitchen
export CFN_SQS_QUEUE=cfncluster-chef                    # create an SQS queue
export CFN_DDB_TABLE=cfncluster-chef                    # setup table as cfncluster-chef