CfnCluster¶

CfnCluster (“cloud formation cluster”) is a framework that deploys and maintains high performance computing clusters on Amazon Web Services (AWS). Developed by AWS, CfnCluster facilitates both quick start proof of concepts (POCs) and production deployments. CfnCluster supports many different types of clustered applications and can easily be extended to support different frameworks. Download CfnCluster today to see how CfnCluster’s command line interface leverages AWS CloudFormation templates and other AWS cloud services.

Getting started with CfnCluster¶

CfnCluster (“cloud formation cluster”) is a framework that deploys and maintains high performance computing clusters on Amazon Web Services (AWS). Developed by AWS, CfnCluster facilitates both quick start proof of concepts (POCs) and production deployments. CfnCluster supports many different types of clustered applications and can easily be extended to support different frameworks. Download CfnCluster today to see how CfnCluster’s command line interface leverages AWS CloudFormation templates and other AWS cloud services.

Installing CfnCluster¶

The current working version is CfnCluster-v0.0.22. The CLI is written in python and uses BOTO for AWS actions. You can install the CLI with the following commands, depending on your OS.

Linux/OSX¶

$ sudo pip install cfncluster

or:

$ sudo easy_install cfncluster

Windows¶

Windows support is experimental!!

Install the following packages:

Python2.7 - https://www.python.org/download/
setuptools - https://pypi.python.org/pypi/setuptools#windows-7-or-graphical-install

Once installed, you should update the Environment Variables to have the Python install directory and Python Scripts directory in the PATH, for example: C:\Python27;C:\Python27\Scripts

Now it should be possible to run the following within a command prompt window:

C:\> easy_install CfnCluster

Upgrading¶

To upgrade an older version of CfnCluster, you can use either of the following commands, depening on how it was originally installed:

$ sudo pip install --upgrade cfncluster

or

$ sudo easy_install -U cfncluster

Remember when upgrading to check that the exiting config is compatible with the latest version installed.

Configuring CfnCluster¶

Once installed you will need to setup some initial config. The easiest way to do this is below:

$ cfncluster configure

This configure wizard will prompt you for everything you need to create your cluster. You will first be prompted for your cluster name, which is the logical name of your cluster.

Cluster Name [mycluster]:

Next, you will be prompted for your AWS Access & Secret Keys. Enter the keys for an IAM user with administrative privledges. These can also be read from your environment variaables or the aws CLI config.

AWS Access Key ID []:
AWS Secret Access Key ID []:

Now, you will be presented with a list of valid AWS region identifiers. Choose the region in which you’d like your cluster to run.

Acceptable Values for AWS Region ID:
    us-east-1
    cn-north-1
    ap-northeast-1
    eu-west-1
    ap-southeast-1
    ap-southeast-2
    us-west-2
    us-gov-west-1
    us-west-1
    eu-central-1
    sa-east-1
AWS Region ID []:

Choose a descriptive name for your VPC. Typically, this will something like “production” or “test”.

VPC Name [myvpc]:

Next, you will need to choose a keypair that already exists in EC2 in order to log into your master instance. If you do not already have a keypair, refer to the EC2 documentation on EC2 Key Pairs - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html

Acceptable Values for Key Name:
    keypair1
    keypair-test
    production-key
Key Name []:

Choose the VPC ID in which you’d like your cluster launched into.

Acceptable Values for VPC ID:
    vpc-1kd24879
    vpc-blk4982d
VPC ID []:

Finally, choose the subnet in which you’d like your master server to run in.

Acceptable Values for Master Subnet ID:
    subnet-9k284a6f
    subnet-1k01g357
    subnet-b921nv04
Master Subnet ID []:

Next, a simple cluster launches into a VPC and uses an existing subnet which supports public IP’s i.e. the route table for the subnet is 0.0.0.0/0 => igw-xxxxxx. The VPC must have “DNS Resolution = yes” and “DNS Hostnames = yes”. It should also have DHCP options with the correct “domain-name” for the region, as defined in the docs: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_DHCP_Options.html

Once all of those settings contain valid values, you can launch the cluster by repeating the command that was used at the start.

$ cfncluster create mycluster

Once the cluster reaches the “CREATE_COMPLETE” status, you can connect using your normal SSH client/settings. For more details on connecting to EC2 instances, check the EC2 User Guide - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-connect-to-instance-linux.html#using-ssh-client

Working with CfnCluster¶

Network Configurations¶

CfnCluster leverages Amazon Virtual Private Cloud(VPC) for networking. This provides a very flexiable and configurable networking platform to deploy clusters within. CfnCluster support the following high-level configurations:

Single subnet, master and compute in the same subnet
Two subnets, master in one subnet and compute new private subnet
Two subnets, master in one subnet and compute in exisiting private subnet

All of these configurations can operate with or without public IP addressing. It can also be deployed to leverage an HTTP proxy for all AWS requests. The combinations of these configurations result in many different deployment scenario’s, ranging from a single public subnet with all access over the Internet, to fully private via AWS Direct Connect and HTTP proxy for all traffic.

Below are some architeture diagrams for some of those scenarios:

CfnCluster in a single public subnet

The configuration for this architecture, requires the following settings:

note that all values are examples only

[vpc public]
vpc_id = vpc-a1b2c3d4
master_subnet_id = subnet-a1b2c3d4

CfnCluster using two subnets(new private)

The configuration for this architecture, requires the following settings:

note that all values are examples only

[vpc public-private]
vpc_id = vpc-a1b2c3d4
master_subnet_id = subnet-a1b2c3d4
compute_subnet_cidr = 10.0.1.0/24

CfnCluster in a private subnet connected using Direct Connect

The configuration for this architecture, requires the following settings:

note that all values are examples only

[vpc private-dx]
vpc_id = vpc-a1b2c3d4
master_subnet_id = subnet-a1b2c3d4
proxy_server = http://proxy.corp.net:8080
use_public_ips = false

Custom Bootstrap Actions¶

CfnCluster can execute arbritary code either before(pre) or after(post) the main bootstrap action during cluster creation. This code is typically stored in S3 and accessed via HTTP(S) during cluster creation. The code will be executed as root and can be in any script language supppoted by the cluster OS, typically bash or python.

pre-install actions are called before any cluster deployment bootstrap such as configuring NAT, EBS and the scheduler. Typical pre-install actions may include modifying storage, adding extra users or packages.

post-install actions are called after cluster bootstrap is complete, as the last action before an instance is considered complete. Typical post-install actions may include changing scheduler settings, modofying storage or packages.

Arguments can be passed to scripts by specifying them in the config. These will be passed double-quoted to the pre/post-install actions.

If a pre/post-install actions fails, then the instance bootstrap will be considered failed and it will not continue. Success is signalled with an exit code of 0, any other exit code will be considered a fail.

Configuration¶

The following config settings are used to define pre/post-install actions and arguments. All options are optional and are not required for basic cluster install.

# URL to a preinstall script. This is executed before any of the boot_as_* scripts are run
# (defaults to NONE for the default template)
pre_install = NONE
# Arguments to be passed to preinstall script
# (defaults to NONE for the default template)
pre_install_args = NONE
# URL to a postinstall script. This is executed after any of the boot_as_* scripts are run
# (defaults to NONE for the default template)
post_install = NONE
# Arguments to be passed to postinstall script
# (defaults to NONE for the default template)
post_install_args = NONE

Example¶

The following are some steps to create a simple post install script that installs the R packages in a cluster.

Create an script. For the R example, see below

#!/bin/bash

yum -y install --enablerepo=epel R

Upload the script with the correct permissions to S3

aws s3 cp --acl public-read /path/to/myscript.sh s3://<bucket-name>/myscript.sh

Update CfnCluster config to include the new post install action

[cluster default]
...
post_install = https://<bucket-name>.s3.amazonaws.com/myscript.sh

Lauch a cluster

cfncluster create mycluster

Working with S3¶

Accessing S3 within CfnCluster can be controlled through two parameters in the CfnCluster config.

# Specify S3 resource which cfncluster nodes will be granted read-only access
# (defaults to NONE for the default template)
s3_read_resource = NONE
# Specify S3 resource which cfncluster nodes will be granted read-write access
# (defaults to NONE for the default template)
s3_read_write_resource = NONE

Both parameters except either * or a valid S3 ARN. For details of how to specify S3 ARNs, please see http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-s3

Configuration¶

cfncluster uses the file ~/.cfncluster/config by default for all configuration parameters.

You can see an example configuration file site-packages/cfncluster/examples/config

Layout¶

Configuration is defined in multiple sections. Required sections are “global”, “aws”, one “cluster”, and one “subnet”.

A section starts with the section name in brackets, followed by parameters and configuration.

[global]
cluster_template = default
update_check = true
sanity_check = true

Configuration Options¶

global¶

Global configuration options related to cfncluster.

[global]

cluster_template¶

The name of the cluster section used for the cluster.

See the Cluster Definition.

cluster_template = default

update_check¶

Whether or not to check for updates to cfncluster.

update_check = true

sanity_check¶

Attempts to validate that resources defined in parameters actually exist.

sanity_check = true

aws¶

This is the AWS credentials section (required). These settings apply to all clusters.

If not defined, boto will attempt to use a) enviornment or b) EC2 IAM role.

[aws]
aws_access_key_id = #your_aws_access_key_id
aws_secret_access_key = #your_secret_access_key

# Defaults to us-east-1 if not defined in enviornment or below
aws_region_name = #region

cluster¶

You can define one or more clusters for different types of jobs or workloads.

Each cluster has it’s own configuration based on your needs.

The format is [cluster <clustername>].

[cluster default]

key_name¶

Name of an existing EC2 KeyPair to enable SSH access to the instances.

key_name = mykey

template_url¶

Overrides the path to the cloudformation template used to create the cluster

Defaults to https://s3.amazonaws.com/cfncluster-<aws_region_name>/templates/cfncluster-<version>.cfn.json.

template_url = https://s3.amazonaws.com/cfncluster-us-east-1/templates/cfncluster.cfn.json

compute_instance_type¶

The EC2 instance type used for the cluster compute nodes.

Defaults to t2.micro for default template.

compute_instance_type = t2.micro

master_instance_type¶

The EC2 instance type use for the master node.

This defaults to t2.micro for default template.

master_instance_type = t2.micro

initial_queue_size¶

The inital number of EC2 instances to launch as compute nodes in the cluster.

The default is 2 for default template.

initial_queue_size = 2

max_queue_size¶

The maximum number of EC2 instances that can be launched in the cluster.

This defaults to 10 for the default template.

max_queue_size = 10

maintain_initial_size¶

Boolean flag to set autoscaling group to maintain initial size.

If set to true, the Auto Scaling group will never have fewer members than the value of initial_queue_size. It will still allow the cluster to scale up to the value of max_queue_size.

Setting to false allows the Auto Scaling group to scale down to 0 members, so resources will not sit idle when they aren’t needed.

Defaults to false for the default template.

maintain_initial_size = false

scheduler¶

Scheduler to be used with the cluster. Valid options are sge, openlava, or torque.

Defaults to sge for the default template.

scheduler = sge

cluster_type¶

Type of cluster to launch i.e. ondemand or spot

Defaults to ondemand for the default template.

cluster_type = ondemand

spot_price¶

If cluster_type is set to spot, the maximum spot price for the ComputeFleet.

spot_price = 0.00

custom_ami¶

ID of a Custom AMI, to use instead of default published AMI’s.

custom_ami = NONE