StarCluster

StarCluster is a utility for creating and managing general purpose computing clusters hosted on Amazon's Elastic Compute Cloud (EC2). StarCluster minimizes the administrative overhead associated with obtaining, configuring, and managing a traditional computing cluster used in research labs or for general distributed computing applications.

Getting started is easy, you only need StarCluster and an Amazon AWS account.

For those that are interested in learning more there is an article about StarCluster on http://www.hpcinthecloud.com http://www.hpcinthecloud.com/features/StarCluster-Brings-HPC-to-the-Amazon-Cloud-94099324.html There is also a screencast of installing, configuring, launching, and terminating an HPC cluster on Amazon EC2:

http://www.hpcinthecloud.com/blogs/MITs-StarCluster-An-Update-with-Screencast-94599554.html

To get started, the user creates a simple configuration file with their AWS account details and a few cluster preferences (e.g. number of machines, machine type, ssh keypairs, etc). After creating the configuration file and running StarCluster's "start" command, a cluster of Linux machines configured with the Sun Grid Engine queuing system, an NFS-shared /home directory, and OpenMPI with password-less ssh is created and ready to go out-of-the-box. Running StarCluster's "stop" command will shutdown the cluster and stop paying for service. This allows the user to only pay for what they use.

StarCluster can also utilize Amazon's Elastic Block Storage (EBS) volumes to provide persistent data storage for a cluster. EBS volumes allow you to store large amounts of data in the Amazon cloud and are also easy to back-up and replicate in the cloud. StarCluster will mount and NFS-share any volumes specified in the config. StarCluster's "createvolume" command provides the ability to automatically create, format, and partition new EBS volumes for use with StarCluster.

StarCluster provides a Ubuntu-based Amazon Machine Image (AMI) in 32bit and 64bit architectures. The AMI contains an optimized NumPy/SciPy/Atlas/Blas/Lapack installation compiled for the larger Amazon EC2 instance types. The AMI also comes with Sun Grid Engine (SGE) and OpenMPI compiled with SGE support. The public AMI can easily be customized by launching a single instance of the public AMI, installing additional software on the instance, and then using StarCluster's "createimage" command to completely automate the process of creating a new AMI from an EC2 instance.

What is new in this release:

• New StarCluster Ubuntu 11.10 EBS AMIs available including a new HVM AMI for use with CC/CC2/CG instance types! Includes OpenGridScheduler (to replace SGE), Condor, Hadoop/Dumbo, MPICH2, OpenMPI, IPython 0.12 (parallel+notebook), and more. Run dpkg -l on the AMIs for the full list of packages.
• The new StarCluster Ubuntu 11.10 EBS HVM AMI additionally comes with NVIDIA driver, CUDA Toolkit/SDK, PyCuda, PyOpenCL, and Magma for GPU computing.
• These new AMIs are available in all AWS regions. The new us-east-1 11.10 EBS AMIs are now the defaults in the StarCluster config template. Use the listpublic command to obtain a list of available StarCluster AMIs in other regions:
• $ starcluster -r sa-east-1 listpublic
• NOTE: The HVM AMI is currently only available in the us-east-1 region until other regions support HVM.

What is new in version 0.91:

• support for launching and managing multiple clusters on EC2
• added "listclusters" command for showing all active clusters on EC2
• support for attaching and NFS-sharing multiple EBS volumes
• added createimage and createvolume commands for easily creating new AMIs and EBS volumes for use with StarCluster
• experimental support for launching clusters using spot instances
• added support for StarCluster "plugins" that provide the ability to perform additional configuration/setup routines on top of StarCluster's default cluster configuration
• added "listpublic" command for listing all available public StarCluser AMIs that can be used with StarCluster
• bash/zsh command line completion for StarCluster's command line interface