EMC launched ECS 2.0 at DockerCon 2015 and a great big bonus was included – you can download a community edition for FREE. The devs were an early adopter of containerization so this software defined object storage platform runs as, you guessed it, a Docker container. Let’s deploy it in AWS.
This will walk you through the steps to run the free community supported, non-prod version of ECS in AWS as a Docker container using a CentOS instance. You can start at section 2 to deploy the container to a bare metal or vSphere CentOS machine. If you do, be sure to add a disk of at least 100GB to be used for ECS persistent storage.
Updated 7/13/15: Note- There are regular updates occurring to the installation scripts located at the EMCECS Github page. Please review the documentation there to ensure you are using the latest commands.
Overview of Steps:
-
Launch and Configure a CentOS Instance in AWS
-
Execute ECS Deployment Script
-
Configure ECS with Script
Launch and Configure a CentOS Instance in AWS
- Login to the AWS Console and navigate to EC2
- Click Launch Instance. Excited yet?
- Choose AWS Marketplace, enter CentOS, and click Select.
- Updated 6/29: Thanks @mcowger for pointing out a lower cost instance option. Select the instance
M4.2xlargeR3.xlarge ($0.35/hr) as this meets the minimum requirements of 30GB of RAM and 4 CPUs. Then, click Next. The cloud costs money. This could add up if you leave it running! NOTE: ECS can run with less than 30GB of RAM with some tweaks not covered here. - Select or create a new VPC. Click Next.
- Click Add New Volume, enter a size of at least 100GB, and select Magnetic for the volume type. Again, this surely is not free captain cloud builder.
- next…
- Configure the security group to allow the ports necessary to access ECS. I recommend restricting access to specific IP addresses. The initial deployment will have a well known default password.
- No need to boot from SSD unless you want to pay for it.
- OMG Launch it already! Ain’t the cloudz so simple? SHAMELESS PLUG ALERT: you too can have a simple on-prem cloud > EMC Federation Hybrid Cloud gets you there quickly. Tell them I sent you. If only I got referral payments…
- Create a new key pair unless you already have existing keys to use. Do not lose this or give it to your neighbor. They will hack your ECS instance.
- Launch Status. Houston, we have a new instance in the cloud. Time to make it rain ECS swift or S3 objects.
- Select your new instance and copy your public address. You will need this to navigate to your fabulous new cloud instance. Tell the CIO you are using the cloud. RUN!
- Open terminal on your Mac. PC users go get a Mac or an ssh client. Execute:
ssh -i <yourkey>.pem centos@<youraddressfromabove>
- yes
- Updates. Yummy ->
sudo yum update
- Install required bits ->
sudo yum install git tar wget
Execute ECS Deployment Script
You can perform these steps on any docker capable machine with enough CPU and RAM.
- Clone the ECS repository from github:
git clone https://github.com/EMCECS/ECS-CommunityEdition
-
cd ECS-CommunityEdition/ecs-single-node/
- Get the device name for the volume you added and the NIC name.
sudo fdisk -l
then
ifconfig -a
Also note the IP address of eth0 for later.
- Run the configuration script substituting what you have from the previous commands. Here is what it looks like in my super CentOS machine in the cloud.
sudo python step1_ecs_singlenode_install.py --disks xvdb --ethadapter eth0 --hostname <yourhostname>
- If you failed to mess up these steps you will eventually see a prompt instructing you to navigate to the web interface. See what I did there?
- SUCCESS!! You just deployed a cloud-scale object platform with geo-distribution capabilities. Nice.
- Put it on your resume.
- You can verify the container is running with a
sudo docker ps
command.
- You can now navigate to the web interface but be sure to continue on to the next section to configure the object settings.
Configure ECS with Script
- This section is quick. One script. You can do it. The main thing you need to be sure to change is the ECSNodes flag. Use your NIC IP from above. The other options can be customized if you want.
sudo python step2_object_provisioning.py --ECSNodes=10.0.1.10 --Namespace=ns1 --ObjectVArray=ova1 --ObjectVPool=ovp1 --UserName=emccode --DataStoreName=ds1 --VDCName=vdc1 --MethodName
- Let the script complete. Notice it is using the ECS REST API to configure the virtual data center, virtual array, and more.
Now that you have a fully functional Elastic Cloud Storage platform deployed, go push all of your objects up there. You can use it for HDFS too. Check out the ECS 2.0 Product Documentation Index to really geek out. Also check out the handiwork of the EMC {code} team.