Tuesday, February 1, 2011

Auto Scaling With Amazon EC2

Creating an auto scaled system using an Amazon load balancer is an interesting task that I did recently. We have an Amazon EC2 image with WSO2 Application Server installed. Creating an image with WSO2 WSAS installed is described here

Amazon EC2 API Tools
You will need Amazon EC2 API tools to create the image yourself. You can install it using "sudo apt-get install ec2-api-tools" in debian based operating systems, or you can download it from Amazon S3. These tools provide a client interface to the Amazon EC2 web service, to register and launch instance and more.

Configure
pradeeban@pradeeban:~/Downloads$ ec2-run-instances -K KEY.pem  -C CERT.pem

Instance Details
You can get the instance details using the InstanceID i-xxxxxxxx you get above.

ec2-describe-instances -K KEY.pem -C CERT.pem  i-xxxxxxxx or get the details of all the instances by,
ec2-describe-instances -K KEY.pem -C CERT.pem

[Providing the relevant public key and cert, KEY.pem and CERT.pem.]


Load Balancing with Auto Scaling
Now we come to the interesting part. That is auto scaling the Amazon EC2 Image with the load.

Download and set up Elastic Load Balancing API tools
Download it here, extract, and set the path up appropriately.
export AWS_ELB_HOME=/home/pradeeban/program/ElasticLoadBalancing-1.0.11.1
export PATH=$PATH:$AWS_ELB_HOME/bin

ELB Quick Reference Card
Downloading and setting up Auto Scaling API tools
Download it here, extract, and set the path up appropriately.

export AWS_AUTO_SCALING_HOME=/home/pradeeban/programs/AutoScaling-1.0.33.1
export PATH=$PATH:$AWS_AUTO_SCALING_HOME/bin



Creating a Load Balancer
pradeeban@pradeeban:~/Downloads$ elb-create-lb  autoscalelb --headers --listener "lb-port=80,instance-port=9763,protocol=http" --listener "lb-port=443,instance-port=9443,protocol=tcp" --availability-zones us-east-1c -K KEY.pem -C CERT.pem
DNS_NAME  DNS_NAME
DNS_NAME  autoscalelb-1316227031.us-east-1.elb.amazonaws.com

Describe ELB
elb-describe-lbs autoscalelb -K KEY.pem -C CERT.pem
LOAD_BALANCER  autoscalelb  autoscalelb-1316227031.us-east-1.elb.amazonaws.com  2011-01-28T09:40:54.750Z

Register instances with the load balancer
elb-register-instances-with-lb autoscalelb --instances i-xxxxxxxx -K KEY.pem -C CERT.pemINSTANCE_ID  i-xxxxxxxx

Configuring a health check
pradeeban@pradeeban:~/Downloads$ elb-configure-healthcheck  autoscalelb --headers --target "TCP:9763" --interval 5 --timeout 3 --unhealthy-threshold 2 --healthy-threshold 2 -K KEY.pem -C CERT.pem
HEALTH_CHECK  TARGET    INTERVAL  TIMEOUT  HEALTHY_THRESHOLD  UNHEALTHY_THRESHOLD
HEALTH_CHECK  TCP:9763  5         3        2                  2


 
Creating an AutoScaled System

Launching configuration for Amazon EC2 framework to launch new Amazon instances.
pradeeban@pradeeban:~/Downloads$ as-create-launch-config autoscalelc --image-id ami-xxxxxxxx --instance-type m1.large -K KEY.pem -C CERT.pem
OK-Created launch config



You can choose the instance type (m1.small, m1.large, and m1.xlarge) based on the requirements.

Creating Auto Scaling Group
pradeeban@pradeeban:~/Downloads$ as-create-auto-scaling-group autoscleasg --availability-zones us-east-1c --launch-configuration autoscalelc --min-size 1 --max-size 10 --load-balancers autoscalelb -K KEY.pem -C CERT.pem
OK-Created AutoScalingGroup


Describe auto scaling groups
pradeeban@pradeeban:~/Downloads$ as-describe-auto-scaling-groups autoscleasg -K KEY.pem -C CERT.pem
AUTO-SCALING-GROUP  autoscleasg  autoscalelc  us-east-1c  autoscalelb  1  10  1


Configuring a trigger with start actions according to the load.
pradeeban@pradeeban:~/Downloads$ as-create-or-update-trigger autoscaletrigger --auto-scaling-group autoscleasg --namespace "AWS/ELB" --measure Latency --statistic Average --dimensions "LoadBalancerName=autoscalelb" --period 60 --lower-threshold 0.5 --upper-threshold 1.2 --lower-breach-increment=-1 --upper-breach-increment 1 --breach-duration 120 -K KEY.pem -C CERT.pem
DEPRECATED: This command is deprecated and included only to facilitate migration to the new trigger mechanism.  You should use this command for migration purposes only.
OK-Created/Updated trigger



measure
You can choose the measure, based on your auto-scaling requirements. Let it be CPUUtilization, Latency, or Load. You will have to choose this wisely based on the application types, let them be CPU-intense applications, or huge applications, or applications that consume too much time. 

Now you notice as-create-or-update-trigger is depreciated. You can use scale up and scale down policies, along with the cloud-watch tools as described below instead!


Amazon CloudWatch API Tools
Downloading and Setting up 
Download it here, extract, and set the path up appropriately, to monitor the AWS cloud resources.

export AWS_CLOUDWATCH_HOME=/home/pradeeban/programs/CloudWatch-1.0.9.5
export PATH=$PATH:$AWS_CLOUDWATCH_HOME/bin

Now we have to define the scale up and scale down policies on scaling up and down the system based on the load, along with the monitoring.

Scale-up Policy
pradeeban@pradeeban:~/pem$ as-put-scaling-policy MyScaleUpPolicy1 --auto-scaling-group autoscleasg1 --adjustment=1 --type ChangeInCapacity --cooldown 300 -K KEY.pem -C CERT.pem
arn:aws:autoscaling:us-east-1:xxxxxxxxxxxxxx:scalingPolicy:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:autoScalingGroupName/autoscleasg1:policyName/MyScaleUpPolicy1

pradeeban@pradeeban:~/pem$ mon-put-metric-alarm MyHighCPUAlarm1 --comparison-operator GreaterThanThreshold --evaluation-periods 1 --metric-name CPUUtilization --namespace "AWS/EC2" --period 600 --statistic Average --threshold 80 --alarm-actions arn:aws:autoscaling:us-east-1:xxxxxxxxxxxxxx:scalingPolicy:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:autoScalingGroupName/autoscleasg1:policyName/MyScaleUpPolicy1 --dimensions "AutoScalingGroupName=autoscleasg" -K KEY.pem -C CERT.pem
OK-Created Alarm

Scale Down Policy
pradeeban@pradeeban:~/pem$ as-put-scaling-policy MyScaleDownPolicy1 --auto-scaling-group autoscleasg1 --adjustment=-1 --type ChangeInCapacity --cooldown 300 -K KEY.pem -C CERT.pem
arn:aws:autoscaling:us-east-1:xxxxxxxxxxxxxx:scalingPolicy:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:autoScalingGroupName/autoscleasg1:policyName/MyScaleDownPolicy1

pradeeban@pradeeban:~/pem$ mon-put-metric-alarm MyLowCPUAlarm --comparison-operator LessThanThreshold --evaluation-periods 1 --metric-name CPUUtilization --namespace "AWS/EC2" --period 600 --statistic Average --threshold 40 --alarm-actions arn:aws:autoscaling:us-east-1:xxxxxxxxxxxxxx:scalingPolicy:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:autoScalingGroupName/autoscleasg1:policyName/MyScaleDownPolicy1 --dimensions "AutoScalingGroupName=autoscleasg" -K KEY.pem -C CERT.pem
OK-Created Alarm

Now the auto scaling group gets an instance, as the minimum number of instances in the auto scaled system has been set to 1,
pradeeban@pradeeban:~/Downloads$ as-describe-auto-scaling-groups autoscleasg -K KEY.pem -C CERT.pem
AUTO-SCALING-GROUP  autoscleasg  autoscalelc  us-east-1c  autoscalelb  1  10  1
INSTANCE  i-xxxxxxxx  us-east-1c  InService  Healthy  autoscalelc

Once the elastic load balancer is set fine and triggered, it starts new nodes or remove the existing nodes according to the load. Following these steps, the system can be load balanced with autoscaling.


Load Balanced Instances' Health
Initially,
pradeeban@pradeeban:~/pem$ elb-describe-instance-health autoscalelb –headers -K KEY.pem -C CERT.pem
INSTANCE_ID  INSTANCE_ID  STATE      DESCRIPTION  REASON-CODE
INSTANCE_ID  i-xxxxxxxx   InService  N/A          N/A

Later with the load, you may see at least a new instance.
pradeeban@pradeeban:~/pem$ elb-describe-instance-health autoscalelb –headers -K KEY.pem -C CERT.pem
INSTANCE_ID  INSTANCE_ID  STATE      DESCRIPTION  REASON-CODE
INSTANCE_ID  i-xxxxxxxx   InService  N/A          N/A
INSTANCE_ID  i-yyyyyyyy   InService  Active Instance

After a few failed attempts, an instance will be marked as 'OutOfService' with the reason, 'Instance has failed at least the UnhealthyThreshold number of health checks consecutively.' or so.

10 comments:

  1. Dear Pradeeban,

    Thanks for the article. We have shared some insights and pointers about AWS AutoScaling in this blog post (with presentation) :

    We hope your blog readers will find it useful.

    URL:
    http://cloudblog.8kmiles.com/2011/04/20/auto-scaling-using-amazon-web-services/

    Thanks,
    Harish Ganesan
    http://cloud.8kmiles.com

    ReplyDelete
  2. Hi Harish,
    Glad to see you found my post useful.

    Thank you.
    Regards,
    Pradeeban.

    ReplyDelete
  3. hi Kathiravelu,

    how do you handle this scenario? (assumption that all instances are running with autoscaling)
    1. how do you patching of your os instances with less downtime?
    2. how do you handle the application code change and apply to your instances?

    thanks!

    ReplyDelete
  4. Hi Kathiravelu, you've got a good guide here... it looks like you posted this in 2011 :) but it is still relevant today.
    There are online tools that you can use as well to manage some of these things if you don't want to use the command line tools. EZAutoScaling and ylastic are some. I work at EZAutoScaling, so I can set you up with a "free-forever" account if you want to use it. It's not Open Source :) and I don't know if you need an auto scaling tool today, but I thought you might find it useful.

    ReplyDelete
  5. Hi Boyan,
    Thanks for your comment. As you said, this was written in early 2011. Now all these commands are available in AWS panel, with a nice graphic user interface. So most of the users would not need this, unless they like a command line, like me.

    Regards,
    Pradeeban.

    ReplyDelete
    Replies
    1. Hah. Yup.
      Where in the AWS panel do you go to manage auto scaling? I haven't been able to find it.
      Thanks!

      Delete
    2. Yeah Kathiravelu, where do you see this in the AWS UI?

      Delete
    3. Though not obvious, you should be able to define and locate the autoscaling policies at CloudWatch. Find something similar to, https://console.aws.amazon.com/cloudwatch/home?region=us-east-1

      You can create alarms https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#s=Alarms&alarmFilter=inAlarm using the relevant metrics such as EC2, ELB, .. https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#s=Metrics&metricFilter=Pattern%3DAWS/EC2%26Category%3D

      In short, pls go through the Cloud Watch API for auto scaling.

      Delete
  6. Hi,
    You have to set alarms in a place such as https://console.aws.amazon.com/cloudwatch/home?region=us-east-1, defining the scaling policies. Pls refer to the respective guides.

    Regards,
    Pradeeban.

    ReplyDelete