Benchmarking Appserver remotely - ab, nohup, ec2, and more ..

This time we wanted to benchmark WSO2 Stratos Application Server instances running on two amazon ec2 instances. Both of the instances are proxied by WSO2 Load Balancer. Here we have used appserver.cloud1.wso2.com which has only the load balancing, and appserver.cloud2.wso2.com with load balancing and auto-scaling.

Now we picked Java Bench (A clone of Apache Bench - ab) for the purpose to benchmark the application server running over the two amazon ec2 instances. It was a long running test from another ec2 instance. This test lasted for more than 30 hours. We were able to benchmark and also notice a few interesting stuff too.

Here even after closing the connection to the remote instance where the test is running, the test should keep running.

Now we are running the test using the load of web service calls on a web service that is deployed on Application Server. The service is a simple adder service.

The below is stored as request.xml

From two directories cloud1 and cloud2, from the instance,

nohup ab -p request.xml -n 10000 -c 200 -k -H "SOAPAction: multiply" -T "application/soap+xml; charset=UTF-8" http://appserver.cloud1.wso2.com/services/HelloService &

nohup ab -p request.xml -n 10000 -c 200 -k -H "SOAPAction: multiply" -T "application/soap+xml; charset=UTF-8" http://appserver.cloud2.wso2.com/services/HelloService &

Here nohup comes handy. Even after you terminate the connection to the remote host, the test will continue, unless you find the process and kill it using kill -9.

You can view the nohup output from the above directories cloud1 and cloud2 above using tail -f nohup.out 

But since ab itself can't handle this much of a bulky requests at once, it may fail as below, timing ab out for the specific requests.

Benchmarking appserver.cloud1.wso2.com (be patient)
apr_poll: The timeout specified has expired (70007)

So let's create a shell script instead, to prevent firing all the requests to ab at once, like a million of requests. 

while true; do ab -p request4.xml -n 4000 -c 200 -k -H "SOAPAction: multiply" -T "application/soap+xml; charset=UTF-8" http://appserver.cloud1.wso2.com/services/HelloService; sleep 3; ab -p request4.xml -n 4000 -c 200 -k -H "SOAPAction: multiply" -T "application/soap+xml; charset=UTF-8" http://appserver.cloud2.wso2.com/services/HelloService; sleep 3; done

Save the above script (test.sh). Now by running 

nohup sh test.sh &

you can start the load test for both the instances.

ubuntu@ip-10-120-61-234:~$ nohup sh dual.sh &
[1] 6512
ubuntu@ip-10-120-61-234:~$ nohup: ignoring input and appending output to `nohup.out'

Here 6512 above indicates the pid of dual.sh, which is the benchmarking process of ab.

You can see the nohup output for the above script with the results from both the instances alternating, with a gap of 3 seconds. Here the time interval between two chunks of requests on the same instance is 6 seconds.

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking appserver.cloud1.wso2.com (be patient)
Completed 400 requests
Completed 800 requests
Completed 1200 requests
Completed 1600 requests
Completed 2000 requests
Completed 2400 requests
Completed 2800 requests
Completed 3200 requests
Completed 3600 requests
Completed 4000 requests
Finished 4000 requests


Server Software:        WSO2
Server Hostname:        appserver.cloud1.wso2.com
Server Port:            80

Document Path:          /services/HelloService
Document Length:        260 bytes

Concurrency Level:      200
Time taken for tests:   25.772 seconds
Complete requests:      4000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1652000 bytes
Total POSTed:           2188000
HTML transferred:       1040000 bytes
Requests per second:    155.20 [#/sec] (mean)
Time per request:       1288.623 [ms] (mean)
Time per request:       6.443 [ms] (mean, across all concurrent requests)
Transfer rate:          62.60 [Kbytes/sec] received
                        82.91 kb/s sent
                        145.50 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    2   5.3      1      81
Processing:     7 1256 711.9   1290    4351
Waiting:        6 1256 711.9   1290    4351
Total:          7 1258 711.7   1293    4363

Percentage of the requests served within a certain time (ms)
  50%   1293
  66%   1696
  75%   1883
  80%   1957
  90%   2165
  95%   2292
  98%   2416
  99%   2447
 100%   4363 (longest request)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking appserver.cloud2.wso2.com (be patient)
Completed 400 requests
Completed 800 requests
Completed 1200 requests
Completed 1600 requests
Completed 2000 requests
Completed 2400 requests
Completed 2800 requests
Completed 3200 requests
Completed 3600 requests
Completed 4000 requests
Finished 4000 requests


Server Software:        WSO2
Server Hostname:        appserver.cloud2.wso2.com
Server Port:            80

Document Path:          /services/HelloService
Document Length:        260 bytes

Concurrency Level:      200
Time taken for tests:   20.210 seconds
Complete requests:      4000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1652000 bytes
Total POSTed:           2188000
HTML transferred:       1040000 bytes
Requests per second:    197.92 [#/sec] (mean)
Time per request:       1010.486 [ms] (mean)
Time per request:       5.052 [ms] (mean, across all concurrent requests)
Transfer rate:          79.83 [Kbytes/sec] received
                        105.73 kb/s sent
                        185.55 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    1   1.1      1      14
Processing:     6  987 577.5    990    3700
Waiting:        6  987 577.5    990    3700
Total:          6  988 577.4    990    3700

Percentage of the requests served within a certain time (ms)
  50%    990
  66%   1292
  75%   1468
  80%   1570
  90%   1782
  95%   1907
  98%   1986
  99%   2013
 100%   3700 (longest request)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

This nohup.out contains the useful load testing results.

Auto Scaling With Amazon EC2 - II

We created an Amazon ELB as discussed here. Let's see more of it now.

Appserver
The load balancer we have created listens on the port 80 and 443 and forwards requests to 9763 and 9443. Say now we need to delete the listeners.

elb-delete-lb-listeners autoscalelb --lb-ports 80 443 -K KEY.pem -C CERT.pem  
    Warning: Deleting a LoadBalancer listener can lead to service disruption to
    any customers connected to the LoadBalancer listener. Are you sure you want
    to delete this LoadBalancer listener? [Ny]N
elb-delete-lb-listeners:  User stopped the execution of elb-delete-lb-listeners.

(Providing 'N' stops the action. You can proceed deleting the listener, by giving 'y' as the response.)

You can also create more listeners

elb-create-lb-listeners autoscalelb --headers --listener "lb-port=8280,instance-port=9763,protocol=http" --listener "lb-port=8243,instance-port=9443,protocol=tcp" -K KEY.pem -C CERT.pem

OK-Creating LoadBalancer Listener

The load balancer now listens on the port 8280 and 8243 and forwards requests to 9763 and 9443.

Now what will happen if you forcefully try to kill the instances initiated by the ELB? It will create one more identical instance immediately, handling the failover case.


Fail over

pradeeban@pradeeban:~/pem$ as-describe-auto-scaling-groups autoscleasg -K KEY.pem -C CERT.pem

Initially,

AUTO-SCALING-GROUP  autoscleasg  autoscalelc  us-east-1c  autoscalelb  1  10  1

INSTANCE  i-xxxxxxxx  us-east-1c  InService  Healthy  autoscalelc

When we killed the instance.

AUTO-SCALING-GROUP  autoscleasg  autoscalelc  us-east-1c  autoscalelb  1  10  1

INSTANCE  i-xxxxxxxx  us-east-1c  Terminating  Unhealthy  autoscalelc

INSTANCE  i-yyyyyyyy  us-east-1c  Pending      Healthy    autoscalelc

After a few seconds,

AUTO-SCALING-GROUP  autoscleasg  autoscalelc  us-east-1c  autoscalelb  1  10  1

INSTANCE  i-yyyyyyyy  us-east-1c  InService  Healthy  autoscalelc

Sounds cool.. So how to terminate the instances that ELB creates?


Shall we try deleting?

pradeeban@pradeeban:~/pem$ as-delete-auto-scaling-group autoscleasg -K KEY.pem -C CERT.pem

    Are you sure you want to delete this AutoScalingGroup? [Ny]y

as-delete-auto-scaling-group:  Service error: You cannot delete an AutoScalingGroup while there are instances

 still in the group.  AWSRequestId:aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa

Min = Max = 0
ok! So let's stop auto-scaling process by setting the minimum and maximum to zero.

pradeeban@pradeeban:~/pem$ as-update-auto-scaling-group autoscleasg --min-size 0 --max-size 0 -K KEY.pem -C CERT.pem

OK-Updated AutoScalingGroup

pradeeban@pradeeban:~/pem$ as-describe-auto-scaling-groups autoscleasg -K KEY.pem -C CERT.pem

AUTO-SCALING-GROUP  autoscleasg  autoscalelc  us-east-1c  autoscalelb  0  0  0

INSTANCE  i-yyyyyyyy  us-east-1c  InService  Healthy  autoscalelc

Delete
Now let's try to delete the auto-scaling group once more -- yes, as we have set it to zero, it should be possible now.

pradeeban@pradeeban:~/pem$ as-delete-auto-scaling-group autoscleasg -K KEY.pem -C CERT.pem

  
    Are you sure you want to delete this AutoScalingGroup? [Ny]y

as-delete-auto-scaling-group:  Service error: You cannot delete an AutoScalingGroup while there are scaling activities in progress for that group.

 AWSRequestId:bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbb

oh.. Let's wait a minute for that scaling down activity to finish!
Again!

pradeeban@pradeeban:~/pem$ as-delete-auto-scaling-group autoscleasg -K KEY.pem -C CERT.pem

          

    Are you sure you want to delete this AutoScalingGroup? [Ny]y

OK-Deleted AutoScalingGroup

Done!

Now, let's check once more whether we have deleted it properly.. ;)

pradeeban@pradeeban:~/pem$ as-describe-auto-scaling-groups autoscleasg -K KEY.pem -C CERT.pem

No AutoScalingGroups found

Yes, we have deleted the Amazon ELB along with the scaling group of nodes it created.  (We are sinners! ;))

Auto Scaling With Amazon EC2

Creating an auto scaled system using an Amazon load balancer is an interesting task that I did recently. We have an Amazon EC2 image with WSO2 Application Server installed. Creating an image with WSO2 WSAS installed is described here. 

Amazon EC2 API Tools

You will need Amazon EC2 API tools to create the image yourself. You can install it using "sudo apt-get install ec2-api-tools" in debian based operating systems, or you can download it from Amazon S3. These tools provide a client interface to the Amazon EC2 web service, to register and launch instance and more.

Configure

pradeeban@pradeeban:~/Downloads$ ec2-run-instances -K KEY.pem  -C CERT.pem

Instance Details

You can get the instance details using the InstanceID i-xxxxxxxx you get above.

ec2-describe-instances -K KEY.pem -C CERT.pem  i-xxxxxxxx or get the details of all the instances by,

ec2-describe-instances -K KEY.pem -C CERT.pem

[Providing the relevant public key and cert, KEY.pem and CERT.pem.]


Load Balancing with Auto Scaling
Now we come to the interesting part. That is auto scaling the Amazon EC2 Image with the load.

Download and set up Elastic Load Balancing API tools
Download it here, extract, and set the path up appropriately.

export AWS_ELB_HOME=/home/pradeeban/program/ElasticLoadBalancing-1.0.11.1

export PATH=$PATH:$AWS_ELB_HOME/bin

ELB Quick Reference Card
Downloading and setting up Auto Scaling API tools
Download it here, extract, and set the path up appropriately.

export AWS_AUTO_SCALING_HOME=/home/pradeeban/programs/AutoScaling-1.0.33.1

export PATH=$PATH:$AWS_AUTO_SCALING_HOME/bin

Creating a Load Balancer

pradeeban@pradeeban:~/Downloads$ elb-create-lb  autoscalelb --headers --listener "lb-port=80,instance-port=9763,protocol=http" --listener "lb-port=443,instance-port=9443,protocol=tcp" --availability-zones us-east-1c -K KEY.pem -C CERT.pem
DNS_NAME  DNS_NAME
DNS_NAME  autoscalelb-1316227031.us-east-1.elb.amazonaws.com

Describe ELB
elb-describe-lbs autoscalelb -K KEY.pem -C CERT.pem
LOAD_BALANCER  autoscalelb  autoscalelb-1316227031.us-east-1.elb.amazonaws.com  2011-01-28T09:40:54.750Z

Register instances with the load balancer
elb-register-instances-with-lb autoscalelb --instances i-xxxxxxxx -K KEY.pem -C CERT.pemINSTANCE_ID  i-xxxxxxxx

Configuring a health check
pradeeban@pradeeban:~/Downloads$ elb-configure-healthcheck  autoscalelb --headers --target "TCP:9763" --interval 5 --timeout 3 --unhealthy-threshold 2 --healthy-threshold 2 -K KEY.pem -C CERT.pem
HEALTH_CHECK  TARGET    INTERVAL  TIMEOUT  HEALTHY_THRESHOLD  UNHEALTHY_THRESHOLD
HEALTH_CHECK  TCP:9763  5         3        2                  2


 
Creating an AutoScaled System

Launching configuration for Amazon EC2 framework to launch new Amazon instances.
pradeeban@pradeeban:~/Downloads$ as-create-launch-config autoscalelc --image-id ami-xxxxxxxx --instance-type m1.large -K KEY.pem -C CERT.pem
OK-Created launch config


You can choose the instance type (m1.small, m1.large, and m1.xlarge) based on the requirements.

Creating Auto Scaling Group
pradeeban@pradeeban:~/Downloads$ as-create-auto-scaling-group autoscleasg --availability-zones us-east-1c --launch-configuration autoscalelc --min-size 1 --max-size 10 --load-balancers autoscalelb -K KEY.pem -C CERT.pem
OK-Created AutoScalingGroup

Describe auto scaling groups
pradeeban@pradeeban:~/Downloads$ as-describe-auto-scaling-groups autoscleasg -K KEY.pem -C CERT.pem
AUTO-SCALING-GROUP  autoscleasg  autoscalelc  us-east-1c  autoscalelb  1  10  1


Configuring a trigger with start actions according to the load.
pradeeban@pradeeban:~/Downloads$ as-create-or-update-trigger autoscaletrigger --auto-scaling-group autoscleasg --namespace "AWS/ELB" --measure Latency --statistic Average --dimensions "LoadBalancerName=autoscalelb" --period 60 --lower-threshold 0.5 --upper-threshold 1.2 --lower-breach-increment=-1 --upper-breach-increment 1 --breach-duration 120 -K KEY.pem -C CERT.pem
DEPRECATED: This command is deprecated and included only to facilitate migration to the new trigger mechanism.  You should use this command for migration purposes only.
OK-Created/Updated trigger

measure

You can choose the measure, based on your auto-scaling requirements. Let it be CPUUtilization, Latency, or Load. You will have to choose this wisely based on the application types, let them be CPU-intense applications, or huge applications, or applications that consume too much time. 

Now you notice as-create-or-update-trigger is depreciated. You can use scale up and scale down policies, along with the cloud-watch tools as described below instead!

Amazon CloudWatch API Tools
Downloading and Setting up 
Download it here, extract, and set the path up appropriately, to monitor the AWS cloud resources.

export AWS_CLOUDWATCH_HOME=/home/pradeeban/programs/CloudWatch-1.0.9.5

export PATH=$PATH:$AWS_CLOUDWATCH_HOME/bin

Now we have to define the scale up and scale down policies on scaling up and down the system based on the load, along with the monitoring.

Scale-up Policy
pradeeban@pradeeban:~/pem$ as-put-scaling-policy MyScaleUpPolicy1 --auto-scaling-group autoscleasg1 --adjustment=1 --type ChangeInCapacity --cooldown 300 -K KEY.pem -C CERT.pem
arn:aws:autoscaling:us-east-1:xxxxxxxxxxxxxx:scalingPolicy:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:autoScalingGroupName/autoscleasg1:policyName/MyScaleUpPolicy1

pradeeban@pradeeban:~/pem$ mon-put-metric-alarm MyHighCPUAlarm1 --comparison-operator GreaterThanThreshold --evaluation-periods 1 --metric-name CPUUtilization --namespace "AWS/EC2" --period 600 --statistic Average --threshold 80 --alarm-actions arn:aws:autoscaling:us-east-1:xxxxxxxxxxxxxx:scalingPolicy:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:autoScalingGroupName/autoscleasg1:policyName/MyScaleUpPolicy1 --dimensions "AutoScalingGroupName=autoscleasg" -K KEY.pem -C CERT.pem
OK-Created Alarm

Scale Down Policy
pradeeban@pradeeban:~/pem$ as-put-scaling-policy MyScaleDownPolicy1 --auto-scaling-group autoscleasg1 --adjustment=-1 --type ChangeInCapacity --cooldown 300 -K KEY.pem -C CERT.pem
arn:aws:autoscaling:us-east-1:xxxxxxxxxxxxxx:scalingPolicy:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:autoScalingGroupName/autoscleasg1:policyName/MyScaleDownPolicy1

pradeeban@pradeeban:~/pem$ mon-put-metric-alarm MyLowCPUAlarm --comparison-operator LessThanThreshold --evaluation-periods 1 --metric-name CPUUtilization --namespace "AWS/EC2" --period 600 --statistic Average --threshold 40 --alarm-actions arn:aws:autoscaling:us-east-1:xxxxxxxxxxxxxx:scalingPolicy:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:autoScalingGroupName/autoscleasg1:policyName/MyScaleDownPolicy1 --dimensions "AutoScalingGroupName=autoscleasg" -K KEY.pem -C CERT.pem
OK-Created Alarm

Now the auto scaling group gets an instance, as the minimum number of instances in the auto scaled system has been set to 1,
pradeeban@pradeeban:~/Downloads$ as-describe-auto-scaling-groups autoscleasg -K KEY.pem -C CERT.pem
AUTO-SCALING-GROUP  autoscleasg  autoscalelc  us-east-1c  autoscalelb  1  10  1
INSTANCE  i-xxxxxxxx  us-east-1c  InService  Healthy  autoscalelc

Once the elastic load balancer is set fine and triggered, it starts new nodes or remove the existing nodes according to the load. Following these steps, the system can be load balanced with autoscaling.

Load Balanced Instances' Health
Initially,

pradeeban@pradeeban:~/pem$ elb-describe-instance-health autoscalelb –headers -K KEY.pem -C CERT.pem
INSTANCE_ID  INSTANCE_ID  STATE      DESCRIPTION  REASON-CODE
INSTANCE_ID  i-xxxxxxxx   InService  N/A          N/A

Later with the load, you may see at least a new instance.

pradeeban@pradeeban:~/pem$ elb-describe-instance-health autoscalelb –headers -K KEY.pem -C CERT.pem
INSTANCE_ID  INSTANCE_ID  STATE      DESCRIPTION  REASON-CODE
INSTANCE_ID  i-xxxxxxxx   InService  N/A          N/A

INSTANCE_ID  i-yyyyyyyy   InService  Active Instance

After a few failed attempts, an instance will be marked as 'OutOfService' with the reason, 'Instance has failed at least the UnhealthyThreshold number of health checks consecutively.' or so.

Some interesting read
Setting up an elastic load balancer
Automatically scaling a LAMP application in the cloud
Monitoring performance with Amazon CloudWatch
Examining Load Average - Choosing the metric