Being a mail star.. :D

So it was a huge collection of unread mail. Today I took some time to read through the 1000s of them. Nevertheless it feels so robotic, having an inbox with a few hundred thousands of mail - 80% mail from IT-groups, svn, etc, 10% fun mail from friends bulk sent, 5% spam/scam/hoax, 5% other group mails. No personal mail at all, almost. Regardless of the fact that I have proper mail filters, I have also missed some interesting mails with valuable information long time, it seems. Nothing personal though.

The Arpanet dialogues dated back to 1975 - This archive of Arpanet dialogues  provides the initial conversations between Marcel Broodthaers, Jane Fonda, Ronald Reagan & Edward Said and Samir Amin, Steve Biko, Francis Fukuyama & Minoru Yamasaki. It is good to notice that Sri Lanka was mentioned in their first conversation. :)

At an interview, 

The interviewer says, "Our work environment expects you to work 24/7. We do not provide you with any casual leave, unless given a medical evidence or a police report. You should also be ready to work on holidays. You should be on time to office and should be strictly on formal kit, always." 

The interviewee smiles, "Sir, I suit your job much."

"Are you confident?, Why do you say so?" asks the interviewer.

"Yes Sir, I am a masochist."

"There! You are selected"..

My mail box is funny.. :D

Benchmarking Appserver remotely - ab, nohup, ec2, and more ..

This time we wanted to benchmark WSO2 Stratos Application Server instances running on two amazon ec2 instances. Both of the instances are proxied by WSO2 Load Balancer. Here we have used appserver.cloud1.wso2.com which has only the load balancing, and appserver.cloud2.wso2.com with load balancing and auto-scaling.

Now we picked Java Bench (A clone of Apache Bench - ab) for the purpose to benchmark the application server running over the two amazon ec2 instances. It was a long running test from another ec2 instance. This test lasted for more than 30 hours. We were able to benchmark and also notice a few interesting stuff too.

Here even after closing the connection to the remote instance where the test is running, the test should keep running.

Now we are running the test using the load of web service calls on a web service that is deployed on Application Server. The service is a simple adder service.

The below is stored as request.xml

From two directories cloud1 and cloud2, from the instance,

nohup ab -p request.xml -n 10000 -c 200 -k -H "SOAPAction: multiply" -T "application/soap+xml; charset=UTF-8" http://appserver.cloud1.wso2.com/services/HelloService &

nohup ab -p request.xml -n 10000 -c 200 -k -H "SOAPAction: multiply" -T "application/soap+xml; charset=UTF-8" http://appserver.cloud2.wso2.com/services/HelloService &

Here nohup comes handy. Even after you terminate the connection to the remote host, the test will continue, unless you find the process and kill it using kill -9.

You can view the nohup output from the above directories cloud1 and cloud2 above using tail -f nohup.out 

But since ab itself can't handle this much of a bulky requests at once, it may fail as below, timing ab out for the specific requests.

Benchmarking appserver.cloud1.wso2.com (be patient)
apr_poll: The timeout specified has expired (70007)

So let's create a shell script instead, to prevent firing all the requests to ab at once, like a million of requests. 

while true; do ab -p request4.xml -n 4000 -c 200 -k -H "SOAPAction: multiply" -T "application/soap+xml; charset=UTF-8" http://appserver.cloud1.wso2.com/services/HelloService; sleep 3; ab -p request4.xml -n 4000 -c 200 -k -H "SOAPAction: multiply" -T "application/soap+xml; charset=UTF-8" http://appserver.cloud2.wso2.com/services/HelloService; sleep 3; done

Save the above script (test.sh). Now by running 

nohup sh test.sh &

you can start the load test for both the instances.

ubuntu@ip-10-120-61-234:~$ nohup sh dual.sh &
[1] 6512
ubuntu@ip-10-120-61-234:~$ nohup: ignoring input and appending output to `nohup.out'

Here 6512 above indicates the pid of dual.sh, which is the benchmarking process of ab.

You can see the nohup output for the above script with the results from both the instances alternating, with a gap of 3 seconds. Here the time interval between two chunks of requests on the same instance is 6 seconds.

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking appserver.cloud1.wso2.com (be patient)
Completed 400 requests
Completed 800 requests
Completed 1200 requests
Completed 1600 requests
Completed 2000 requests
Completed 2400 requests
Completed 2800 requests
Completed 3200 requests
Completed 3600 requests
Completed 4000 requests
Finished 4000 requests


Server Software:        WSO2
Server Hostname:        appserver.cloud1.wso2.com
Server Port:            80

Document Path:          /services/HelloService
Document Length:        260 bytes

Concurrency Level:      200
Time taken for tests:   25.772 seconds
Complete requests:      4000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1652000 bytes
Total POSTed:           2188000
HTML transferred:       1040000 bytes
Requests per second:    155.20 [#/sec] (mean)
Time per request:       1288.623 [ms] (mean)
Time per request:       6.443 [ms] (mean, across all concurrent requests)
Transfer rate:          62.60 [Kbytes/sec] received
                        82.91 kb/s sent
                        145.50 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    2   5.3      1      81
Processing:     7 1256 711.9   1290    4351
Waiting:        6 1256 711.9   1290    4351
Total:          7 1258 711.7   1293    4363

Percentage of the requests served within a certain time (ms)
  50%   1293
  66%   1696
  75%   1883
  80%   1957
  90%   2165
  95%   2292
  98%   2416
  99%   2447
 100%   4363 (longest request)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking appserver.cloud2.wso2.com (be patient)
Completed 400 requests
Completed 800 requests
Completed 1200 requests
Completed 1600 requests
Completed 2000 requests
Completed 2400 requests
Completed 2800 requests
Completed 3200 requests
Completed 3600 requests
Completed 4000 requests
Finished 4000 requests


Server Software:        WSO2
Server Hostname:        appserver.cloud2.wso2.com
Server Port:            80

Document Path:          /services/HelloService
Document Length:        260 bytes

Concurrency Level:      200
Time taken for tests:   20.210 seconds
Complete requests:      4000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1652000 bytes
Total POSTed:           2188000
HTML transferred:       1040000 bytes
Requests per second:    197.92 [#/sec] (mean)
Time per request:       1010.486 [ms] (mean)
Time per request:       5.052 [ms] (mean, across all concurrent requests)
Transfer rate:          79.83 [Kbytes/sec] received
                        105.73 kb/s sent
                        185.55 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    1   1.1      1      14
Processing:     6  987 577.5    990    3700
Waiting:        6  987 577.5    990    3700
Total:          6  988 577.4    990    3700

Percentage of the requests served within a certain time (ms)
  50%    990
  66%   1292
  75%   1468
  80%   1570
  90%   1782
  95%   1907
  98%   1986
  99%   2013
 100%   3700 (longest request)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

This nohup.out contains the useful load testing results.

Fixing 'Too many open files' issue in Load Balancer

When load balancing a production system like Stratos with heavy load of requests, or while load testing it, you may face a 'Too many open files' issue as below.

[2011-06-09 20:48:31,852]  WARN - HttpCoreNIOListener System may be unstable: IOReactor encountered a checked exception : Too many open files
java.io.IOException: Too many open files
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
    at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvent(DefaultListeningIOReactor.java:129)
    at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:113)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:315)
    at org.apache.synapse.transport.nhttp.HttpCoreNIOListener$2.run(HttpCoreNIOListener.java:253)
    at java.lang.Thread.run(Thread.java:662)

This is a common exception in load balancer, if the maximum allowed files  (ulimit) to be open is set to low.

You can get the number by, ulimit -Sn. By default it can be as low as 1024 for your desktop, server, or the ec2 instance. You can fix the above exception by increasing the ulimit by giving a higher value (given 655350 here).

ulimit -n 655350

This will fix the above issue.

Setting up Stratos locally in your laptop

Now you can check out the trunk code of Carbon from the repository, build, and configure WSO2 Stratos in a few steps.

Check out https://svn.wso2.org/repos/wso2/trunk/carbon
mvn clean install from carbon directory

Set up the following in ~/.bashrc (change the values accordingly for the email addresses, carbon directory, the transport sender, and the other information given below.)

# Stratos setup constants
export stratos_local_ip="10.100.0.15"

export CARBON_DIR="/home/pradeeban/trunk/stratos/carbon"
export STRATOS_DIR=$CARBON_DIR/stratos_home
export STRATOS_VERSION=1.1.0-SNAPSHOT
export SSO_ENABLED=true
export CREATE_DB=true
export STRATOS_MAIL_TRANSPORT=' ********.***.***25 false false cloud-noreply@wso2.com '
export NOTIFICATION_EMAIL=pradeeban@wso2.com

If you are going to use the stratos binaries, you can use the downloaded packs, usually available from the builders online (find it here, for the Stratos-1.5.1 binaries), as of the trunk or development branch version or from the wso2.org download area, for a released version.

Place the downloaded zips in to a location and also define that location in bashrc profile.
export PACKS_DIR="/home/pradeeban/Downloads"

Now you may run the set up script as above.

Running Stratos Set up script - carbon/build/stratos-setup.pl
./stratos-setup.pl

Add the following lines to your /etc/hosts file

127.0.0.1 cloud-test.wso2.com
127.0.0.1 identity.cloud-test.wso2.com
127.0.0.1 governance.cloud-test.wso2.com
127.0.0.1 appserver.cloud-test.wso2.com
127.0.0.1 bam.cloud-test.wso2.com
127.0.0.1 dss.cloud-test.wso2.com
127.0.0.1 bps.cloud-test.wso2.com
127.0.0.1 brs.cloud-test.wso2.com
127.0.0.1 cep.cloud-test.wso2.com
127.0.0.1 esb.cloud-test.wso2.com
127.0.0.1 gadget.cloud-test.wso2.com
127.0.0.1 mb.cloud-test.wso2.com
127.0.0.1 mashup.cloud-test.wso2.com


Start all the Stratos services
from $stratos_home
sh stratos.sh start all

Stop all the Stratos services
form $stratos_home
sh stratos.sh stop all

Also you can start and stop the services individually or as a chosen subset of all the available services.

Here by pointing the browser to cloud-test.wso2.com, you can access Stratos Manager. Pls note that in the configuration we have enabled SSO. You can also set up the Stratos services with SSO disabled instead, by setting export SSO_ENABLED=false, above in .bashrc profile. 

Since we have enabled SSO as above, once you have logged in to one of the services, you can just click and enter the other services without logging into each of the other services separately. Here identity.cloud-test.wso2.com is the url for WSO2 Identity Server as a Service, and governance.cloud-test.wso2.com is WSO2 Governance Registry as a Service, and the other urls as given above lists the respective services.

Await the official Stratos-1.5 release soon!

How to close an open port in unix/linux environments

In many cases it is not uncommon to see exception like "java.net.BindException: Address already in use" when running programs. Sometimes this exception occurs even after the program that was using the particular address/port has been terminated. That is mostly due to improperly terminated application. Here the port is still assigned to the process of the particular program. How can we release the port and let the new program use that specific port, without throwing address already in use exceptions.

First let's find the process attached to the program running on the particular port.

pradeeban@pradeeban:~/LB/wso2lb-1.0.0/bin$ lsof | grep 8243
notificat  1802  pradeeban  txt       REG                8,4     53296   13638243 /usr/lib/gnome-panel/notification-area-applet
firefox-b  1821  pradeeban   52u     IPv4             896036       0t0        TCP pradeeban.local:43306->pradeeban.local:8243 (ESTABLISHED)
java      10500  pradeeban  113u     IPv6             867578       0t0        TCP pradeeban.local:8243->pradeeban.local:45898 (CLOSE_WAIT)
java      10500  pradeeban  300u     IPv6             867966       0t0        TCP pradeeban.local:8243->pradeeban.local:46006 (CLOSE_WAIT)
java      10500  pradeeban  471u     IPv6             858667       0t0        TCP *:8243 (LISTEN)
Here, lsof lists open files. From the above command, we notice that a process having the pid of 10500 is listening on the port 8243.

If you know the name of the running application process or part of it, you can use

ps -xa | grep

instead, to get the pid.

Now we can easily release the port 8243 by killing the relevant process (process of pid 10500).

pradeeban@pradeeban:~/LB/wso2lb-1.0.0/bin$ kill 10500