Software-Defined Simulations for Continuous Development of Cloud and Data Center Networks

Today I am presenting my full paper at CoopIS. This is one of the core works of my PhD, and something that I have been working for, for the last 2 years. The presentation and the abstract are given below. The full paper can be found here.

Today is also my last day at Rhodes, and back to Portugal tomorrow very early morning! It was an amazing week at this lovely island with full of sessions and networking. I also had another paper in the conference last year as well, which was held in the same location, same days. However, I was unable to physically join the conference due to clash in travel schedules. Luckily, this time I made it!

Software-Defined Simulations for Continuous Development of Cloud and Data Center Networks from Pradeeban Kathiravelu

Abstract: Cloud network systems and applications are tested in simulation and emulation environments prior to physical deployments, at different stages of development. Software-Defined Networking (SDN) enables separating logic and execution from the data plane consisting of switches and hosts, to a logically centralized control plane. The global view and control available to the controller enable incremental updates, management, and allocation of resources to the networks. However, unlike the physical networks or the networks emulated by the emulators, current network simulators still lack integration with the SDN controllers.

Hence, currently it is impossible to efficiently orchestrate a simulated network through a centralized controller, or realistically model the controller algorithms and SDN architectures without having the resources for a one-to-one emulation. To address this, this paper presents SDNSim, an SDN simulation middleware, which leverages the principles of SDN for continuous development of cloud and data center networks. SDNSim is an “SDN-aware” network simulator that integrates with the controller through plugins for southbound protocols such as OpenFlow, to execute the algorithms incrementally thus deployed in the control plane.

~~ Life in the small world..~~

London felt colder than Stockholm!

I presented my paper at UCC 2014, on the 8th of December, 2014. This is my master thesis paper. This is the second (also, the last) paper produced from my masters thesis.All the other papers from now onwards are my PhD papers. :P

UK has become the 10th country that I have visited. Have a long list to go. :D My London trip (7th - 12th, December 2014) had two interesting highlights. Second one was, I presented my paper, which was effectively the core of my master thesis work. It was a full paper (so I had 20 + 5 minutes to present).

First one was meeting my room mate (I only had one room mate so far - that was in Stockholm), who made my stay in Stockholm remarkably pleasant, sharing the happy and sad moments of life. We met after almost one year, as we both had moved to different universities, thanks to our mobility programs. 

When we move to another country (or even city), we start everything from the scratch. We learn new things; we change a bit; we hopefully improve. When we move out from the place, we don't move completely. We leave part of us wandering around, in the place that we lived. We become scattered - our memories become. I have only lived in 3 places so far - Colombo (Sri Lanka) -> Lisbon (Portugal) -> Colombo (Sri Lanka) -> Stockholm (Sweden) -> Lisbon (Portugal) was my life in a nutshell since 2012. When I left Lisbon to continue my studies in KTH, Stockholm, I missed Lisbon. Same happened to a considerable extend when I moved back to Lisbon. I missed Stockholm.

We always gain new friends - some we even call family - brothers and sisters from many countries. When we go back, the people we met may have moved out. But the memories remain. 

Meeting my brother who tolerated my weird nature and owl life style for one whole semester surely made a highlight of my London trip. I got the interesting updates since I left Stockholm, while having a quick tour around London. We will meet again. Small world, after all. 

UCC 2014 - An Adaptive Distributed Simulator for Cloud and MapReduce Algorithms and Architectures

Part of my master thesis work has already been presented a work-in-progress paper at MASCOTS 2014. Now, it has also produced a full paper, that is accepted as a full paper to IEEE/ACM 7th International Conference on Utility and Cloud Computing (UCC 2014), to be presented in London, 8th - 11th, December 2014. 

An Adaptive Distributed Simulator for Cloud and MapReduce Algorithms and Architectures

Scalability and performance are crucial for simulations as much as accuracy is. Due to the limited availability and access to the variety of resources, cloud and MapReduce solutions are often evaluated on simulator platforms. As the complexity of the architectures and algorithms keep increasing, simulations themselves become large and resource-hungry. Simulators can be designed to be adaptive, exploiting the clusters and data-grid platforms. This paper describes the research for the design, development, and evaluation of a complete fully parallel and distributed cloud and MapReduce simulator (Cloud2Sim), leveraging the Java in-memory data grid platforms. Cloud2Sim provides a concurrent and distributed cloud simulator, by extending CloudSim cloud simulator, using Hazelcast in-memory key-value store. It also provides an assessment of the MapReduce implementations of Hazelcast and Infinispan, with means of simulating MapReduce executions. Cloud2Sim scales out the cloud and MapReduce simulations to multiple nodes running Hazelcast and Infinispan, based on load. The distributed execution model and adaptive scaling solution could further be leveraged as a general purpose auto-scaler middleware for a multi-tenanted deployment.

Master Thesis Defence and End of EMDC 2012 - 2014

I defended my master thesis last Friday, the 19th of September, 5.30 - 7.00 P.M. My thesis was titled, "An Elastic Middleware Platform for Concurrent and Distributed Cloud and MapReduce Simulations". It had produced 2 publications - 1 work-in-progress paper already published and the other accepted as a full paper, at the time of the defence. I secured 18 out of 20 as the grade, after the defence. I was the last to defend the master thesis in our batch. Hence, my defence also marks the completion of the EMDC batch 2012/2014.

My thesis presentation slides can be found below.
An Elastic Middleware Platform for Concurrent and Distributed Cloud and MapReduce Simulations from Kathiravelu Pradeeban.

 

I kept my blog updated about my life and studies with my 2 years of master studies with EMDC. Thanks for following them. This probably will be one of the last posts on EMDC, except for the occasional posts recalling EMDC from the future. Now I am continuing my PhD from the same university, under the sister program, EMJD-DC. You may expect to find about my PhD life here as well,  as a continuing story of time line from EMDC posts. Interestingly, EMDC has become the mostly blogged title in my blog. I hope EMJD-DC will overtake, eventually.

Special thanks to my supervisor Prof. LV. Thanks to all my friends who came to cheer me up for my defence. My defence also marks the end of the EMDC 2012/2014 batch.. For me, the defence was just a small interval/break before continuing EMJD-DC - Not leaving the place or university. Good bye everyone of EMDC 2012-2014 who is now doing PhD in other schools or working in big corporations.. See you again soon someday somewhere.. 

Keep in touch..




Dissertação: An Elastic Middleware Platform for Concurrent and Distributed Cloud and MapReduce Simulations
Candidato: Pradeeban Kathiravelu
Presidente: Professor José Carlos Alves Pereira Monteiro
Orientador: Professor Luís Manuel Antunes Veiga
Vogal: Doutor Ricardo Jorge Freire Dias
Data: 19/09/2014 -17h30/ Sala 0.20, Pavilhão de Informática II, IST, Alameda
Abstract: Cloud Computing researches involve a tremendous amount of entities such as users, applications, and virtual machines. Due to the limited access and often variable availability of such resources, researchers have their prototypes tested against the simulation environments, opposed to the real cloud environments.  Existing cloud simulation environments such as CloudSim and EmuSim are executed sequentially, where a more advanced cloud simulation tool could be created extending them, leveraging the latest technologies as well as the availability of multi-core computers and the clusters in the research laboratories. While computing has been evolving with multi core programming, MapReduce paradigms, and middleware platforms, cloud and MapReduce simulations still fail to exploit these developments themselves. This research develops Cloud2Sim , which tries to fill the gap between the simulations and the actual technology that they are trying to simulate.
First, Cloud2Sim provides a concurrent and distributed cloud simulator, by extending CloudSim cloud simulator, using Hazelcast in-memory key-value store. Then, it also provides a quick assessment to MapReduce implementations of Hazelcast and Infinispan, adaptively distributing the execution to a cluster, providing means of simulating MapReduce executions. The dynamic scaler solution scales out the cloud and MapReduce simulations to multiple nodes running Hazelcast and Infinispan, based on load. The distributed execution model and adaptive scaling solution could be leveraged as a general purpose auto scaler middleware for a multi-tenanted deployment.

keywords: Cloud Computing, Simulation, Auto Scaling, MapReduce, Volunteer Computing, Cycle Sharing, Distributed Execution

MASCOTS 2014 - Concurrent and Distributed CloudSim Simulations

IEEE 22nd International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2014) was held at Université Paris Descartes in Paris, France, from 9th - 11th, September 2014. I presented my paper, which was a work-in-progress paper based on my master thesis, " Cloud2Sim - An Elastic Middleware Platform for Concurrent and Distributed Cloud and MapReduce Simulations"

The paper is titled, "Concurrent and Distributed CloudSim Simulations". My presentation slides are given below.

Concurrent and Distributed CloudSim Simulations from Kathiravelu Pradeeban

Given below is the abstract of the paper:

Cloud Computing researches involve a tremendous amount of entities such as users, applications, and virtual machines. Due to the limited access and often variable availability of such resources, researchers have their prototypes tested against the simulation environments, opposed to the real cloud environments. Existing cloud simulation environments such as CloudSim and EmuSim are executed sequentially, where a more advanced cloud simulation tool could be created extending them, leveraging the latest technologies as well as the availability of multi-core computers and the clusters in the research laboratories. This research seeks to develop Cloud2Sim, a concurrent and distributed cloud simulator, extending CloudSim while exploiting the features provided by Hazelcast, Infinispan and Hibernate Search to distribute the storage and execution of the simulation. 

MapReduce Implementations - Hazelcast Vs Infinispan

I was testing the MapReduce implementation of Hazlecast with the recent release of Hazelcast 3.2. Then I decided to compare the performance with the Infinispan 6.0.2 MapReduce implementation.

Infinispan outperforming Hazelcast MapReduce implementation

Infinispan outperformed Hazelcast in the sample MapReduce implementation tested on different scenarios, in a single instance, as shown by the figure. Infinispan still outperformed Hazelcast in the nodes up to 6.

Is Infinispan really faster than Hazelcast? Probably it is, as shown by scala-map-benchmarks. Probably, it is something to do with the scenarios, as discussed in Hazelcast group. However, this difference is huge, unlike the previous benchmarks. My opinion is, it is something to do with the still immature MapReduce implementation of Hazelcast, as Hazelcast proven to be quite effective for my other distributed execution tasks. 

If your use case is centred around the MapReduce implementation, I would suggest Infinispan over Hazelcast, as Hazelcast implementation is quite buggy as of 3.2. I have encountered 3 issues so far - a known issue #2105 that was reproduced during MapReduce executions and two other (probably MapReduce implementation specific) issues that I reported - #2354 (Update: This issue has been fixed for 3.2.2 and 3.3 versions of Hazlecast. Thanks Noctarius for attending to this) and #2359. Hazelcast MapReduce might turn to be more scalable and highly performing, once these issues are addressed.

It should be noted that the API of the initial roots of Hazelcast MapReduce implementation (code-named, CastMapR) was inspired heavily by that of the stable and matured MapReduce implementation of Infinispan. The Hazelcast word-count MapReduce example hence follows the same design of that from Infinispan.

I am using Hazelcast 3.2 and Infinispan 6.0.2 for my master thesis at INESC-ID Lisboa. Wait for more updates from the awesome Lisbon. ^_^


Note:
These results are part of the paper given below, which was published in 2014 December. Please cite the paper, if you used these results in your research work.
Kathiravelu, P. & L. Veiga (2014). An Adaptive Distributed Simulator for Cloud and MapReduce Algorithms and Architectures. In IEEE/ACM 7th International Conference on Utility and Cloud Computing (UCC 2014), London, UK. pp. 79 – 88. IEEE Computer Society.