Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service Composition

Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service Composition from Pradeeban Kathiravelu

Today, I presented my paper titled, "Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service Composition" at ICWS 2016 in San Francisco. The presentation slides can be found above. This paper was based on the short-term scientific mission (STSM) I had at the University of Rijeka, Croatia last year. I was part of the organization committee as a student volunteer in ICWS/CLOUD/SCC/Services/MS/BigData_Congress 2016. This was the first time I was more than just a presenter in a conference. It was a great experience for all the student volunteers!

Abstract: eScience consists of computation-intensive workflows executing on highly distributed networks. Service compositions aggregate web services to automate scientific and enterprise business processes. Along with the increased demand for data quality and Quality of Service (QoS) for an accurate outcome in a shorter completion time, execution of the eScience workflows and service compositions are also required to be distributed efficiently across various geo-distributed nodes. This paper presents Mayan, a Software-Defined Networking (SDN) based approach for service composition.

Mayan i) facilitates an adaptive execution of scientific workflows, ii) offers a more efficient service composition by leveraging distributed execution frameworks, in addition to the traditional web service engines, and iii) enables a very large-scale reliable service composition by finding and consuming the current best-fit among the multiple implementations or deployments of the same service.

SPREAD — System for Sharing and Publishing Research Data

We presented our work on MEDIator/SPREAD at SIIM 2016 in Portland, OR. 

 

In this study we developed a set of web-services that abstract the task of replicating and publishing subsets of data that are stored in curated biomedical data repositories. The overarching goal is to develop tools that can be configured against data repositories and give researchers, during the course of a research study, the ability to seamlessly share, with collaborators, data subsets; and upon conclusion of the study, publish the datasets for wider consumption.

 

The full paper can be found here.

Protecting web sites from the "foreign powers".

OK, this is controversial. :P I found that some websites actually do not function properly outside their intended audience. This (see the screenshot) is what I found when I tried to access trump.com from Portugal. This works well inside the US. Interestingly the Georgia power electricity cooperation web site was malfunctioning from Portugal too.

Conflicting log4j implementations

Fixing log4j conflicts can be time consuming. 

I was getting the below exception in my maven build:

Failed to instantiate SLF4J LoggerFactory
Reported exception:
java.lang.NoClassDefFoundError: org/apache/log4j/Level
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
    at org.apache.drill.jdbc.Driver.(Driver.java:34)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.getDatabaseDriver(DrillProvider.java:50)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.getDatabaseDriver(DrillProvider.java:13)
    at edu.emory.cci.bindaas.datasource.provider.genericsql.AbstractSQLProvider.init(AbstractSQLProvider.java:31)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.init(DrillProvider.java:21)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1614)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1555)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1485)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:524)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:461)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:626)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:60)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:325)
    at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290)
    at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:501)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 31 more
14 Jun 2016 15:19:46,059 ERROR activator.ContextLoaderListener - Application context refresh failed (OsgiBundleXmlApplicationContext(bundle=drill-datasource-provider, config=osgibundle:/META-INF/spring/*.xml))
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'drillProvider' defined in URL [bundleentry://36.fwk1254526270/META-INF/spring/drill-beans.xml]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/log4j/Level


Choosing  log4j-over-slf4j.jar instead of slf4j-log4j12.jar fixed the error. Means, I should have both log4j-over-slf4j-1.7.21.jar and slf4j-api-1.7.21.jar. Not slf4j-log4j12.jar and slf4j-api-1.7.21.jar.

Including both implementations (log4j-over-slf4j.jar AND slf4j-log4j12.jar) will generate the below exception:
Caused by: java.lang.IllegalStateException: Detected both log4j-over-slf4j.jar AND bound slf4j-log4j12.jar on the class path, preempting StackOverflowError. See also http://www.slf4j.org/codes.html#log4jDelegationLoop for more details.
    at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:54)

Drill in an OSGi container inside a bundle

I was getting the error:

14 Jun 2016 15:36:21,906 ERROR activator.ContextLoaderListener - Application context refresh failed (OsgiBundleXmlApplicationContext(bundle=drill-datasource-provider, config=osgibundle:/META-INF/spring/*.xml))
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'drillProvider' defined in URL [bundleentry://14.fwk1254526270/META-INF/spring/drill-beans.xml]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/drill/jdbc/impl/DriverImpl
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1488)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:524)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:461)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:626)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:60)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:325)
    at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290)
    at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NoClassDefFoundError: org/apache/drill/jdbc/impl/DriverImpl
    at org.apache.drill.jdbc.Driver.(Driver.java:66)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.getDatabaseDriver(DrillProvider.java:50)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.getDatabaseDriver(DrillProvider.java:13)
    at edu.emory.cci.bindaas.datasource.provider.genericsql.AbstractSQLProvider.init(AbstractSQLProvider.java:31)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.init(DrillProvider.java:21)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1614)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1555)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1485)
    ... 14 more
14 Jun 2016 15:36:21,920 ERROR startup.DependencyWaiterApplicationContextExecutor - Unable to create application context for [drill-datasource-provider], unsatisfied dependencies: none
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'drillProvider' defined in URL [bundleentry://14.fwk1254526270/META-INF/spring/drill-beans.xml]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/drill/jdbc/impl/DriverImpl
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1488)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:524)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:461)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:626)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:60)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:325)
    at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290)
    at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NoClassDefFoundError: org/apache/drill/jdbc/impl/DriverImpl
    at org.apache.drill.jdbc.Driver.(Driver.java:66)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.getDatabaseDriver(DrillProvider.java:50)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.getDatabaseDriver(DrillProvider.java:13)
    at edu.emory.cci.bindaas.datasource.provider.genericsql.AbstractSQLProvider.init(AbstractSQLProvider.java:31)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.init(DrillProvider.java:21)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1614)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1555)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1485)
    ... 14 more
14 Jun 2016 15:36:22,068 DEBUG util.DynamicProperties - Loaded default properties


I am sure the DriverImpl class exists inside the drill-java-1.6.0.jar. I had a bunch of jars in the class path, with the relevant entries in the beans.xml.
drill-common-1.6.0.jar
drill-java-exec-1.6.0.jar
drill-jdbc-1.6.0.jar
drill-logical-1.6.0.jar
drill-memory-base-1.6.0.jar
drill-protocol-1.6.0.jar
drill-rpc-1.6.0.jar

I just had to replace all of these with the drill-jdbc-all-1.6.0.jar to the class path and beans.xml to fix this weird error. Probably there is a better fix. But this worked for me. :)

Ant's optional JARs and Maven

Maven is a convenient tool for building and maintaining large multi-module projects. At times it can be tricky to fix a large project that comes with various plugins though. I was trying to incorporate some ant scripts into Maven with maven-antrun-plugin and encountered the below error.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.3:run (compile-dist) on project distribution: An Ant BuildException has occured: The following error occurred while executing this line:

[ERROR] /home/pradeeban/bindaas/build.xml:149: Problem: failed to create task or type propertyfile

[ERROR] Cause: the class org.apache.tools.ant.taskdefs.optional.PropertyFile was not found.

[ERROR] This looks like one of Ant's optional components.

[ERROR] Action: Check that the appropriate optional JAR exists in

[ERROR] -ANT_HOME/lib

[ERROR] -the IDE Ant configuration dialogs

[ERROR]

[ERROR] Do not panic, this is a common problem.

[ERROR] The commonest cause is a missing JAR.

[ERROR]

[ERROR] This is not a bug; it is a configuration problem

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn -rf :distribution

From what I found from the Internet, I downloaded and moved

ant-contrib-1.0b3.jar

ant-nodeps-1.8.1.jar

to the ANT-HOME/lib directory in order to fix this.

Now, I got the below error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.3:run (compile-dist) on project distribution: An Ant BuildException has occured: The following error occurred while executing this line:

[ERROR] /home/pradeeban/bindaas/build.xml:149: Problem: failed to create task or type propertyfile

[ERROR] Cause: Could not load a dependent class org/apache/tools/ant/util/LayoutPreservingProperties

[ERROR] It is not enough to have Ant's optional JARs

[ERROR] you need the JAR files that the optional tasks depend upon.

[ERROR] Ant's optional task dependencies are listed in the manual.

[ERROR] Action: Determine what extra JAR files are needed, and place them in one of:

[ERROR] -ANT_HOME/lib

[ERROR] -the IDE Ant configuration dialogs

[ERROR]

[ERROR] Do not panic, this is a common problem.

[ERROR] The commonest cause is a missing JAR.

[ERROR]

[ERROR] This is not a bug; it is a configuration problem

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn -rf :distribution

After a few minutes of thinking, and ensuring that my Maven and Ant are in the latest versions (as of now), I realized, this must be something to do with the version of the maven-antrun-plugin. I was right!

If you encounter this issue, check the version of the plugin, and define it to the latest if it is not defined (or if it is defined to an earlier version).

                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-antrun-plugin</artifactId>
                    <version>1.8</version>
                </plugin>

The issue was fixed successfully.

Moringa Leaf Powder from Atlanta YDFM

Moringa leaf powder, the "super food".

Some marketing material, back side.

Your Dekalb Farmers Market (YDFM), one of my favourites in Atlanta. Every visit, I find something new. During the last couple of visits, I noticed something called "Moringa tea" and "Moringa Powder" from Africa. This time, I decided to buy it. I thought it was 8.99 USD. But at the cashier, I realized it was 18.99 USD. However, I still bought it. I came home, and attempted to consume it mixing with some warm milk. It turned out to be the powder made from the leaves of drumstick or horseradish tree, which in Sinhala called "Moringa". Moringa leaves are something which are readily and freely available in Sri Lanka.

While we consume the drumstick itself, I never thought we eat the leaf powder. It is bitter, and funny I spent this much for the leaf powder (probably 100 times more expensive than in Sri Lanka). This powder also tastes like the bay leaf powder, which I also bought in YDFM for just 1 USD (around twentieth of the price). Probably I am the only one who bought this overpriced item in the recent days. Surprisingly, being a Sri Lankan, I should have known what does moringa mean. :)

[MiPro] A FIRM Approach for Software-Defined Service Composition

We presented our work at MiPro-CTI 2016 (The 39th international convention on information and communication technology, electronics and microelectronics - Telecommunications & Information /CTI) in Croatia. The paper was based on my first STSM (short-term scientific mission) ICT COST Action IC1304, a part of my PhD. It is based on our work in Croatia during last year.

Abstract: Service composition is an aggregate of services often leveraged to automate the enterprise business processes. While Service Oriented Architecture (SOA) has been a forefront of service composition, services can be realized as efficient distributed and parallel constructs such as MapReduce, which are not typically exploited in service composition. With the advent of Software-Defined Networking (SDN), global view and control of the entire network is made available to the networking controller, which can further be leveraged in application level. This paper presents FIRM , an approach for Software-Defined Service Composition by leveraging SDN and MapReduce. FIRM comprises Find, Invoke, Return, and Manage, as the core procedures in achieving a QoS-Aware Service Composition.

The proceedings and full text can be found here.

Software-Defined Approach for QoS and Data Quality in Multi-Tenant Clouds

This time we had the EMJD-DC Spring Event co-located with the 2nd Portugal|UT Austin Summer School in Systems and Networking at Costa da Caparica. I presented my PhD thesis at the event. The presentation slides are given below.

 

Software-Defined Approach for QoS and Data Quality in Multi-Tenant Clouds from Kathiravelu Pradeeban