Thursday, June 30, 2016

Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service Composition




Today, I presented my paper titled, "Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service Composition" at ICWS 2016 in San Francisco. The presentation slides can be found above. This paper was based on the short-term scientific mission (STSM) I had at the University of Rijeka, Croatia last year.

Abstract: eScience consists of computation-intensive workflows executing on highly distributed networks. Service compositions aggregate web services to automate scientific and enterprise business processes. Along with the increased demand for data quality and Quality of Service (QoS) for an accurate outcome in a shorter completion time, execution of the eScience workflows and service compositions are also required to be distributed efficiently across various geo-distributed nodes. This paper presents Mayan, a Software-Defined Networking (SDN) based approach for service composition.

Mayan i) facilitates an adaptive execution of scientific workflows, ii) offers a more efficient service composition by leveraging distributed execution frameworks, in addition to the traditional web service engines, and iii) enables a very large-scale reliable service composition by finding and consuming the current best-fit among the multiple implementations or deployments of the same service.

Saturday, June 18, 2016

Protecting web sites from the "foreign powers".

OK, this is controversial. :P I found that some websites actually do not function properly outside their intended audience. This (see the screenshot) is what I found when I tried to access trump.com from Portugal. This works well inside the US. Interestingly the Georgia power electricity cooperation web site was malfunctioning from Portugal too.

Wednesday, June 15, 2016

Conflicting log4j implementations

Fixing log4j conflicts can be time consuming. 

I was getting the below exception in my maven build:

Failed to instantiate SLF4J LoggerFactory
Reported exception:
java.lang.NoClassDefFoundError: org/apache/log4j/Level
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
    at org.apache.drill.jdbc.Driver.(Driver.java:34)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.getDatabaseDriver(DrillProvider.java:50)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.getDatabaseDriver(DrillProvider.java:13)
    at edu.emory.cci.bindaas.datasource.provider.genericsql.AbstractSQLProvider.init(AbstractSQLProvider.java:31)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.init(DrillProvider.java:21)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1614)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1555)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1485)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:524)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:461)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:626)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:60)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:325)
    at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290)
    at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level
    at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:501)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 31 more
14 Jun 2016 15:19:46,059 ERROR activator.ContextLoaderListener - Application context refresh failed (OsgiBundleXmlApplicationContext(bundle=drill-datasource-provider, config=osgibundle:/META-INF/spring/*.xml))
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'drillProvider' defined in URL [bundleentry://36.fwk1254526270/META-INF/spring/drill-beans.xml]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/log4j/Level


Choosing  log4j-over-slf4j.jar instead of slf4j-log4j12.jar fixed the error. Means, I should have both log4j-over-slf4j-1.7.21.jar and slf4j-api-1.7.21.jar. Not slf4j-log4j12.jar and slf4j-api-1.7.21.jar.

Including both implementations (log4j-over-slf4j.jar AND slf4j-log4j12.jar) will generate the below exception:
Caused by: java.lang.IllegalStateException: Detected both log4j-over-slf4j.jar AND bound slf4j-log4j12.jar on the class path, preempting StackOverflowError. See also http://www.slf4j.org/codes.html#log4jDelegationLoop for more details.
    at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:54)

Drill in an OSGi container inside a bundle

I was getting the error:
14 Jun 2016 15:36:21,906 ERROR activator.ContextLoaderListener - Application context refresh failed (OsgiBundleXmlApplicationContext(bundle=drill-datasource-provider, config=osgibundle:/META-INF/spring/*.xml))
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'drillProvider' defined in URL [bundleentry://14.fwk1254526270/META-INF/spring/drill-beans.xml]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/drill/jdbc/impl/DriverImpl
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1488)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:524)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:461)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:626)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:60)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:325)
    at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290)
    at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NoClassDefFoundError: org/apache/drill/jdbc/impl/DriverImpl
    at org.apache.drill.jdbc.Driver.(Driver.java:66)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.getDatabaseDriver(DrillProvider.java:50)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.getDatabaseDriver(DrillProvider.java:13)
    at edu.emory.cci.bindaas.datasource.provider.genericsql.AbstractSQLProvider.init(AbstractSQLProvider.java:31)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.init(DrillProvider.java:21)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1614)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1555)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1485)
    ... 14 more
14 Jun 2016 15:36:21,920 ERROR startup.DependencyWaiterApplicationContextExecutor - Unable to create application context for [drill-datasource-provider], unsatisfied dependencies: none
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'drillProvider' defined in URL [bundleentry://14.fwk1254526270/META-INF/spring/drill-beans.xml]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/drill/jdbc/impl/DriverImpl
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1488)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:524)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:461)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:626)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:60)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:325)
    at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
    at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290)
    at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NoClassDefFoundError: org/apache/drill/jdbc/impl/DriverImpl
    at org.apache.drill.jdbc.Driver.(Driver.java:66)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.getDatabaseDriver(DrillProvider.java:50)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.getDatabaseDriver(DrillProvider.java:13)
    at edu.emory.cci.bindaas.datasource.provider.genericsql.AbstractSQLProvider.init(AbstractSQLProvider.java:31)
    at edu.emory.cci.bindaas.datasource.provider.drill.DrillProvider.init(DrillProvider.java:21)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1614)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1555)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1485)
    ... 14 more
14 Jun 2016 15:36:22,068 DEBUG util.DynamicProperties - Loaded default properties


I am sure the DriverImpl class exists inside the drill-java-1.6.0.jar. I had a bunch of jars in the class path, with the relevant entries in the beans.xml.
drill-common-1.6.0.jar
drill-java-exec-1.6.0.jar
drill-jdbc-1.6.0.jar
drill-logical-1.6.0.jar
drill-memory-base-1.6.0.jar
drill-protocol-1.6.0.jar
drill-rpc-1.6.0.jar


I just had to replace all of these with the drill-jdbc-all-1.6.0.jar to the class path and beans.xml to fix this weird error. Probably there is a better fix. But this worked for me. :)

Ant's optional JARs and Maven

Maven is a convenient tool for building and maintaining large multi-module projects. At times it can be tricky to fix a large project that comes with various plugins though. I was trying to incorporate some ant scripts into Maven with maven-antrun-plugin and encountered the below error.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.3:run (compile-dist) on project distribution: An Ant BuildException has occured: The following error occurred while executing this line:
[ERROR] /home/pradeeban/bindaas/build.xml:149: Problem: failed to create task or type propertyfile
[ERROR] Cause: the class org.apache.tools.ant.taskdefs.optional.PropertyFile was not found.
[ERROR] This looks like one of Ant's optional components.
[ERROR] Action: Check that the appropriate optional JAR exists in
[ERROR] -ANT_HOME/lib
[ERROR] -the IDE Ant configuration dialogs
[ERROR]
[ERROR] Do not panic, this is a common problem.
[ERROR] The commonest cause is a missing JAR.
[ERROR]
[ERROR] This is not a bug; it is a configuration problem
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn -rf :distribution

From what I found from the Internet, I downloaded and moved
ant-contrib-1.0b3.jar
ant-nodeps-1.8.1.jar
to the ANT-HOME/lib directory in order to fix this.

Now, I got the below error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.3:run (compile-dist) on project distribution: An Ant BuildException has occured: The following error occurred while executing this line:
[ERROR] /home/pradeeban/bindaas/build.xml:149: Problem: failed to create task or type propertyfile
[ERROR] Cause: Could not load a dependent class org/apache/tools/ant/util/LayoutPreservingProperties
[ERROR] It is not enough to have Ant's optional JARs
[ERROR] you need the JAR files that the optional tasks depend upon.
[ERROR] Ant's optional task dependencies are listed in the manual.
[ERROR] Action: Determine what extra JAR files are needed, and place them in one of:
[ERROR] -ANT_HOME/lib
[ERROR] -the IDE Ant configuration dialogs
[ERROR]
[ERROR] Do not panic, this is a common problem.
[ERROR] The commonest cause is a missing JAR.
[ERROR]
[ERROR] This is not a bug; it is a configuration problem
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn -rf :distribution

After a few minutes of thinking, and ensuring that my Maven and Ant are in the latest versions (as of now), I realized, this must be something to do with the version of the maven-antrun-plugin. I was right!

If you encounter this issue, check the version of the plugin, and define it to the latest if it is not defined (or if it is defined to an earlier version).
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-antrun-plugin</artifactId>
                    <version>1.8</version>
                </plugin>

The issue was fixed successfully.