Design decisions of the TCIA data replication and synchronization tool

The initial presentation on the design alternatives and the design decisions of the new TCIA data replication - synchronization tool.

Building and deploying OGSA-DAI/CXF

Building OGSA-DAI/CXF from Trunk
Check out the source code from the svn using,

svn co  https://ogsa-dai.svn.sourceforge.net/svnroot/ogsa-dai ogsa-dai

Go to the folder ogsa-dai/third-party/packages/cxf/2.4.1 and download and extract cxf distribution into a folder. This will make your CXF home as, ogsa-dai/third-party/packages/cxf/2.4.1/apache-cxf-2.4.1. Here the cxf libs will end up in ogsa-dai/third-party/packages/cxf/2.4.1/apache-cxf-2.4.1/lib. The cxf layer has been tested earlier with Apache CXF-2.2.6 and currently is used with Apache CXF 2.4.1. You might try with the other versions where it gets the directory structure of ogsa-dai/third-party/packages/cxf/{version.number}/apache-cxf-{version.number}. 

Building

ogsa-dai/ogsa-dai/trunk/release-scripts/ogsa-dai/cxf$ ant

This will create the OGSA-DAI/CXF source distribution inside ogsa-dai/trunk/release-scripts/ogsa-dai/cxf/build/ogsadai-4.1-cxf-2.4.1-src and binary distribution is built in ogsa-dai/trunk/release-scripts/ogsa-dai/cxf/build/ogsadai-4.1-cxf-2.4.1-src/build/ogsadai-4.1-cxf-2.4.1-bin. 

Deploying

ogsa-dai/ogsa-dai/trunk/release-scripts/ogsa-dai/cxf/build/ogsadai-4.1-cxf-2.4.1-src/build/ogsadai-4.1-cxf-2.4.1-bin$ ant

The built binary is deployed as directory under the Tomcat container in $CATALINA_HOME/webapps/dai. Pls note that with the later releases of ogsadai and cxf, the version numbers in the respective directory structure too will change. However the directory structure will more or less will stay the same.

After deploying, restart the Tomcat Server. Make sure that ogsadai is deployed correct, by referring to the url http://localhost:8080/dai/services/. Services wsdl's can also be viewed from the services listing of dai in Tomcat.

DataRequestExecutionService

DataResourceInformationService 

DataSinkService

DataSourceService 

RequestManagementService

SessionManagementService

Feb 4, 2011 4:59:13 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory dai
Feb 4, 2011 4:59:18 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://ogsadai.org.uk/namespaces/2007/04/service/dataresource/service}DataResourceInformationService from class uk.org.ogsadai.service.cxf.dataresource.WSDataResourceInformationServicePortType
Feb 4, 2011 4:59:19 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /DataResourceInformationService
Feb 4, 2011 4:59:19 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://ogsadai.org.uk/namespaces/2007/04/service/datasink/service}DataSinkService from class uk.org.ogsadai.service.cxf.datasink.WSDataSinkServicePortType
Feb 4, 2011 4:59:20 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /DataSinkService
Feb 4, 2011 4:59:20 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://ogsadai.org.uk/namespaces/2007/04/service/datasource/service}DataSourceService from class uk.org.ogsadai.service.cxf.datasource.WSDataSourceServicePortType
Feb 4, 2011 4:59:20 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /DataSourceService
Feb 4, 2011 4:59:20 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://ogsadai.org.uk/namespaces/2007/04/service/execution/service}DataRequestExecutionService from class uk.org.ogsadai.service.cxf.execution.WSDataRequestExecutionServicePortType
Feb 4, 2011 4:59:20 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /DataRequestExecutionService
Feb 4, 2011 4:59:20 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://ogsadai.org.uk/namespaces/2007/04/service/request/service}RequestManagementService from class uk.org.ogsadai.service.cxf.request.WSRequestManagementServicePortType
Feb 4, 2011 4:59:20 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /RequestManagementService
Feb 4, 2011 4:59:20 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://ogsadai.org.uk/namespaces/2007/04/service/session/service}SessionManagementService from class uk.org.ogsadai.service.cxf.session.WSSessionManagementServicePortType
Feb 4, 2011 4:59:21 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /SessionManagementService
Feb 4, 2011 4:59:21 PM org.apache.cxf.transport.servlet.CXFServlet updateContext
INFO: Load the bus with application context
Feb 4, 2011 4:59:21 PM org.apache.cxf.transport.servlet.AbstractCXFServlet replaceDestinationFactory
INFO: Servlet transport factory already registered
Feb 4, 2011 4:59:21 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory docs
Feb 4, 2011 4:59:21 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory ROOT
Feb 4, 2011 4:59:21 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory examples
Feb 4, 2011 4:59:22 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
Feb 4, 2011 4:59:22 PM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
Feb 4, 2011 4:59:22 PM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/69  config=null
Feb 4, 2011 4:59:22 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 28740 ms

OGSA-DAI ~Presenting in CXF~

Coding has started to accelerate. Revision 1065 customizes the built-in JAXB mapping, by an external binding file. This changes the default mapping of xs:dateTime from XMLGregorianCalendar to java.util.Calendar. For the ease of further commits, I added the respective client and server implementation folder structure in the commit 1066 along with the utility class CXFUtils.

1069 extends the Request Status Builder for the cxf client. 1070 includes the utility methods for handling the exceptions for the client toolkit. 1076 was an interesting  [ ;) ] fix, which fixes some years which I overlooked.

As with the common case of CXF stub generation, Lists are generated instead of arrays.
An example is,
-  final DataType[] inputData = inputBean.getInputLiteral();
+  final DataType[] inputData = inputBean.getInputLiteral().toArray(new DataType[inputBean.getInputLiteral().size()]);

 -              inputType.setInputLiteral(mCurrentData.getDataTypeArray());
+           inputType.getInputLiteral().clear();
+           inputType.getInputLiteral().addAll(Arrays.asList(mCurrentData.getDataTypeArray()));

1093 fixes some issues in an earlier commit related to this.

1094 and 1095 includes the CXF compliant Request management and Session Management Services respectively. 1096 is the CXF implementation of  the WS-ResourceLifetime portType operations.

1118 includes classes in execution/workflows. 

Classes in execution/Worflows.

Considerable changes in the stub generation includes,
(1) Inner class InputsType.Input replaces the class InputsTypeInput.

(2) Getters of the list returns a reference to the live list, not a snapshot. Hence to an item, getOutputStream().add(newItem);

from the method  public List getOutputStream() in OutputsType class.

(3) Regarding the generated fault types, in case of Axis, it was in the format of XXXType from the stub which needed to be used in the code, in cxf stubs, we have XXXTypes associated with XXX, where XXX from the stub is used in the code. [XXX denotes the faults.]

These changes in the stub code are reflected in the respective server side code.

1119 - The CXF implementation of Request Builder implementation. 1122, handles the AddressingUtils, which parses the SOAP Message Header.

@WebServiceProvider
public class AddressingUtils
{
....
    @Resource
    private static WebServiceContext context;


        SOAPMessageContext soapMsgContext = (SOAPMessageContext) context.getMessageContext();
        SOAPEnvelope envelope = soapMsgContext.getMessage().getSOAPPart().getEnvelope();

Commit 1128 defines CXF_MESSAGE_CONTEXT_ERROR. Commits 1144, 1154, and 1173 are for the cxf compliant data sink service, execution, and data resource information services, while 1161 and 1162 implement the data source and intrinsics  portType operations. 1159 is for the initialization class for CXF-compliant presentation layers, which is being modified at the moment. 1172 does some minor fixes in the exceptions thrown.

Designing an alternative presentation layer!

My first post on the OGSA-DAI blog, on my GSoC 2010 project, with an introduction to the project, initial design decisions, as well as the Google Summer of Codes updates upto the moment, can be found here.

Commit 994 has become my first commit. Apache CXF 2.2.6 has to be downloaded from here and extracted to "ogsa-dai/third-party/packages/cxf/2.2.6" *** (as the case with the Axis presentation layer). After that, "ant all" from"trunk/presentation/cxf/stubs" will produce ogsadai-4.1-cxf-stubs-1.0.jar. This commit is just for the creation of the stubs jar using CXF, and further commits will follow. After the commit, I have included the module info the  repository structure trac page. The OGSA-DAI nightly tests check the modules by running each night on the server [1].

***
cxf.lib.dir=${packages.dir}/cxf/2.2.6/apache-cxf-2.2.6/lib

cxf.version=2.2.6

Axis specific wsdl (data_request_execution_service_service.wsdl)
<soap:address location="http://localhost:8080/axis/services/DataRequestExecutionService"/>

CXF specific wsdl
<soap:address location="http://localhost:8080/services/DataRequestExecutionService"/>


  <wsdl:import namespace="http://ogsadai.org.uk/namespaces/2007/04/service/execution/bindings" location="data_request_execution_service_bindings.wsdl"/>
  <wsdl:service name="DataRequestExecutionService">
    <wsdl:port name="WSDataRequestExecutionServicePortTypePort" binding="binding:WSDataRequestExecutionServicePortTypeSOAPBinding">
      <soap:address location="http://localhost:8080/services/DataRequestExecutionService"/>
    </wsdl:port>
  </wsdl:service>

Apache CXF seems to be releasing the builds once in a two months. Currently they have released 2.2.8, though I have started with 2.2.6. I have seen many articles on migrating from Apache Axis 1.x to Axis 2. But articles on migrating to CXF from Axis is rare. Though this makes the task more challenging, it has also given me a chance for a research which of course is really cool.

Java WS Core/Final WSRF Migration Guide  describes the code changes of the Java WS Core Framework of Globus Alliance, including WS-Addressing and Faults.

Building the modules using ANT instead of plain java or CXF WSDLToJava alone is interesting and make the task easier, given the fact that we have many wsdls. The below code segments** are for the existing presentation layers, which are removed for cxf at the moment, as CXF comes with support for WS-Addressing 1.0.  (yet to discuss whether to use the CXF libs or the customized ones from globus/ws-addressing as it is for the existing layers). More to analyze the wsdd for the Axis based layer, and I may need to customize the JAXB bindings, as well. 

**
# WS-Addressing
addressing.lib.dir=${dependencies.dir}/globus/ws-addressing/addressing/1.0

[1]  Test Reports 20/05/2010 22:05:50 : daitest

[2] Coding Guidelines

In this post, WSDL code was written using the HTML encoder/decoder provided here.

[GSoC 2010] Alternative Presentation Layer for OGSA-DAI

Having the vision of sharing data resources to enable collaboration, Open Grid Services Architecture Data Access and Integration (OGSA-DAI) provides access, transformation, integration, and delivery of data. It supports Apache Axis 1.4 and Globus Toolkit (gt-4.2), available to download for both the two platforms.
 
Under Open Middleware Infrastructure Institute (OMII-UK), an alternative presentation layer using Apache CXF instead of the outdated Axis which is currently used, is discussed as a project for Google Summer of Codes 2010, which I found very interesting for this summer.  

Project Proposal

Project Mentor: Ally Hume

OGSA-DAI has a layered architecture consists of the data layer, business logic layer, presentation layer, and client layer, when considering the architecture in bottom-up. Presentation layer encapsulates the required functionalities to expose the data service resources using the web service interfaces. Presentation layer currently consists of two realization, where each of them have  a WSDL that describes the interface. 

Both the realizations are compliant with Web Services Resource Framework (WSRF), and communicate with the respective client stubs using SOAP over HTTP. Hence design and implementation of an alternative RESTful presentation layer using CXF JAX-RS (The Java API for RESTful Web Services) is also included in this project, as it will be lightweight and best for bespoke clients.

This seems an interesting initiative as the performance and the simplicity will be highly improved. I have seen reasonable comparison of Axis 2 and CXF [1], [2], though they must be outdated by this time.

[1] Technical Notes on OGSA-DAI
[2] Resource on writing a presentation layer
[3] OGSA-DAI Tutorial
[4] FAQ: OGSA-DAI WSI-specific (Based on Axis 1.4 web service stack)
[5] FAQ: OGSA-DAI WSRF-specific (Based on the globus toolkit)
[6] General FAQ
[7] Known Problems
[8] Binary Transfer Benchmarks 
[9] GSoC 2010 Announcement in OGSA-DAI blog
[10] Google Summer of Code 2010 site