[GSoC 2016] MediCurator : Near Duplicate Detection for Medical Data Warehouse Construction

This summer, at the Department of Biomedical Informatics, Emory University (Emory BMI), we have another set of intelligent students working on interesting projects. I have been mentoring Yiru Chen (Irene) from Peking University, on the project "MediCurator: Near Duplicate Detection for Medical Data Warehouse Construction" for the past couple of months. Currently we have reached the final stages of the project, as the student evaluation period starts on the 15th of August. This post is a summary of this successful GSoC, as well as a history behind the near duplicate detection efforts.

The early history of MediCurator
MediCurator was a research prototype that I initially developed based on my paper ∂u∂u Multi-Tenanted Framework: Distributed Near Duplicate Detection for Big Data (CoopIS'15) as part of my data quality research, along with my GSoC 2015 work on data integration. The early results were presented as a poster at AMIA 2016 in San Francisco.

MediCurator and Infinispan

Now we have a more complete implementation of MediCurator and a use case for medical data, thanks to the support provided by GSoC. For her implementation, Irene did some benchmarks before choosing to go with the Infinispan's latest distributed streams for the distributed execution. (You may find some interesting discussion on the Infinispan distributed streams here.)

MediCurator Usecase
MediCurator is a data quality platform for the ETL workflows in data warehouse construction. It optimizes the bandwidth usage by avoiding the duplicate downloads, and optimizes the storage by eliminating the near duplicates in the warehouse thus increasing the data quality. When data is downloaded, the source locations are tracked, and when data is updated in the source at a latter time, the subsequent download process will download only the new data.

Similarly, data is deduplicated at the data warehouse, as near duplicates could be present there since data is integrated from multiple data sources. Here the data pairs are evaluated for near duplicates in a distributed manner, with duplicate pairs stored separately, while the clean data stays in the warehouse. The duplicate detection workflow also considers the corrupted data/metadata, and synchronizes/downloads the clean data from the source.

This is useful for medical images due to the large scale of the data, often binary in nature along with textual metadata. Efficiency of MediCurator is ensured through its in-memory data grid-based architecture. MediCurator fits well with the landscape of distributed data integration and federation platforms developed at Emory BMI.

More Details on GSoC 2016

Irene developed the entire code base from scratch as an open source project. MediCurator also has a ReadTheDocs* based documentation which gives more detailed description to the project. In addition, you may learn the summary of weekly progresses at Irene's blog. MediCurator's scope remained dynamic throughout the project. MediCurator has download tracking and detecting duplicates across the datasets online and offline, in addition to the near duplicate detection. Most of the code was developed exclusively having the cancer imaging archive (TCIA) as the core data source with DICOM as the default data format, while maintaining relevant interfaces and APIs for extension to other data sources and data types.

Future Work

The summer was productive. It included both research and implementations. The GSoC time is limited to 4 months (including the community bonding period), and we are reaching a successful end to a yet another Google Summer of Code. Nevertheless, we hope to work on a research publication with combined results on MediCurator, along with the previous ∂u∂u** and SDN-based Mayan (presented at ICWS 2016) approaches in November. This will be our first publication with Irene on her findings and implementations, with further evaluations on the clusters in INESC-ID Lisboa. More updates on this later (possibly after publishing the paper ;)).

Concluding Remarks
This is my 4th time in the Google Summer of Code as a mentor, and 3rd time as the primary mentor for a project. Previously I mentored 2 successful students in 2011 and 2012 for AbiWord. I wish every student success as they reach the end of their summer of code.

* I recommend ReadTheDocs. You should give a try!
** You may find the paper on ∂u∂u interesting, if you are into data quality or distributed near duplicate detection.

Data Sharing Synchronization System with Infinispan

The below presentation describes part of the data sharing and synchronization system that I developed during the summer. This presentation focuses mostly on the medical data and images, along with the cancer imaging archive (TCIA) as the data source. The latter part of the summer was spent, extending the tool for the other data sources, such as Amazon S3, CA Microscope, or even meta data stored in a CSV file.

  

Big Data – An Introduction and a Look from In-Memory Data Grids

With the increasing number of sensors, computers, and smart phones more and more data is stored digitally. Traditionally, the Internet followed a model of information provider – information consumer, where the web sites produced information, which was merely consumed by a number of users. Nevertheless, Web 2.0 sees more user engagement. An average Internet user nowadays is not a mere consumer. Rather, she produces content through the social media and forums and engages with the existing data over the Internet. User interactions can be mined to produce more interesting information. With the huge amount of data produced, it is apparent that the data storage and manipulation should scale as well. Big data is a paradigm that attempts to handle data in a larger scale compared to the traditional means of data storage and access mechanisms such as relational databases.

Store, Process, and Access the 3 "V"

3V of Big Data
Information in the Internet keeps growing exponentially, as more and more information is made public through the wires, making a paper-less work environment. There is a definite paradigm shift on the view of historical data, from mere log files in the backup disks, to useful information for analytics and data mining, stored in data warehouses. For example, The Internet Archive Wayback Machine contains data as much as 2 PB (Peta bytes), which also keeps growing at a rate of 20 TB per month. However, Big data is not just defined by its gigantic volume. Rather, it also depends on the velocity and variety. Velocity defines how fast the data flows in and out of the system. In a weather forecast system, temperature, relative humidity, wind speed, and other measurements from the sensors flow in each second and they should be processed real-time, efficiently. This process involves a huge volume of data moving with a high velocity. The third V, variety, illustrates that the data can be of heterogeneous formats. It can be composed of a set of raw images, numbers, video, or music files. Data in multiple formats should be mashed up and processed to find the interesting information. In the weather forecasting scenario, each sensor may feed its output in different formats to the server. A Big data system differs from a relational data base system in its ability to store, access, and process such a complicated data set.

Pattern Recognition, Data Mining, Machine Learning, and Big Data
A big data solution may not be huge in volume. Obama's big data campaign had 10 TB of data initially, in various forms. Data was processed at a very high speed, as new data was made available by the volunteers and analysts frequently. Hence the campaign had high variety and velocity, with a relatively low amount of volume. This also required 66,000 simulations to be run everyday. The real-time processing of the simulation outcomes required parallel executions. Recently, a study made by Facebook on users' emotions and how the updates seen by the users affect their emotions and posts they share, created some uneasiness among the users. Nevertheless, mining user responses to optimize the business outcomes is nothing new. Regular A/B testing carried out by almost all the mainstream public web sites is an example where different layouts are tested against same content, or different titles and captions are placed, to find which of them persuade the readers to click the link, subscribe, spend more time, or even purchase an item. Patterns are recognized from millions of responses and the analytics are carried out on a big scale big data solutions. Machine learning analytics find recurring patterns and provide futuristic predictions based on the numbers. Big data solutions provide a whole new opportunity for the data mining domain in finding associations and mining.

Scalability and Simplicity

Big Data Solution over In-Memory Data Grid

Increasing volume and velocity of big data requires larger and more powerful computers to scale up. Cloud storage and data-as-a-service solutions started to replace high-end computers with the abundant resources of multiple utility computers. NoSQL solutions were developed having simplicity, horizontal scalability, and big data economy in mind. NoSQL databases have flexible data models that enable storing the variety of data objects of big data, unlike the relational data bases that come with a strict schema. NoSQL solutions can be categorized according to their design and functionality. Key–Value Stores, Column-Oriented Stores, Document-Oriented Stores, and Graph Databases can be considered major categories of the NoSQL data bases, which store data persistently in disk or in-memory.

In-Memory Data-Grids for Big Data
In-Memory Data-Grids (IMDGs) such as Infinispan, Hazelcast, Gridgain, Gigaspaces XAP, VMware vFabric Gemfire, IBM eXtremeScale, and Oracle Coherence exploit the abundant storage, processing, and memory resources in computer clusters, to provide a unified view of the nodes in the cluster. This model of shared storage, memory, and processing enables execution of larger tasks that cannot be executed effectively on a single node. While persistent storages use disk as a storage, in-memory data grids use memory as the storage, adhering to the commonly stated phrase “Memory is the new disk”. Data grids share computing resources among the instances, providing a unified view of a super computer. The abundant availability of memory enables the efficient use of computer cache, thus speeding up the processes beyond linear speedups.

While in-memory data grids have the functionality of integrating with a persistent store, in case of limited available memory to hold a very large object space, persistence of the objects stored in-memory is generally ensured through backups. Data is replicated synchronously or asynchronously based on the configurations. These ensure faster transactions compared to the cheaper disk accesses which provide slower response time. Distributed execution framework implementations handles the “Process” stage of the big data seamlessly, as they execute the algorithms in a distributed manner over the big data. MapReduce frameworks are hence implemented over the in-memory key-value stores such as Infinispan and Hazelcast using the distributed execution frameworks, following the MapReduce model of Hadoop.

Data storage vendors are focusing in positioning their products to withstand the big data storm. Oracle Big Data SQL is a recent product from Oracle, that provides single and optimized SQL query for distributed data, bringing back the simplicity of having a unified view as a traditional simple data base. As one can presume, Oracle Big Data SQL supports multiple data sources including NoSQL solutions, not limiting itself to Oracle database. Interoperability of such platforms show a favorable future for data mining and warehousing.

Conclusion
Data keeps growing, and currently what seems big may turn out to be tiny in the future. Data is measured in Exabytes (EB), when it comes to large data-oriented companies such as Facebook and Google, which was not a frequented measure a decade ago. The paradigm shift towards big data is rather economical, than technical. Though complex computations can be run over super computers with the availability of large resources, an in-memory data grid utilizes the existing resources from the utility computers, aligning with the big data economy. While more and more tools are developed for the sake of scalability and efficiency, research challenges such as security and privacy should not be taken lightly.

Fault-tolerant data replication and synchronization with Infinispan

Figure 1. Deployment

Fault-tolerance

Having multiple instances running over different nodes provide fault-tolerance, as when one node terminates, the other nodes have the backup replica of the partitions stored in the terminated node. Figure 1 shows the higher level deployment view of the solution.

Design

Two distributed cache instances exist in InfDataAccessIntegration.
    protected static Cache userReplicasMap;
    protected static Cache replicaSetsMap;

userReplicasMap is a mapping of userId -> Array of replicaSetIDs. UserID could be the logged in user name. (for now, testing with random strings).

replicaSetsMap is a mapping of replicaSetID -> replicaSet

Figure 2. Core class hierarchy

Though this could be replaced with a single cache instance with the mapping of userID -> replicaSets, I decided to go with this design, as having two cache instances will be more efficient during searches, duplicates, and push changes. Hence, I decided to go with two cache instances design.

InfDataAccessIntegration provides the API for publisher/consumer, TCIAInvoker (which extends InterfaceManager, an abstract class I created) implements the TCIA integration to invoke these methods. Figure 2 provides a core class hierarchy of the system.

 

Figure 3. Execution Flow

Execution Flow

The execution flow is depicted by Figure 3.

* User logs in -> logIn() checks whether the user has already stored replicaSets from the Infinispan distributed Cache. If so, execute them all again. This would be changed later as we do not have to execute all. Rather, we need to execute for the diffs.

* The user performs new searches, for the images, series, collections, and the other meta data. New searches will create and write the replicaSet to the distributed cache, before returning the results.

The replicaSet for the image will be as,
TCIAConstants.IMAGE_TAG + "getImage?SeriesInstanceUID=" + seriesInstanceUID

For other information (meta data), such as collections, series, etc,
TCIAConstants.META_TAG + query;

Here, query is something like, "getSeries?format=" + format +
                "&Collection=" + collection +
                "&PatientID=" + patientID +
                "&StudyInstanceUID=" + studyInstanceUID +
                "&Modality=" + modality;

When a new instance starts now, and invokes the log in action for the same user, it will execute the queries for the stored replicaSets again, and reproduce the same results.

Further updates will be posted, when they are available. :-)

Summer in Lisboa.. ^_^

After some sunny days, it has started to rain heavily again in Lisboa. Coding period began yesterday, and I have committed some interesting bits to the project. Please have a look at the status update presentation below. Wait for more updates. :)

MapReduce Implementations - Hazelcast Vs Infinispan

I was testing the MapReduce implementation of Hazlecast with the recent release of Hazelcast 3.2. Then I decided to compare the performance with the Infinispan 6.0.2 MapReduce implementation.

Infinispan outperforming Hazelcast MapReduce implementation

Infinispan outperformed Hazelcast in the sample MapReduce implementation tested on different scenarios, in a single instance, as shown by the figure. Infinispan still outperformed Hazelcast in the nodes up to 6.

Is Infinispan really faster than Hazelcast? Probably it is, as shown by scala-map-benchmarks. Probably, it is something to do with the scenarios, as discussed in Hazelcast group. However, this difference is huge, unlike the previous benchmarks. My opinion is, it is something to do with the still immature MapReduce implementation of Hazelcast, as Hazelcast proven to be quite effective for my other distributed execution tasks. 

If your use case is centred around the MapReduce implementation, I would suggest Infinispan over Hazelcast, as Hazelcast implementation is quite buggy as of 3.2. I have encountered 3 issues so far - a known issue #2105 that was reproduced during MapReduce executions and two other (probably MapReduce implementation specific) issues that I reported - #2354 (Update: This issue has been fixed for 3.2.2 and 3.3 versions of Hazlecast. Thanks Noctarius for attending to this) and #2359. Hazelcast MapReduce might turn to be more scalable and highly performing, once these issues are addressed.

It should be noted that the API of the initial roots of Hazelcast MapReduce implementation (code-named, CastMapR) was inspired heavily by that of the stable and matured MapReduce implementation of Infinispan. The Hazelcast word-count MapReduce example hence follows the same design of that from Infinispan.

I am using Hazelcast 3.2 and Infinispan 6.0.2 for my master thesis at INESC-ID Lisboa. Wait for more updates from the awesome Lisbon. ^_^


Note:
These results are part of the paper given below, which was published in 2014 December. Please cite the paper, if you used these results in your research work.
Kathiravelu, P. & L. Veiga (2014). An Adaptive Distributed Simulator for Cloud and MapReduce Algorithms and Architectures. In IEEE/ACM 7th International Conference on Utility and Cloud Computing (UCC 2014), London, UK. pp. 79 – 88. IEEE Computer Society.