[GSoC 2016] MediCurator : Near Duplicate Detection for Medical Data Warehouse Construction

This summer, at the Department of Biomedical Informatics, Emory University (Emory BMI), we have another set of intelligent students working on interesting projects. I have been mentoring Yiru Chen (Irene) from Peking University, on the project "MediCurator: Near Duplicate Detection for Medical Data Warehouse Construction" for the past couple of months. Currently we have reached the final stages of the project, as the student evaluation period starts on the 15th of August. This post is a summary of this successful GSoC, as well as a history behind the near duplicate detection efforts.

The early history of MediCurator
MediCurator was a research prototype that I initially developed based on my paper ∂u∂u Multi-Tenanted Framework: Distributed Near Duplicate Detection for Big Data (CoopIS'15) as part of my data quality research, along with my GSoC 2015 work on data integration. The early results were presented as a poster at AMIA 2016 in San Francisco.

MediCurator and Infinispan

Now we have a more complete implementation of MediCurator and a use case for medical data, thanks to the support provided by GSoC. For her implementation, Irene did some benchmarks before choosing to go with the Infinispan's latest distributed streams for the distributed execution. (You may find some interesting discussion on the Infinispan distributed streams here.)

MediCurator Usecase
MediCurator is a data quality platform for the ETL workflows in data warehouse construction. It optimizes the bandwidth usage by avoiding the duplicate downloads, and optimizes the storage by eliminating the near duplicates in the warehouse thus increasing the data quality. When data is downloaded, the source locations are tracked, and when data is updated in the source at a latter time, the subsequent download process will download only the new data.

Similarly, data is deduplicated at the data warehouse, as near duplicates could be present there since data is integrated from multiple data sources. Here the data pairs are evaluated for near duplicates in a distributed manner, with duplicate pairs stored separately, while the clean data stays in the warehouse. The duplicate detection workflow also considers the corrupted data/metadata, and synchronizes/downloads the clean data from the source.

This is useful for medical images due to the large scale of the data, often binary in nature along with textual metadata. Efficiency of MediCurator is ensured through its in-memory data grid-based architecture. MediCurator fits well with the landscape of distributed data integration and federation platforms developed at Emory BMI.

More Details on GSoC 2016

Irene developed the entire code base from scratch as an open source project. MediCurator also has a ReadTheDocs* based documentation which gives more detailed description to the project. In addition, you may learn the summary of weekly progresses at Irene's blog. MediCurator's scope remained dynamic throughout the project. MediCurator has download tracking and detecting duplicates across the datasets online and offline, in addition to the near duplicate detection. Most of the code was developed exclusively having the cancer imaging archive (TCIA) as the core data source with DICOM as the default data format, while maintaining relevant interfaces and APIs for extension to other data sources and data types.

Future Work

The summer was productive. It included both research and implementations. The GSoC time is limited to 4 months (including the community bonding period), and we are reaching a successful end to a yet another Google Summer of Code. Nevertheless, we hope to work on a research publication with combined results on MediCurator, along with the previous ∂u∂u** and SDN-based Mayan (presented at ICWS 2016) approaches in November. This will be our first publication with Irene on her findings and implementations, with further evaluations on the clusters in INESC-ID Lisboa. More updates on this later (possibly after publishing the paper ;)).

Concluding Remarks
This is my 4th time in the Google Summer of Code as a mentor, and 3rd time as the primary mentor for a project. Previously I mentored 2 successful students in 2011 and 2012 for AbiWord. I wish every student success as they reach the end of their summer of code.

* I recommend ReadTheDocs. You should give a try!
** You may find the paper on ∂u∂u interesting, if you are into data quality or distributed near duplicate detection.

SDN-Based Enhancements to QoS and Data Quality in Multi-Tenanted Data Center Clouds

I am currently in Berlin, presenting 3 of my papers at IC2E conference at TU Berlin. Given below is my doctoral symposium paper presentation, which basically summarizes the progress of my PhD during these first two years of my EMJD-DC life.

SDN-Based Enhancements to QoS and Data Quality in Multi-Tenanted Data Center Clouds from Kathiravelu Pradeeban

Abstract: Tenants assume various roles in the enterprise data center networks, requiring a differentiated Quality of Service (QoS), data quality and isolation guarantees among them. Traditionally, data storage and processing are handled in either distributed, or centralized manner. While distributed execution offers a higher horizontal scalability, it often comes with a trade-off of lack of centralized control, and hence often with a decreased accuracy and management efficiency.

Software-Defined Networking (SDN) offers a global view of the entire data center network to a logically centralized controller. Hence, it provides the best of both worlds with minimal compromises: (i) scalability of the large-scale distributed systems. (ii) unified management capabilities of the traditional centralized systems. By deploying an extended SDN controller architecture, we attempt to enhance the data quality of stored and processed data and increase the QoS of the multi-tenanted data center network clouds.

AMIA Joint Summits 2016 and MediCurator

I presented two of my papers at AMIA 2016 Joint Summits on Translational Science in San Francisco this week. One of them was a podium abstract, while the other was a poster. This was the first time I presented a poster, and I enjoyed it. You may find the slides below.

Near Duplicate Detection for Medical Imaging Data Warehouse Construction from Kathiravelu Pradeeban

Abstract: Medical data warehouses and image archives are constructed by integrating multiple private and public data sources. Finding almost identical entries is crucial for warehouse construction. Existing solutions tend to be too specific, such as Master Patient Index (MPI) for patient records. Multiple dimensions and attributes including medications, clinical, and pathological data should be considered for a complete duplicate detection and elimination. This paper describes MediCurator, a generic near duplicate detection platform for medical data warehouses. 

CURRENT CONFERENCE AND TRACK: CRI: Clinical and research data collection, curation, preservation, or sharing

I enjoyed the conference. However, San Francisco was unpleasantly dirty, specially the neighbourhood close to the conference location (Parc 55 Hilton).

Data Quality and ∂u∂u

Data Quality is a course I did at the university, as part of the 5 courses requirement for my PhD.  This course was remarkable at so many levels. 

First, I had to go to Taguspark after two years, for this course. Second, I really learned a lot. Third, I was the only student for the course. Usually individual courses are expensive. Perks of being a PhD student. Finally, we had a full paper at CoopIS 2015 (CoopIS is an "A" rated conference), as a result of this course work, which was awesome. I scored 19 out of 20 for this course. I read a lot of papers for this subject, and I spent majority of my time for this module in my second semester. Good memories. :)

Read more about our paper here. The full paper can be accessed here.

∂u∂u Multi-Tenanted Framework: Distributed Near Duplicate Detection for Big Data

Our paper "∂u∂u Multi-Tenanted Framework: Distributed Near Duplicate Detection for Big Data" Pradeeban Kathiravelu, Helena Galhardas, Luís Veiga was presented at the 23rd International Conference on COOPERATIVE INFORMATION SYSTEMS (CoopIS 2015), 28-30 October 2015, Rhodes, Greece.

Due a clash in my travel schedules, I could not attend the conference and present the paper myself. Hence, my supervisor and co-author Prof. Luís Veiga presented the paper in Greece on the 29th of October.

Abstract.
Near duplicate detection algorithms have been proposed and implemented in order to detect and eliminate duplicate entries from massive datasets. Due to the differences in data representation (such as measurement units) across different data sources, potential duplicates may not be textually identical, even though they refer to the same real-world entity. As data warehouses typically contain data coming from several heterogeneous data sources, detecting near duplicates in a data warehouse requires a considerable memory and processing power.

Traditionally, near duplicate detection algorithms are sequential and operate on a single computer. While parallel and distributed frameworks have recently been exploited in scaling the existing algorithms to operate over larger datasets, they are often focused on distributing a few chosen algorithms using frameworks such as MapReduce. A common distribution strategy and framework to parallelize the execution of the existing similarity join algorithms is still lacking.

In-Memory Data Grids (IMDG) offer a distributed storage and execution, giving the illusion of a single large computer over multiple computing nodes in a cluster. This paper presents the research, design, and implementation of ∂u∂u, a distributed near duplicate detection framework, with preliminary evaluations measuring its performance and achieved speed up. ∂u∂u leverages the distributed shared memory and execution model provided by IMDG to execute existing near duplicate detection algorithms in a parallel and multi-tenanted environment. As a unified near duplicate detection framework for big data, ∂u∂u efficiently distributes the algorithms over utility computers in research labs and private clouds and grids.

The full paper can be accessed here.

∂u∂u Multi-Tenanted Framework: Distributed Near Duplicate Detection for Big Data from Kathiravelu Pradeeban

Efficient Duplicate Detection Over Massive Data Sets

This is the fourth presentation of the Data Quality module that I am presenting today.

Indexing Techniques for Scalable Record Linkage and Deduplication

This is the third presentation of the Data Quality module that I presented today.

Data Cleaning

This is the second presentation for the Data Quality module of my PhD. Please refer to the references indicated for a more in-depth analysis, as this presentation is entirely based on them.

Introduction to Data Quality

I have a course on Data Quality for my PhD. Given below is a presentation I prepared for the course, that gives an introduction to data quality, based on a selected set of papers.