Configuring Drill with HDFS

Apache Drill can be used to query Hadoop and HDFS in a very efficient way. The steps are as below. Refer to the linked pages for each step.

1. Configure Hadoop 

There may be some errors encountered when executing Hadoop with Java 8 with the default configurations. Further, formatting the file system will fix any error from Hadoop.

  $ $HADOOP_HOME/bin/hdfs namenode -format

2. Start Hadoop NameNode daemon and DataNode daemon -
$ $HADOOP_HOME/sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/pradeeban/programs/hadoop-2.7.0/logs/hadoop-pradeeban-namenode-llovizna.out
localhost: starting datanode, logging to /home/pradeeban/programs/hadoop-2.7.0/logs/hadoop-pradeeban-datanode-llovizna.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/pradeeban/programs/hadoop-2.7.0/logs/hadoop-pradeeban-secondarynamenode-llovizna.out

Once you are done, stop the daemons with

$ $HADOOP_HOME/sbin/stop-dfs.sh

3. Browse the web interface for the name node - http://localhost:50070/

4. Configure Drill

5. Launch Drill in Embedded mode -
$ $DRILL_HOME/bin/drill-embedded 

6. Browse the web interface for Drill - http://localhost:8047/

7. Configure Hive

$ $HADOOP_HOME/bin/hadoop fs -mkdir       /tmp
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive
$ $HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive/warehouse
$ $HADOOP_HOME/bin/hadoop fs -chmod g+w   /user/hive/warehouse

Configure Hive Metastore.
Configure with MySQL.

http://sanjivblogs.blogspot.pt/2014/12/install-and-configure-hive.html  

8. Run Hive Metastore and Hive
$ $HIVE_HOME/bin/hive --service metastore &

Start HiveServer2
$HIVE_HOME/bin/hiveserver2

Alternatively, Hive server1 can be started from 
$ $HIVE_HOME/bin/hive

If you encounter URISyntaxException, have a look at this.

9.  Configure and enable Storage Plugin for Hive in Drill
{
  "type": "hive",
  "enabled": true,
  "configProps": {
    "hive.metastore.uris": "thrift://localhost:9083",
    "hive.metastore.sasl.enabled": "false"
  }
}

10. Query Hive from Drill.
SELECT firstname,lastname FROM hive.`customers` 

11. This discusses configuring Drill with Mongo.

Completion of EMDC - II

KTH President at the Welcome Reception

Finally, we had the KTH graduation on the 27th of May. It was pleasant to meet many of my Erasmus Mundus friends in Stockholm after a long time, in a familiar place. Remember, my post Completion of EMDC" on the IST graduation on the 23rd of May? I spent 75% of my masters in IST. But still it was KTH graduation that really made me feel the completion. Majority of us from EMDC - IST were back in Stockholm after 1.5 years for the graduation. The graduation ceremony took place in Stockholm city hall, where we also had the inauguration ceremony in 2013 August, when we just started our semester in KTH. So much nostalgic.

I ordered the two individual photos taken for 165*2 SEK (35,44 Euro), and transferred the money through my bank from Portugal to Sweden. It cost me 41,60 Euro extra for the foreign transaction fees. It usually charged me this when I transferred money to Sri Lanka. I did not know that they also charge to transfer to Sweden! It is always free to transfer money inside Portugal (also probably inside all the euro-zone countries). I was informed by my bank that it was the charge from the Swedish bank - not my bank. My Portuguese bank does not charge for online money transaction inside Europe. Whatever that is, it seems bank money transfer is not really a good idea.

Now waiting for the official photos to arrive by email. When I came to Portugal in 2012, I just wanted to finish my masters successfully with pretty good results. I did not think that I will stay in Portugal beyond that, though I really had plans to do EMJD-DC, my current PhD program, from the very beginning. But my interest was to do it in KTH. Eventually, my shift changed and I started to like IST and INESC-ID Lisboa. That is how I ended up staying much longer in Lisbon than I initially thought.

In the photo, Prof. Peter Gudmundson, President of KTH, during the welcome reception on the 30th of August, 2013. Same location - same people - nice to be there listening to him again speaking in Swedish and English. Next to Sri Lanka and Portugal, Sweden was my third home for a short period. I often feel guilty realizing that I did not travel much during my 4.5 months stay in Sweden, except Stockholm and Malmo, in 2013. This visit [26th - 29th, May 2015] was a quick recap of my entire stay in a very short time. I will be back for my third visit soon, I believe.

As of now, I don't really have solid plans on my next stop. I will know it when the time comes closer.

Understanding and Protecting Ourselves from Buggy Device Drivers

Given below is the 4th presentation for the NSOM module, which I presented today.

Completion of EMDC

Instituto Superior Técnico (which is considered the top tech university of the country), Universidade de Lisboa being my core/base university of my double masters (EMDC) and PhD (EMJD-DC), I am spending more time here than all the other universities that I have stayed with. Yesterday, we had the graduation day at IST, which was a small and simple event in the main campus auditorium Salão Nobre. This is some day that I was waiting for, when I came to Portugal in 2012 August. However, yesterday was a regular Saturday of my PhD life, having some break. Interesting to see how time has changed.

Scheduling for Parallel and Multi-Core Systems

Third presentation for the NSOM module, based on 3 of the most cited papers in the domain. :D