Hadoop Ecosystem at a Glance

Contributed by Khurshidali Shaikh on 18 Mar 2013

• 
• Tweet
• Get in Touch

Hadoop as we know is a massive scalable distributed framework for processing large amount of data(several Petabytes) across  a cluster (1000s) of commodity computers.

The Hadoop ecosystem has grown over the last few years and there is a lot of jargon in terms of tools and frameworks. Also, there are many organizations who are investing & innovating in Hadoop to make it better and easier. Based on my reading about Hadoop over the last few weeks, here is a mind map that gives a quick glance at the Hadoop ecosystem. This should be useful to get a high-level picture of the ecosystem.

Core

• HDFS: Hadoop Distribute File System (HDFS) is a distributed file system designed to run on a commodity cluster of machines. HDFS is highly fault tolerant and is useful for processing large data sets. A Map Reduce job, described below, typically processes data stored in HDFS. Files in HDFS are split into blocks, typically 64MB or 128MB, and stored across nodes in the cluster. Each block of data is also replicated across more nodes, generally three, in the cluster to avoid data loss in case of a node failure.
• MapReduce: MapReduce is a software framework for processing large data sets, petabyte scale, on a cluster of commodity hardware.  When MapReduce jobs are run, Hadoop splits the input and locates the nodes on the cluster. The actual jobs are then run at or close to the node where the data is residing so that the data is as close to the computation node as possible. This avoids transfer of huge amount of data across the network so that the network does not become a bottleneck or get flooded.

Distributions

• Apache: Purely Open Source distribution of Hadoop maintained by the community at Apache Software Foundation.
• Cloudera: Cloudera’s distribution of Hadoop that is built on top of Apache Hadoop. The distribution includes capabilities such as management, security, high availability and integration with a wide variety of hardware and software solutions. Cloudera is the leading distributor of Hadoop.
• Horton Works: This also builds on the Open Source Apache Hadoop with claims to enterprise readiness. It also claims to be the only distribution that is available for Windows servers.
• MapR: Hadoop distribution with some unique features, most notably the ability to mount the Hadoop cluster over NFS.
• Intel: Intel’s Open Source version of Hadoop distribution.
• Greenplum:  Greenplum’s distribution is called Pivotal HD. One of the highlights of this distribution is a SQL-based database engine on Hadoop that allows querying of data in Hadoop using SQL.
• Amazon EMR : Amazon’s hosted version of MapReduce is called Elastic Map Reduce. This is part of the Amazon Web Services (AWS). EMR allows a Hadoop cluster to be deployed and MapReduce jobs to be run in the AWS cloud with just a few clicks.

Related Projects

• Avro : Avro is a data serialization framework that is useful in Hadoop and other systems. The framework allows one to define schema which is language independent so that data can be interchanged between different languages.e.g  A Hadoop client in a different language can use Avro as the data serialization framework to communicate with the Hadoop server which is in Java.
• Pig : Framework for analyzing large data sets using a high level language called Pig Latin. Scripts written in Pig Latin are compiled by the framework in MapReduce jobs which are run on the Hadoop cluster. Pig eases development of MapReduce jobs. A set of MapReduce jobs which may take hundreds of lines of code can be written with just few lines of Pig Latin scripts. At Yahoo > 60% of Hadoop usage is on Pig.
• Hive : Hive is a data warehouse framework that stores querying of large data sets stored in Hadoop. To do this, Hive provides a high-level SQL like language called HiveQL. Traditional MapReduce programs can be plugged into HiveQL where its more efficient to have these instead of HiveQL.
• HBase : HBase is a distributed scalable data store based on Hadoop. HBase is a distributed, versioned, column-oriented database modeled after Google’s BigTable.
• Mahout : Mahout is a scalable Machine learning library. Mahout utilizes Hadoop to achieve massive scalability.
• YARN : YARN is the next generation of MapReduce a.k.a MapReduce 2. The MapReduce framework was overhauled using YARN to overcome the scalability bottlenecks in earlier version of MapReduce when it was run over a very large cluster(thousands of nodes).
• Ozzie : In a real life scenario, a MapReduce deployment typically involves running a sequence of MapReduce and other pre and post processing jobs at scheduled times or based on data availability. Ozzie is a workflow scheduler system that eases the creation and management of these workflows. A workflow is defined using XML in which one can do HDFS operations, run MapReduce jobs, Pig scripts, Streaming jobs, branching, chaining, etc.
• Flume : A distributed, reliable and available service for collecting, aggregating and moving log data to HDFS. This is typically useful in systems where log data needs to be moved to HDFS periodically for processing.
• Sqoop : Sqoop is designed for transferring data between Hadoop and relational databases.
• Cascading : Application framework for building applications using Hadoop.

Related Technologies

Below is a list of related BigData technologies but which follow an architecture different from Hadoop.

• Twitter Storm : As opposed to Hadoop which is a batch processing system, Storm is a distributed real-time processing system developed by Twitter. Storm is fast, scalable and easy to use.
• HPCC : High Performance Computing Cluster (HPCC) is an MPP (Massive Parallel Processing) computing platform for solving Big Data problems. HPCC follows an architecture different from Hadoop. It boasts a few differences and advantages over Hadoop.
• Dremel : A scalable interactive ad-hoc query system for analysis of read-only nested data by Google. Google’s BigQuery service is reportedly based on Dremel.

This is not an exhaustive list and there may be many other projects, tools, and organizations related to Hadoop. I have tried to touch the most popular ones here.

In future posts, I wish to talk in detail about some of the less-talked-about projects in the above list but which can be really useful in day-to-day running and managing of a Hadoop cluster.

 

Visit us at Neevtech.com to know more about our offerings.

Tags: Bigdata, cloudera, hadoop, HDFS, mapr, mapreduce, YARN