Hadoop essentials : delve into the key concepts of Hadoop and get a thorough understanding of the Hadoop ecosystem

Cover 1 Copyright 3 Credits 4 About the Author 5 Acknowledgments 6 About the Reviewers 7 www.PacktPub.com 9 Table of Contents 10 Preface 16 Chapter 1: Introduction to Big Data and Hadoop 22 V's of big data 23 Volume 23 Velocity 24 Variety 24 Understanding big data 24 NoSQL 25 Types of NoSQL databases 26 Analytical database 27 Who is creating the big data? 27 Big data use cases 27 Big data use case patterns 29 Big data as a storage pattern 29 Big data as a data transformation pattern 30 Big data for a data analysis pattern 31 Big data for data in a real-time pattern 32 Big data for a low latency caching pattern 33 Hadoop 34 Hadoop history 35 Description 35 Advantages of Hadoop 36 Uses of Hadoop 37 Hadoop ecosystem 37 Apache Hadoop 38 Hadoop distributions 39 Pillars of Hadoop—HDFS, MapReduce, and YARN 40 Data access components – Hive and Pig 40 Data storage component – HBase 40 Data ingestion in Hadoop– Sqoop and Flume 41 Streaming and real-time analysis – Storm and Spark 41 Summary 41 Chapter 2: Hadoop Ecosystem 42 Traditional systems 42 Database trend 43 Hadoop use cases 44 Hadoop basic data flow 45 Hadoop integration 46 The Hadoop ecosystem 46 Distributed filesystem 47 HDFS 47 Distributed programming 48 NoSQL databases 49 Apache HBase 49 Data ingestion 49 Service Programming 50 Apache YARN 50 Apache Zookeeper 51 Scheduling 51 Data analytics and machine learning 51 System management 52 Apache Ambari 52 Summary 52 Chapter 3: Pillars of Hadoop – HDFS, MapReduce, and YARN 54 HDFS 55 Features of HDFS 55 HDFS Architecture 55 NameNode 56 DataNode 57 Checkpoint NameNode or Secondary NameNode 58 BackupNode 58 Data storage in HDFS 58 Read pipeline 59 Write pipeline 60 Rack awareness 61 Advantages of rack awareness in HDFS 61 HDFS Federation 62 Limitations of HDFS 1.0 62 The benefit of HDFS Federation 63 HDFS ports 63 HDFS commands 65 MapReduce 67 MapReduce architecture 67 JobTracker 67 TaskTracker 68 Serialization data types 68 Writable interface 68 WritableComparable interface 68 MapReduce example 69 The MapReduce process 70 Mapper 71 Shuffle and sorting 72 Reducer 72 Speculative execution 72 FileFormats 73 InputFormats 73 RecordReader 74 OutputFormats 74 RecordWriter 75 Writing a MapReduce program 75 Mapper code 76 Reducer code 76 Driver code 77 Auxiliary steps 80 Combiner 81 Partitioner 81 YARN 82 YARN Architecture 83 ResourceManager 84 NodeManager 84 ApplicationMaster 85 Applications powered by YARN 85 Summary 85 Chapter 4: Data Access Components – Hive and Pig 88 Need of a data processing tool on Hadoop 88 Pig 89 Pig data types 89 Pig architecture 90 The logical plan 90 The physical plan 91 The MapReduce plan 91 Pig modes 91 Grunt shell 92 Input data 92 Loading data 93 Dump 94 Store 94 Filter 95 Group By 95 Limit 96 Aggregation 97 Cogroup 97 DESCRIBE 99 EXPLAIN 99 ILLUSTRATE 103 Hive 104 Hive architecture 104 Metastore 105 Query compiler 106 Execution engine 106 Data types and schemas 106 Installing Hive 107 Starting Hive Shell 108 HiveQL 108 DDL (Data Definition Language) operations 108 DML (Data Manipulation Language) operations 111 SQL operation 112 Built-in functions 114 Custom UDF (User Defined Functions) 115 Managing tables (external versus managed) 115 SerDe 116 Partitioning 118 Bucketing 119 Summary 119 Chapter 5: Storage Component - HBase 122 An Overview of HBase 122 Advantages of HBase 123 Architecture of HBase 124 MasterServer 125 RegionServer 125 WAL 126 BlockCache 126 Regions 127 MemStore 127 Zookeeper 128 HBase data model 128 Logical components of data model 128 ACID properties 130 CAP theorem 130 Schema design 130 Write pipeline 131 Read pipeline 132 Compaction 132 Compaction policy 132 Minor compaction 133 Major compaction 133 Splitting 134 Pre-Splitting 134 Auto Splitting 135 Forced Splitting 135 Commands 135 help 135 Create 135 List 136 Put 136 Scan 136 Get 136 Disable 137 Drop 137 HBase Hive integration 137 Performance tuning 138 Compression 138 Filters 139 Counters 141 HBase co-processors 142 Summary 143 Chapter 6: Data Ingestion in Hadoop – Sqoop and Flume 144 Data sources 144 Challenges in data ingestion 145 Sqoop 146 Connectors and drivers 146 Sqoop 1 architecture 146 Limitation of Sqoop 1 147 Sqoop 2 architecture 148 Imports 149 Exports 152 Apache Flume 153 Reliability 154 Flume architecture 155 Multitier topology 155 Flume Master 156 Flume Nodes 156 Components in Agent 157 Channels 159 Examples of configuring Flume 162 Single agent example 162 Multiple flow in an agent 163 Configuring a multi-agent setup 163 Summary 165 Chapter 7: Streaming and Real-time Analysis – Storm and Spark 166 An introduction to Storm 166 Features of Storm 167 Physical architecture of Storm 167 Data architecture of Storm 168 Storm topology 169 Storm on YARN 170 Topology configuration example 170 Spouts 170 Bolts 171 Topology 173 An introduction to Spark 173 Features of Spark 174 Spark framework 174 Spark SQL 175 GraphX 175 MLib 175 Spark streaming 175 Spark architecture 176 Directed Acyclic Graph engine 176 Resilient Distributed Dataset 176 Physical architecture 178 Operations in Spark 178 Transformations 178 Actions 180 Spark example 181 Summary 182 Index 184 Delve into the key concepts of Hadoop and get a thorough understanding of the Hadoop ecosystem In Detail This book jumps into the world of Hadoop ecosystem components and its tools in a simplified manner, and provides you with the skills to utilize them effectively for faster and effective development of Hadoop projects. Starting with the concepts of Hadoop YARN, MapReduce, HDFS, and other Hadoop ecosystem components, you will soon learn many exciting topics such as MapReduce patterns, data management, and real-time data analysis using Hadoop. You will also get acquainted with many Hadoop ecosystem components tools such as Hive, HBase, Pig, Sqoop, Flume, Storm, and Spark. By the end of the book, you will be confident to begin working with Hadoop straightaway and implement the knowledge gained in all your real-world scenarios. What You Will Learn Get introduced to Hadoop, big data, and the pillars of Hadoop such as HDFS, MapReduce, and YARN Understand different use cases of Hadoop along with big data analytics and real-time analysis in Hadoop Explore the Hadoop ecosystem tools and effectively use them for faster development and maintenance of a Hadoop project Demonstrate YARN's capacity for database processing Work with Hive, HBase, and Pig with Hadoop to easily figure out your big data problems Gain insights into widely used tools such as Sqoop, Flume, Storm, and Spark using practical examples About This BookGet to grips with different Hadoop ecosystem tools that can help you achieve scalability, performance, maintainability, and efficiency in your projectsUnderstand the different paradigms of Hadoop and get the most out of it to engage the power of your dataThis is a fast-paced reference guide covering the key components and functionalities of HadoopWho This Book Is ForIf you are a system or application developer interested in learning how to solve practical problems using the Hadoop framework, then this book is ideal for you. This book is also meant for Hadoop professionals who want to find solutions to the different challenges they come across in their Hadoop projects.

About This Book

• Visualize and create advanced level reports and dashboards using Salesforce.com
• Take advantage of creating reports and dashboards in the Salesforce mobile app, updated with Spring '15 release
• A concise and informative guide to solve all your reporting woes

Who This Book Is For

This book is intended for all Salesforce users—administrators, managers, business analysts, or report writers who are new to creating reports or dashboards within Salesforce. Basic knowledge of the Salesforce platform is required.