Practical Hadoop Migration : How to Integrate Your RDBMS with the Hadoop Ecosystem and Re-Architect Relational Applications to NoSQL

Contents at a Glance Contents Foreword About the Author About the Technical Reviewer Acknowledgments Introduction Chapter 1: RDBMS Meets Hadoop: Integrating, Re-Architecting, and Transitioning Conceptual Differences Between Relational and HDFS NoSQL Databases Relational Design and Hadoop in Conjunction: Advantages and Challenges Type of Data Data Volume Business Need Deciding to Integrate, Re-Architect, or Transition Type of Data Type of Application Business Objectives How to Integrate, Re-Architect, or Transition Integration Re-Architecting Using Lambda Architecture Batch Layer Serving Layer Speed Layer Transition to Hadoop/NoSQL Type of Data Data Volume Data Distribution Migrating the Data Summary Part I: Relational Database Management Systems: A Review of Design Principles, Models and Best Practices Chapter 2: Understanding RDBMS Design Principles Overview of Design Methodologies Top-down Bottom-up SSADM Exploring Design Methodologies Top-down Bottom-up SSADM Feasibility Study Investigation of the Current Environment Business System Options Requirements Specification Technical System Options Logical Design Physical Design Pros and Cons of SSADM Components of Database Design Normal Forms First Normal Form Second Normal Form Third Normal Form Keys in Relational Design Optionality and Cardinality Supertypes and Subtypes Summary Chapter 3: Using SSADM for Relational Design Feasibility Study Project Initiation Plan Requirements and User Catalogue Requirements Catalogue User Catalogue Current Environment Description Current System Description Current Physical Data Flow Model Current Logical Data Model Proposed Environment Description Business Activity Model Data Specification Function Specification Problem Definition Feasibility Study Report Requirements Analysis Investigation of Current Environment Current Data Flow Model Current Logical Data Model Requirements Catalogue User Catalogue Logical Data Store/Entity Cross-Reference Logical View of Current Services and System Scope Business System Options Requirements Specification Data Flow Model Logical Data Model Function Definitions GetPlayerInjuryInfo GetPlayerChronicCondInfo GetPlayerContractDetails GetPlayerScheduleInfo CalculateLossOfPlayPremium EvalLossOfPlayClaim Effect Correspondence Diagrams (ECDs) Entity Life Histories (ELHs) Logical System Specification Technical Systems Options Logical Design Update Processing Model Enquiry Processing Model Data Catalogue Physical Design Logical to Physical Transformation Space Estimation Growth Provisioning Optimizing Physical Design Summary Chapter 4: RDBMS Design and Implementation Tools Database Design Tools CASE tools Building and Using Design Layers Categorizing Design Using Subject Areas Display Level of a Model Forward and Reverse Engineering Creating Reusable Components Propagating a Change Easily and Quickly Diagramming Tools Administration and Monitoring Applications Database Administration or Management Applications Monitoring Applications Summary Part II: Hadoop: A Review of the Hadoop Ecosystem, NoSQL Design Principles and Best Practices Chapter 5: The Hadoop Ecosystem Query Tools Spark SQL Presto Analytic Tools Apache Kylin Kylin Architecture In-Memory Processing Tools Flink Flink Architecture Search and Messaging Tools Summary Chapter 6: Re-Architecting for NoSQL: Design Principles, Models and Best Practices Design Principles for Re-Architecting Relational Applications to NoSQL Environments Selecting an Appropriate NoSQL Database Key-Value Stores Document Databases Columnar Databases Graph Databases Domain Description Nodes Labels Relationships Creating Attributes Concurrency and Security for NoSQL Concurrency Security Designing the Transition Model Denormalization of Relational (OLTP) Data Denormalization of Relational (OLAP) Data Implementing the Final Model Columnar Database as a NoSQL Target Document Database as a NoSQL Target Best Practices for NoSQL Re-Architecture Summary Part III: Integrating Relational Database Management Systems with the Hadoop Distributed File System Chapter 7: Data Lake Integration Design Principles Data Lake vs. Data Warehouse Data Warehouse Data Lake Concept of a Data Lake Data Reservoirs Data Reservoir Repositories Data Reservoir Services Governance Engine Authentication Authorization PII Masking Encryption Encryption at Rest Encryption in Transit Data Quality Services Data Cleansing Matching Data Profiling Factors for a Successful Implementation Exploratory Lakes Data Validation for Exploratory Analysis Exploratory Analysis Through Visualizations Correlation Clustering Hierarchical Clustering K-means Clustering Factors for a Successful Implementation Analytical Lakes Using Data for Analytical Models Model Building Steps Using Data as a Staging Area for EDW or Data Mart Real-Time Processing and Analytics Event Stream Processing Complex Event Processing Factors for a Successful Implementation Summary Chapter 8: Implementing SQOOP and Flume-based Data Transfers Deciding on an ETL Tool Sqoop vs. Flume Processing Streaming Data Spark and Spark Streaming Storm Samza Using SQOOP for Data Transfer Using Flume for Data Transfer Flume Architecture Understanding and Using Flume Components Source Sink Implementing Log Consolidation Using Flume Summary Part IV: Transitioning from Relational to NoSQL Design Models Chapter 9: Lambda Architecture for Real-time Hadoop Applications Defining and Using the Lambda Layers Batch Layer Designing Your Master Data Fact-Based Model Applying a Fact-based Model to Relational Applications Building Batch Views Designing Batch Views for Your Fact-based Model Implementing Batch Views Serving Layer ElephantDB Splout SQL Speed Layer Pros and Cons of Using Lambda Benefits of Lambda Issues with Lambda The Kappa Architecture Future Architectures1 A Bit of History Butterfly Architecture Storage for Butterfly Architecture Ampool Example Use Case: Ad Tech Data Pipeline The Data User Profiles Advertisements Content Metadata Ad Serving Logs Computations Ingestion and Streaming Analytics Batch Model Building Interactive and Ad Hoc SQL Queries Summary Chapter 10: Implementing and Optimizing the Transition Hardware Configuration Cluster Configuration Operating System Configuration Hadoop Configuration HDFS Configuration JVM/YARN/MapReduce Configuration Generic JVM Guidelines Generic YARN/MapReduce Guidelines Optimizing MapReduce Applications Optimizing YARN Execution Choosing an Optimal File Format Row-based Formats Text Files Sequence Files Avro Column-based Formats RCFile ORCFile Parquet Indexing Considerations for Performance Compact indexes Bitmap Indexes Choosing a NoSQL Solution and Optimizing Your Data Model Summary Part V: Case Study for Designing and Implementing a Hadoop-based Solution Chapter 11: Case Study: Implementing Lambda Architecture The Business Problem and Solution Solution Design Hardware Software Database Design Considering a Fact-based Model Data Conditions for Fraudulence Batch Layer Design Implementing Batch Layer Implementing the Serving Layer Implementing the Speed Layer Storage Structures (for Master Data and Views) Other Performance Considerations Reference Architectures Changes to Implementation for Latest Architectures Re-Implementation Using Kappa Architecture Changes for Fast Data Architecture Changes for Butterfly Architecture Summary Index Chapter 1: RDBMS Meets Hadoop: Integrating, Re-Architecting, and Transitioning -- Part I: Relational Database Management Systems: A Review of Design Principles, Models, and Best Practices -- Chapter 2: Understanding RDBMS Design Principles -- Chapter 3: Using SSADM for Relational Design -- Chapter 4: RDBMS Design and Implementation Tools.-Part II: Hadoop: A Review of the Hadoop Ecosystem, NoSQL Design Principles and Best Practices -- Chapter 5: The Hadoop Ecosystem -- Chapter 6: Re-Architecting for NoSQL Design Principles, Models, and Best Practices.-Part III: Integrating Relational Database Management Systems with the Hadoop Distributed File System -- Chapter 7: Data Lake Integration Design Principles -- Chapter 8: Implementing SQOOP and Flume-based Data Transfers.-Part IV: Transitioning from Relational to NoSQL Design Models -- Chapter 9: Lambda Architecture for Real-time Hadoop Applications -- Chapter 10: Implementing and Optimizing the Transition -- Part V: Case Study for Designing and Implementing a Hadoop-based Solution -- Chapter 11: Case Study: Implementing Lambda Architecture.;Re-architect relational applications to NoSQL, integrate relational database management systems with the Hadoop ecosystem, and transform and migrate relational data to and from Hadoop components. This book covers the best-practice design approaches to re-architecting your relational applications and transforming your relational data to optimize concurrency, security, denormalization, and performance. Winner of IBM's 2012 Gerstner Award for his implementation of big data and data warehouse initiatives and author of Practical Hadoop Security, author Bhushan Lakhe walks you through the entire transition process. First, he lays out the criteria for deciding what blend of re-architecting, migration, and integration between RDBMS and HDFS best meets your transition objectives. Then he demonstrates how to design your transition model. Lakhe proceeds to cover the selection criteria for ETL tools, the implementation steps for migration with SQOOP- and Flume-based data transfers, and transition optimization techniques for tuning partitions, scheduling aggregations, and redesigning ETL. Finally, he assesses the pros and cons of data lakes and Lambda architecture as integrative solutions and illustrates their implementation with real-world case studies. Hadoop/NoSQL solutions do not offer by default certain relational technology features such as role-based access control, locking for concurrent updates, and various tools for measuring and enhancing performance. Practical Hadoop Migration shows how to use open-source tools to emulate such relational functionalities in Hadoop ecosystem components. What You'll Learn The requirements and design methodologies of relational data and NoSQL models How to decide whether you should migrate your relational applications to big data technologies or integrate them How to transition your relational applications to Hadoop/NoSQL platforms in terms of logical design and physical implementation RDBMS-to-HDFS integration, data transformation, and optimization techniques The situations in which Lambda architecture and data lake solutions should be considered How to select and implement Hadoop-based components and applications to speed transition, optimize integrated performance, and emulate relational functionalities Who This Book Is For The primary readership for Practical Hadoop Migration is database developers, database administrators, enterprise architects, Hadoop/NoSQL developers, and IT leaders. Its secondary readership is project and program managers and advanced students of database and management information systems. Re-architect relational applications to NoSQL, integrate relational database management systems with the Hadoop ecosystem, and transform and migrate relational data to and from Hadoop components. This book covers the best-practice design approaches to re-architecting your relational applications and transforming your relational data to optimize concurrency, security, denormalization, and performance. Winner of IBM’s 2012 Gerstner Award for his implementation of big data and data warehouse initiatives and author of Practical Hadoop Security , author Bhushan Lakhe walks you through the entire transition process. First, he lays out the criteria for deciding what blend of re-architecting, migration, and integration between RDBMS and HDFS best meets your transition objectives. Then he demonstrates how to design your transition model. Lakhe proceeds to cover the selection criteria for ETL tools, the implementation steps for migration with SQOOP- and Flume-based data transfers, and transition optimization techniques for tuning partitions, scheduling aggregations, and redesigning ETL. Finally, he assesses the pros and cons of data lakes and Lambda architecture as integrative solutions and illustrates their implementation with real-world case studies. Hadoop/NoSQL solutions do not offer by default certain relational technology features such as role-based access control, locking for concurrent updates, and various tools for measuring and enhancing performance. Practical Hadoop Migration shows how to use open-source tools to emulate such relational functionalities in Hadoop ecosystem components. What You'll Learn Decide whether you should migrate your relational applications to big data technologies or integrate them Transition your relational applications to Hadoop/NoSQL platforms in terms of logical design and physical implementation Discover RDBMS-to-HDFS integration, data transformation, and optimization techniques Consider when to use Lambda architecture and data lake solutions Select and implement Hadoop-based components and applications to speed transition, optimize integrated performance, and emulate relational functionalities Who This Book Is For Database developers, database administrators, enterprise architects, Hadoop/NoSQL developers, and IT leaders. Its secondary readership is project and program managers and advanced students of database and management information systems.