Data Warehousing in the Age of Big Data (The Morgan Kaufmann Series on Business Intelligence)

__Data Warehousing in the Age of the Big Data__ will help you and your organization make the most of unstructured data with your existing data warehouse. As Big Data continues to revolutionize how we use data, it doesn't have to create more confusion. Expert author Krish Krishnan helps you make sense of how Big Data fits into the world of data warehousing in clear and concise detail. The book is presented in three distinct parts. Part 1 discusses Big Data, its technologies and use cases from early adopters. Part 2 addresses data warehousing, its shortcomings, and new architecture options, workloads, and integration techniques for Big Data and the data warehouse. Part 3 deals with data governance, data visualization, information life-cycle management, data scientists, and implementing a Big Data-ready data warehouse. Extensive appendixes include case studies from vendor implementations and a special segment on how we can build a healthcare information factory. Ultimately, this book will help you navigate through the complex layers of Big Data and data warehousing while providing you information on how to effectively think about using all these technologies and the architectures to design the next-generation data warehouse. * Learn how to leverage Big Data by effectively integrating it into your data warehouse. * Includes real-world examples and use cases that clearly demonstrate Hadoop, NoSQL, HBASE, Hive, and other Big Data technologies * Understand how to optimize and tune your current data warehouse infrastructure and integrate newer infrastructure matching data processing workloads and requirements Front Cover......Page 1 Data Warehousing in the Age of Big Data......Page 4 Copyright Page......Page 5 Contents......Page 8 Acknowledgments......Page 16 About the Author......Page 18 Introduction......Page 20 Part 1: Big Data......Page 22 Part 3: Building the Big Data – Data Warehouse......Page 23 Companion website......Page 24 1 BIG DATA......Page 26 Big Data......Page 28 Why Big Data and why now?......Page 30 Social Media posts......Page 31 Survey data analysis......Page 32 Survey data......Page 33 Integration and analysis......Page 36 Additional data types......Page 38 Further reading......Page 39 Data explosion......Page 40 Machine data......Page 42 Emails......Page 43 Geographic information systems and geo-spatial data......Page 44 Example: Funshots, Inc.......Page 46 Data velocity......Page 48 Social media......Page 49 Data variety......Page 50 Summary......Page 52 Data processing revisited......Page 54 Data processing techniques......Page 55 Storage......Page 56 Processing......Page 57 Shared-everything and shared-nothing architectures......Page 58 Shared-nothing architecture......Page 59 OLTP versus data warehousing......Page 60 Big Data processing......Page 61 Infrastructure explained......Page 64 Telco Big Data study......Page 65 Data processing......Page 67 Introduction......Page 70 Distributed data processing......Page 71 Big Data processing requirements......Page 74 Technologies for Big Data processing......Page 75 Google file system......Page 76 Hadoop......Page 78 HDFS......Page 79 DataNodes......Page 80 HDFS client......Page 81 Heartbeats......Page 82 File system snapshots......Page 83 JobTracker and TaskTracker......Page 84 MapReduce......Page 85 MapReduce programming model......Page 86 MapReduce program design......Page 87 MapReduce job processing and management......Page 88 MapReduce v2 (YARN)......Page 89 YARN scalability......Page 91 Comparison between MapReduce v1 and v2......Page 92 SQL/MapReduce......Page 93 Zookeeper features......Page 94 Failure and recovery......Page 96 Programming with pig latin......Page 97 Common pig command......Page 98 HBase......Page 99 HBase architecture......Page 100 HBase components......Page 101 Write-ahead log......Page 102 Hive......Page 103 Hive architecture......Page 104 Execution: how does hive process queries?......Page 105 Chukwa......Page 107 HCatalog......Page 108 Sqoop1......Page 109 Hadoop summary......Page 110 NoSQL......Page 111 CAP theorem......Page 112 Column family store: Cassandra......Page 113 Data model......Page 114 Data sorting......Page 116 Built-in consistency repair features......Page 117 Cassandra ring architecture......Page 118 Data partitioning......Page 119 Gossip protocol: node management......Page 120 Document database: Riak......Page 121 Textual ETL processing......Page 122 Further reading......Page 124 Introduction......Page 126 Producing electricity from wind......Page 127 Tackling Big Data challenges......Page 129 Surveillance and security: TerraEchos......Page 130 The benefit......Page 131 Correlating sensor data delivers a zero false-positive rate......Page 132 Challenges......Page 133 Solution: getting ready for Big Data analytics......Page 134 Why aster?......Page 135 Overview......Page 136 Making better use of the data resource......Page 137 Solution components......Page 138 Merging human knowledge and technology......Page 139 Solution spotlight......Page 140 Solution......Page 141 Facilitates innovation......Page 142 Overview......Page 143 Enabling a better cross-sell and upsell opportunity......Page 146 Example......Page 147 Summary......Page 148 2 THE DATA WAREHOUSING......Page 150 Introduction......Page 152 Traditional data warehousing, or data warehousing 1.0......Page 153 Data architecture......Page 154 Infrastructure......Page 155 Pitfalls of data warehousing......Page 156 Performance......Page 157 Scalability......Page 160 Architecture approaches to building a data warehouse......Page 162 Pros and cons of datamart BUS architecture approach......Page 164 Data warehouse 2.0......Page 165 Overview of DSS 2.0......Page 166 Further reading......Page 169 Introduction......Page 172 Enterprise data warehouse platform......Page 173 Data warehouse......Page 174 Issues with the data warehouse......Page 175 Replatforming......Page 177 Platform engineering......Page 178 Data engineering......Page 179 Modernizing the data warehouse......Page 180 Current-state analysis......Page 182 Business benefits of modernization......Page 183 Scorecard......Page 184 Program roadmap......Page 185 Summary......Page 187 Current state......Page 188 Defining workloads......Page 189 Understanding workloads......Page 190 Datamarts......Page 192 Analytical databases......Page 193 Data warehouse processing overheads......Page 194 Wide/Wide......Page 195 Narrow/Wide......Page 196 ETL and CDC workloads......Page 197 Measurement......Page 199 Current system design limitations......Page 200 Big Data workloads......Page 201 Technology choices......Page 202 Summary......Page 203 Data warehouse challenges revisited......Page 204 Data volumes......Page 205 Data transport......Page 206 Data warehouse appliance......Page 207 Appliance architecture......Page 208 Data distribution in the appliance......Page 209 Key best practices for deploying a data warehouse appliance......Page 211 Cloud computing......Page 212 Platform as a service......Page 213 Cloud infrastructure......Page 214 Issues facing cloud computing for data warehouse......Page 215 What is data virtualization?......Page 216 Implementing a data virtualization program......Page 218 In-memory technologies......Page 219 Further reading......Page 220 3 BUILDING THE BIG DATA – DATA WAREHOUSE......Page 222 Introduction......Page 224 Data layer......Page 225 Algorithms......Page 227 Technology layer......Page 228 Data classification......Page 229 Workload......Page 231 Physical component integration and architecture......Page 232 Data volumes......Page 233 External data integration......Page 234 Hadoop & RDBMS......Page 236 Big Data appliances......Page 237 Data virtualization......Page 239 Semantic framework......Page 240 Clustering......Page 241 Summary......Page 242 Metadata......Page 244 Process design–level metadata......Page 246 Core business metadata......Page 247 Master data management......Page 248 Processing data in the data warehouse......Page 250 Processing complexity of Big Data......Page 253 Processing Big Data......Page 254 Analysis stage......Page 255 Metadata, master data, and semantic linkage......Page 256 Types of probabilistic links......Page 258 Machine learning......Page 260 Summary......Page 265 Information life-cycle management......Page 266 Goals......Page 267 Executive governance board......Page 268 Business teams......Page 269 Data quality......Page 270 Metadata......Page 271 Information life-cycle management for Big Data......Page 272 Data governance......Page 273 Processing......Page 274 Summary......Page 275 Big Data analytics......Page 276 Data discovery......Page 278 Visualization......Page 279 Summary......Page 280 Customer-centric business transformation......Page 282 Outcomes......Page 285 Hadoop and MySQL drives innovation......Page 286 Benefits......Page 288 Empowering decision making......Page 289 Summary......Page 290 Case study 1: Transforming marketing landscape......Page 292 Case study 2: Streamlining healthcare connectivity with Big Data......Page 296 Case study 3: Improving healthcare quality and costs using Big Data......Page 299 Case study 4: Improving customer support......Page 302 Case study 5: Driving customer-centric transformations......Page 306 Case study 6: Quantifying risk and compliance......Page 308 Case study 7: Delivering a 360° view of customers......Page 309 Executive summary......Page 314 The healthcare information factory......Page 315 A visionary architecture......Page 316 A common patient identifier......Page 317 Integrating data......Page 318 ETL and the collective common data warehouse......Page 319 Common elements of a data warehouse......Page 322 DSS/business intelligence processing......Page 323 Textual data......Page 325 The system of record......Page 332 Metadata......Page 333 Local individual data warehouses......Page 334 Data models and the healthcare information factory......Page 335 Creating the medical data warehouse data model......Page 341 The collective common data model......Page 342 Developing the healthcare information factory......Page 347 Healthcare information factory users......Page 351 Financing the infrastructure......Page 354 Implementing the healthcare information factory......Page 355 Further reading......Page 357 Summary......Page 358 Index......Page 360

A comprehensive revision of the premier resource on master data management (MDM)

Master Data Management and Data Governance explains how to grow revenue, reduce administrative costs, and improve client retention by adopting a customer-focused business framework. The book consists of five major parts:
Part I introduces the set of the business problems and tactical and strategic challenges associated with MDM transformations and provides examples and guidance on industry-specific approaches to solutions.

Part II looks at the processes and technologies of designing and implementing solutions.
Part III examines MDM from the regulatory compliance, privacy, and security viewpoints.
Part IV discusses implementation issues including the scope and complexity of the testing and field deployment strategy. It also provides an overview of the marketplace, leading vendors, including IBM and Oracle, and their products and strategic directions.
Part V offers a high-level analysis of key strategic and tactical approaches of delivering an enterprise-wide holistic business and technology initiative.



Master Data Management and Data Governance covers:

• Major MDM data domains and MDM applications by industry

• Evolution of MDM architecture

• Data modeling and management concerns of MDM architecture

• Architecting for entity and relationships resolution

• Analytical and operational MDM



Full details on MDM

Introduction to Master Data Management and Customer Data Integration; MDM: Overview of Market Drivers and Key Challenges; MDM Applications by Industry; Architectural Considerations; MDM Architecture Classification, Concepts, Principles and Components; Data Management Concerns of MDM Architecture: Entities, Hierarchies and Metadata; MDM Services for Entity and Relationships Resolution and Hierarchy Management; Master Data Modeling; Data Security, Privacy and Regulatory Compliance; Overview of Risk Management for Master Data; Introduction to Information Security and Identity Management; Protecting Content for Secure Master Data Management; Enterprise Security and Data Visibility in Master Data; Implementing and Governing Master Data Management; Building Business Case and Defining Data Governance Framework for MDM; Project Initiation; Identification, Matching, Aggregation and Holistic View of the Master Objects; Beyond Party Match: Merge, Split, Groups and Relationships; Data Synchronization and Integration Styles; Data Governance: Frameworks, Information Quality Processes and Metrics; Additional Implementation Considerations; Master Data Management: Market, Trends and Directions; MDM Roadmap; Regulations and Compliance Rules Impacting Master Data Management Projects

The latest techniques for building a customer-focused enterprise environment'The authors have appreciated that MDM is a complex multidimensional area, and have set out to cover each of these dimensions in sufficient detail to provide adequate practical guidance to anyone implementing MDM. While this necessarily makes the book rather long, it means that the authors achieve a comprehensive treatment of MDM that is lacking in previous works.'-- Malcolm Chisholm, Ph.D., President, AskGet.com Consulting, Inc. Regain control of your master data and maintain a master-entity-centric enterprise data framework using the detailed information in this authoritative guide. Master Data Management and Data Governance, Second Edition provides up-to-date coverage of the most current architecture and technology views and system development and management methods. Discover how to construct an MDM business case and roadmap, build accurate models, deploy data hubs, and implement layered security policies. Legacy system integration, cross-industry challenges, and regulatory compliance are also covered in this comprehensive volume. Plan and implement enterprise-scale MDM and Data Governance solutions Develop master data model Identify, match, and link master records for various domains through entity resolution Improve efficiency and maximize integration using SOA and Web services Ensure compliance with local, state, federal, and international regulations Handle security using authentication, authorization, roles, entitlements, and encryption Defend against identity theft, data compromise, spyware attack, and worm infection Synchronize components and test data quality and system performance

Data Warehousing in the Age of the Big Data will help you and your organization make the most of unstructured data with your existing data warehouse.

As Big Data continues to revolutionize how we use data, it doesn't have to create more confusion. Expert author Krish Krishnan helps you make sense of how Big Data fits into the world of data warehousing in clear and concise detail. The book is presented in three distinct parts. Part 1 discusses Big Data, its technologies and use cases from early adopters. Part 2 addresses data warehousing, its shortcomings, and new architecture options, workloads, and integration techniques for Big Data and the data warehouse. Part 3 deals with data governance, data visualization, information life-cycle management, data scientists, and implementing a Big Data–ready data warehouse. Extensive appendixes include case studies from vendor implementations and a special segment on how we can build a healthcare information factory.

Ultimately, this book will help you navigate through the complex layers of Big Data and data warehousing while providing you information on how to effectively think about using all these technologies and the architectures to design the next-generation data warehouse.

• Learn how to leverage Big Data by effectively integrating it into your data warehouse.
• Includes real-world examples and use cases that clearly demonstrate Hadoop, NoSQL, HBASE, Hive, and other Big Data technologies
• Understand how to optimize and tune your current data warehouse infrastructure and integrate newer infrastructure matching data processing workloads and requirements

"In conclusion as you come to the end of this book, the concept of a Data Warehouse and its primary goal of serving the enterprise version of truth, and being the single platform for all the source of information will continue to remain intact and valid for many years to come. As we have discussed across many chapters and in many case studies, the limitations that existed with the infrastructures to create, manage and deploy Data Warehouses have been largely eliminated with the availability of Big Data technologies and infrastructure platforms, making the goal of the single version of truth a feasible reality. Integrating and extending Big Data into the Data Warehouse, and creating a larger decision support platform will benefit businesses for years to come. This book has touched upon governance and information lifecycle management aspects of Big Data in the larger program, however you can reuse all the current program management techniques that you follow for the Data Warehouse for this program and even implement agile approaches to integrating and managing data in the Data Warehouse. Technologies will continue to evolve in this spectrum and there will be more additions of solutions, which can be integrated if you follow the modular integration approaches to building and managing the Data Warehouse. The Appendix sections contain many more case studies and a special section on Healthcare Information Factory based on Big Data approaches. These are more guiding posts to help you align your thoughts and goals to building and integrating Big Data in your Data Warehouse"-- Provided by publisher. The key to a successful MDM initiative isn’t technology or methods, it’s people: the stakeholders in the organization and their complex ownership of the data that the initiative will affect.

Master Data Management equips you with a deeply practical, business-focused way of thinking about MDM—an understanding that will greatly enhance your ability to communicate with stakeholders and win their support. Moreover, it will help you deserve their support: you’ll master all the details involved in planning and executing an MDM project that leads to measurable improvements in business productivity and effectiveness.

* Presents a comprehensive roadmap that you can adapt to any MDM project.
* Emphasizes the critical goal of maintaining and improving data quality.
* Provides guidelines for determining which data to “master.”
* Examines special issues relating to master data metadata.
* Considers a range of MDM architectural styles.
* Covers the synchronization of master data across the application infrastructure. The key to a successful MDM initiative isn't technology or methods, it's people: the stakeholders in the organization and their complex ownership of the data that the initiative will affect. Master Data Management equips you with a deeply practical, business-focused way of thinking about MDM-an understanding that will greatly enhance your ability to communicate with stakeholders and win their support. Moreover, it will help you deserve their support: you'll master all the details involved in planning and executing an MDM project that leads to measurable improvements in business productivity and effectiveness. * Presents a comprehensive roadmap that you can adapt to any MDM project. * Emphasizes the critical goal of maintaining and improving data quality. * Provides guidelines for determining which data to "master." * Examines special issues relating to master data metadata. * Considers a range of MDM architectural styles. * Covers the synchronization of master data across the application infrastructure Everything you ever wanted to know about growing grapes March and Simon's Organizations has become a classic in the field of organizational management for its broad scope and depth of information. Written by two of the most prominent experts in the field, this book offers invaluable insight on all aspects of organizational culture through deep discussion of organization theory. The definitive reference for topics including bounded rationality, satisficing, inducement/contribution balances, attention focus, uncertainty absorption and more, this seminal text offers authoritative insight with a practical grounding in the field. "The key to a successful MDM initiative isn't technology or methods, it's people: the stakeholders in the organization and their complex ownership of the data that the initiative will affect." "Master Data Management equips you with a deeply practical, business-focused way of thinking about MDM - an understanding that will greatly enhance your ability to communicate with stakeholders and win their support. Moreover, it will help you deserve their support. You'll master all the details involved in planning and executing an MDM project that leads to measurable improvements in business productivity and effectiveness."--Jacket Master data and master data management Coordination, stakeholders, requirements, and planning Components and the maturity model Data governance for master data management Data quality and MDM Metadata management for MDM Identifying master metadata and master data Data modeling for MDM Paradigms and architectures Data consolidation and integration Master data synchronization And the functional services layer Management guidance for MDM. This new edition incorporates a new introduction which places the material in its contemporary context, whilst still preserving the 1958 text. It examines such concepts as bounded rationality, satisficing, inducement/contribution balances, problem solving and uncertainty absorption.