The enterprise Big Data Lake : delivering on the promise of Big Data and data science

The data lake is a daring new approach for harnessing the power of big data technology and providing convenient self-service capabilities. But is it right for your company? This book is based on discussions with practitioners and executives from more than a hundred organizations, ranging from data-driven companies such as Google, LinkedIn, and Facebook, to governments and traditional corporate enterprises. You’ll learn what a data lake is, why enterprises need one, and how to build one successfully with the best practices in this book. Alex Gorelik, CTO and founder of Waterline Data, explains why old systems and processes can no longer support data needs in the enterprise. Then, in a collection of essays about data lake implementation, you’ll examine data lake initiatives, analytic projects, experiences, and best practices from data experts working in various industries. • Get a succinct introduction to data warehousing, big data, and data science • Learn various paths enterprises take to build a data lake • Explore how to build a self-service model and best practices for providing analysts access to the data • Use different methods for architecting your data lake • Discover ways to implement a data lake from experts in different industries Copyright Table of Contents Preface Who Should Read This Book? Conventions Used in This Book O’Reilly Online Learning How to Contact Us Acknowledgments Chapter 1. Introduction to Data Lakes Data Lake Maturity Data Puddles Data Ponds Creating a Successful Data Lake The Right Platform The Right Data The Right Interface The Data Swamp Roadmap to Data Lake Success Standing Up a Data Lake Organizing the Data Lake Setting Up the Data Lake for Self-Service Data Lake Architectures Data Lakes in the Public Cloud Logical Data Lakes Conclusion Chapter 2. Historical Perspective The Drive for Self-Service Data—The Birth of Databases The Analytics Imperative—The Birth of Data Warehousing The Data Warehouse Ecosystem Storing and Querying the Data Loading the Data—Data Integration Tools Organizing and Managing the Data Consuming the Data Conclusion Chapter 3. Introduction to Big Data and Data Science Hadoop Leads the Historic Shift to Big Data The Hadoop File System How Processing and Storage Interact in a MapReduce Job Schema on Read Hadoop Projects Data Science What Should Your Analytics Organization Focus On? Machine Learning Explainability Change Management Conclusion Chapter 4. Starting a Data Lake The What and Why of Hadoop Preventing Proliferation of Data Puddles Taking Advantage of Big Data Leading with Data Science Strategy 1: Offload Existing Functionality Strategy 2: Data Lakes for New Projects Strategy 3: Establish a Central Point of Governance Which Way Is Right for You? Conclusion Chapter 5. From Data Ponds/Big Data Warehouses to Data Lakes Essential Functions of a Data Warehouse Dimensional Modeling for Analytics Integrating Data from Disparate Sources Preserving History Using Slowly Changing Dimensions Limitations of the Data Warehouse as a Historical Repository Moving to a Data Pond Keeping History in a Data Pond Implementing Slowly Changing Dimensions in a Data Pond Growing Data Ponds into a Data Lake—Loading Data That’s Not in the Data Warehouse Raw Data External Data Internet of Things (IoT) and Other Streaming Data Real-Time Data Lakes The Lambda Architecture Data Transformations Target Systems Data Warehouses Operational Data Stores Real-Time Applications and Data Products Conclusion Chapter 6. Optimizing for Self-Service The Beginnings of Self-Service Business Analysts Finding and Understanding Data—Documenting the Enterprise Establishing Trust Provisioning Preparing Data for Analysis Data Wrangling in the Data Lake Situating Data Preparation in Hadoop Common Use Cases for Data Preparation Analyzing and Visualizing The New World of Self-Service Business Intelligence The New Analytic Workflow Gatekeepers to Shopkeepers Governing Self-Service Conclusion Chapter 7. Architecting the Data Lake Organizing the Data Lake Landing or Raw Zone Gold Zone Work Zone Sensitive Zone Multiple Data Lakes Advantages of Keeping Data Lakes Separate Advantages of Merging the Data Lakes Cloud Data Lakes Virtual Data Lakes Data Federation Big Data Virtualization Eliminating Redundancy Conclusion Chapter 8. Cataloging the Data Lake Organizing the Data Technical Metadata Business Metadata Tagging Automated Cataloging Logical Data Management Sensitive Data Management and Access Control Data Quality Relating Disparate Data Establishing Lineage Data Provisioning Tools for Building a Catalog Tool Comparison The Data Ocean Conclusion Chapter 9. Governing Data Access Authorization or Access Control Tag-Based Data Access Policies Deidentifying Sensitive Data Data Sovereignty and Regulatory Compliance Self-Service Access Management Provisioning Data Conclusion Chapter 10. Industry-Specific Perspectives Big Data in Financial Services Consumers, Digitization, and Data Are Changing Finance as We Know It Saving the Bank New Opportunities Offered by New Data Key Processes in Making Use of the Data Lake Value Added by Data Lakes in Financial Services Data Lakes in the Insurance Industry Smart Cities Big Data in Medicine Index About the Author Colophon