Practical Implementation of a Data Lake : Translating Customer Expectations Into Tangible Technical Goals

This book explains how to implement a data lake strategy, covering the technical and business challenges architects commonly face. It also illustrates how and why client requirements should drive architectural decisions. Drawing upon a specific case from his own experience, author Nayanjyoti Paul begins with the consideration from which all subsequent decisions should flow: what does your customer need? He also describes the importance of identifying key stakeholders and the key points to focus on when starting a new project. Next, he takes you through the business and technical requirement-gathering process, and how to translate customer expectations into tangible technical goals. From there, you’ll gain insight into the security model that will allow you to establish security and legal guardrails, as well as different aspects of security from the end user’s perspective. You’ll learn which organizational roles need to be onboarded into the data lake, their responsibilities, the services they need access to, and how the hierarchy of escalations should work. Subsequent chapters explore how to divide your data lakes into zones, organize data for security and access, manage data sensitivity, and techniques used for data obfuscation. Audit and logging capabilities in the data lake are also covered before a deep dive into designing data lakes to handle multiple kinds and file formats and access patterns. The book concludes by focusing on production operationalization and solutions to implement a production setup. After completing this book, you will understand how to implement a data lake, the best practices to employ while doing so, and will be armed with practical tips to solve business problems. What You Will Learn Understand the challenges associated with implementing a data lake Explore the architectural patterns and processes used to design a new data lake Design and implement data lake capabilities Associate business requirements with technical deliverables to drive success Who This Book Is For Data Scientists and Architects, Machine Learning Engineers, and Software Engineers. Table of Contents About the Author About the Technical Reviewer Preface Introduction Chapter 1: Understanding “the Ask” Objective: Asking the Right Questions The Recommendations Decide on the Migration Path, Modernization Techniques, Enhancements, and the Cloud Vendor Assess the Current Challenges Understand Why Modernizing Data Platforms Is Hard Determine the Top Five Issues to Solve Determine What Is Available On-Premise vs. on the Cloud Create the Meetings Needed Throughout the Project Define Common Terms and Jargon Key Takeaways Chapter 2: Enabling the Security Model Objective: Identifying the Security Considerations The Recommendations PII Columns: RBAC, ABAC Features Central Access Control Authentication and Authorization (SAML vs. PING, etc.) Strategy for Data Obfuscation GDPR and Other Data Privacy Ownership of the Platform, Interaction with Other Stakeholders (CISO, Legal Teams, etc.) Legal/Contractual Obligations on Getting/Connecting Data from a Third Party on the Cloud Key Takeaways Chapter 3: Enabling the Organizational Structure Objective: Identifying the Organizational Structure and Role The Recommendations Example Template for the Project Key Takeaways Chapter 4: The Data Lake Setup Objective: Detailed Design of the Data Lake The Recommendations Structuring the Different Zones in the Data Lake Defining the Folder Structure of the Zones with a Hierarchy Structuring Data from Relational Stores (Raw Zone) Structuring Data from Relational Stores (Curated Zone) Structuring Data from Relational Stores (Provisioned/Gold Zone) Managing Data Sensitivity as Part of the Folder Structure Design Setting the Encryption/Data Management Keys for Organizing Data Quick FAQs on the Data-at-Rest and Data-in-Transit Encryption Looking at Data Management Principles Understanding Data Flows Setting the Right Access Control for Each Zone Understanding File Formats and Structures in Each Zone Key Takeaways Chapter 5: Production Playground Objective: Production Playground The Recommendations What Is a Production Playground? What Issues Will This Address? What Is a Production Playground Not ? What Does the Production Playground Consist Of? Key Takeaways Chapter 6: Production Operationalization Objective: Production Operationalization The Recommendations Key Takeaways Chapter 7: Miscellaneous Objective: Advice to Follow Recommendations Managing a Central Framework Along with Project-Specific Extensions Allowing Project Teams to Build “User-Defined Procedures” and Contribute to the Central Framework Advantages and Disadvantages of a Single vs. Multi-account Strategy Creating a New Organizational Unit AWS Account vs. Onboard Teams to a Central IT Managed AWS Account Considerations for Integrating with Schedulers Choosing a Data Warehouse Technology Managing Autoscaling Managing Disaster Recovery AWS Accounts Used for Delivery Data Platform Cost Controls Common Anti-patterns to Avoid One-Size-Fits-All Ignoring Security Data Sprawl Poor Data Governance Lack of Quality Controls Poor Metadata Management Wrong Tools Avoid Over-Engineering Poor Data Integration Unstructured Data Overload Key Takeaways Index