MATURING THE SNOWFLAKE DATA CLOUD : a templated approach to delivering and governing snowflake... in large enterprises

Table of Contents About the Authors About the Technical Reviewer Acknowledgments Chapter 1: Maturing the Snowflake Data Cloud Introducing the Landing Zone Why Do You Need a LZ? How Do You Design a LZ? What Does a LZ Need? Snowflake Adoption What Is a Sandpit Account? Network Policy Data Egress Consumption Sample Configuration Terms of Usage Common Pitfalls Administrative Notes Classic User Interface (UI) SnowSQL PrivateLink Security Monitoring Cost Reporting Share Utilization Usage Reporting Data Catalog Organizations Alerting CDMC Behavior Change Management Disaster Recovery (DR) Summary Chapter 2: Implementing PrivateLink Retrieving the AWS Account ID Snowflake PrivateLink Functions Connectivity to Snowflake Service Creating an STS Profile Creating an IAM User Getting a Federation Token Verifying PrivateLink Is Authorized Authorizing PrivateLink Revoking PrivateLink Identifying PrivateLink Endpoints Configuring AWS VPC Configuring an AWS VPC Endpoint (VPCE) Creating CNAME Records Creating a Hosted Zone Creating a Record Testing PrivateLink Setup Creating a EC2 Instance Enabling VPC Endpoint Access Connecting to VPC Apple Mac Terminal Windows PuTTY/PuTTYgen Test DNS Connect via PrivateLink Endpoint to Reroute Snowflake Internal Stage Traffic Retrieving Snowflake’s Internal S3 Bucket Stage URL Creating an Endpoint Create Route 53/Hosted Zone Service Creating a Record Testing Your Endpoint Cleanup Troubleshooting Guide Confirm Configuration Check Authorized Endpoints Check That Endpoint Regions Align CloudShell Fails to Start IAM Policy Simulator AWS Account on Hold Raising a Support Ticket SSO Over PrivateLink Checklists Summary Chapter 3: Security Monitoring Control Definition What Is a Policy? What Is a Process? What Is a Procedure? Sample Expected Controls Automation Event Extraction Writing to an S3 Bucket Report Email External API Restricted Use of ACCOUNTADMIN Policy Process Disable Snowflake Sales Engineering Users Create the “Break Glass” User Engage the PAM Team Identify the Monitoring Recipients Pre-Use Notification Develop Notifications Test and Sign Off Procedure Disable or Remove Snowflake SE Users Create the “Break Glass” ACCOUNTADMIN User Extract Event Information Deliver Event Information Checklist Network Policy Policy Process Implement Account-Level Network Policy Implement a Third-Party-Specific Network Policy Identify the Monitoring Recipients Network Policy Change Notification Develop Notifications Test and Sign Off Procedure Create Account-Level Network Policy Create a Third-Party-Specific Network Policy Extract Event Information Deliver Event Information Checklist Additional Controls Unauthorized Data Download Policy Snowflake Custom Roles Continuous Monitoring Share Monitoring Tri-Secret Secure Data Masking Multiple Failed Logins Cleanup Periodic Access Recertification User Entitlement Data Shares Reader Accounts Creating a Reader Account Identifying Managed Accounts Additional Information Cleanup Security Information and Event Management (SIEM) Login History Query History Access History SaaS Security Posture Management (SSPM) Archive Data Retention Period Preserving login_history Maintaining login_history Automated Audit Maintenance Semi-Automated Audit Maintenance Deployment Cleanup Summary Chapter 4: Cost Reporting Snowflake Cost Monitoring Cost Drivers Compute Costs Storage Costs Data Transfer Costs Core Components Identifying Consumption Metrics Capturing Consumption Metrics Spoke – Creating a Consumption Metrics Repository Spoke - Creating Consumption Metrics Objects Creating Source Views Creating a Staging Table Creating a Task Rescheduling a Task Suspending a Task Propagating Consumption Metrics Spoke – Replicating a Database to CMS CMS – Ingesting a Replicated Database CMS – Building a Repository CMS – Refreshing a Replicated Database CMS – Accessing an Ingested Database CMS – Creating Consumption Metrics Objects CMS – Summary of Consumption Metrics Reporting Consumption Metrics External Cost Monitoring Maintenance Summary Chapter 5: Share Utilization Data Sharing Options Same Region and Same CSP Different Region or Different CSP Data Sharing Under the Covers Reader Accounts Approach Data Sharing Monitoring Secure Direct Data Share Private Listings Accessed Objects Usage and Billing Consumption of Shares Daily and 28-Day Telemetry Snowflake Marketplace Paid Listing Earnings Listing Subscriptions Daily Purchases Listings Published Data Clean Rooms What Are Data Clean Rooms? Why Data Clean Rooms? Establishing a Data Clean Room Resources Data Governance Perspective Summary Appendix Chapter 6: Usage Reporting Operating Environment Data Usage Constraints Toxic Entitlement Combinations Approach Who Can See What Searching by Object Tag Object Tag Propagation Creating View v_tag_references Tag Visibility by Role Searching by RBAC Creating View v_role_objects Identifying Toxic Entitlement Combinations Extending Monitoring Capability Integrating Object Tags and Object Access Searching by Row-Level Security (RLS) Setup Accessing Row Access Policies Row Access Policy Impact Entitlement, Tags, and Row Access Policies Cleanup Searching by Masking Policy Setup Accessing Data Masking Policies Masking Policy Impact Entitlement, Tags, Row Access Policies, and Masking Cleanup Searching By Tag-Based Masking Who Has Seen What Historic Query Capture Slowly Changing Dimension 2 (SCD2) Row Access Policy Changes Whole Record Hashing Time Travel/Fail Safe What Did They Do With What They Saw? Summary Chapter 7: Data Catalog Overview Accessing Metadata Catalog Database Creating Catalog Schema Views Provisioning Read Access Extending Capability External Tooling Referential Integrity Data Quality Generating Metadata Summary Chapter 8: Organization Impact of Organizational Growth Organic Growth Inorganic Growth Master Service Agreements Quarterly Business Reviews Role Hierarchy Snowflake Organization Features Using the Orgadmin Role Identifying Your Default Organization Enabling Account Replication Disable Account Replication Listing Organization Accounts Delivering an Account Ownership Naming Convention Credit Allocation CSP Provision Upskilling Creating An Account Checklist Renaming An Account Dropping An Account Snowflake Organization Metrics Orgadmin Account Usage Store Access Organization Usage Views Preview Views Other Views View Comparison View Coverage View Usage Provision Read Access Summary Chapter 9: Alerting Setup Creating Email Integration Environment Test system$send_email Creating the JavaScript Email Procedure Creating a Python Email Procedure Configuring Alerts Setting Up a Logging Table Account Level Entitlement Example Account Alert Requirement Configuring an Account Level Alert Extending Alert Capability Alert History Cleanup Summary Chapter 10: Cloud Data Management Capabilities (CDMC) Overview Data Catalog Data Ingestion Patterns Reference Architecture Evidence Repository Repository Overview Repository Setup Object Tagging Document Ingestion Unstructured Ingestion Pattern Template Document Cleanup Script Governance and Accountability Business Cases Are Defined and Governed Business Cases Are Defined Business Cases Are Syndicated and Governed Data Sovereignty and Cross-Border Data Movement Are Managed Data Sovereignty Is Tracked Evidence Views Control Specific View Object Tags Control Reports Summary Appendix – Supporting Documentation Business Case Executive Summary Problem Statement Snowflake Specific Planned Activities Expected Business Outcomes Project Governance Resources Timelines Costs Appendix Solution Design Document Business Case Business Summary System Context Diagram Technology Summary Operational Support Data Lifecycle Feed Onboarding Contract Business Case Feed Context Diagram Usage Data Ownership Component Delivery Project Plan System Catalog Maintenance User Acceptance Testing Operational Support Release into Production Endpoints Authentication Connectivity Tooling Operations Procedures Logging Data Sets Validations Transformations Enrichments Data Management Signoff Chapter 11: Behavior Change Management Explaining Behavior Change Bundles Implementing Behavior Change Bundles All or Nothing Multi-Tenant Considerations Creating a Test Account Checking Behavior Change Bundle Status Enabling Behavior Change Bundles Capturing Test Case Output Disabling Behavior Change Bundles Baseline Test Case Output Actions Behavior Change Summary Checklist Sample Test Harness Summary Chapter 12: Disaster Recovery Snowflake Availability Business Continuity Plan RPO and RTO Service-Level Agreement (SLA) Accuracy Integrity Completeness Timeliness Performance Summary Organizational Considerations Resilience Data Recovery Fault Tolerance Availability Scalability Self-Healing Automation Redundancy Monitoring and Alerting Resilience Summary Scenarios and Options Supported Cloud Regions Legal Agreements CSP Concentration Location Strategy Costs Secure Direct Data Share Database Replication Replicated Objects Currently Available Objects Future Available Objects Unavailable Objects Excluded Objects Manual Configuration Database Replication Limitations Database Replication Step by Step Suspended Account Primary Account Setup 1 Secondary Account Setup 1 Primary Account Setup 2 Disable Database Replication Database Replication Summary Account Replication Single Region Failure Multi-Region Failure Replicated Objects Currently Available Objects Future Available Objects Unavailable Objects and Entitlements Excluded Objects Manual Configuration Account Replication Limitations Account Replication Step by Step Replication Groups/Failover Groups Replication Group Limitations Primary Account Setup 1 Secondary Account Setup 1 Global Identifiers Primary Account Setup 2 Secondary Account Setup 2 Refresh Secondary Refresh Costs Promote Secondary to Primary Primary Account Setup 3 Secondary Account Setup 3 Account Replication Summary Client Redirect Redirect Scenarios Client Redirect Step By Step Primary Account Setup 1 Secondary Account Setup 1 Private Connectivity Redirecting Client Connections Secondary Account Setup 2 Primary Account Setup 2 Client Redirect Considerations Client Redirect Summary Holistic Approach External Data Source Failure CSP Tooling Feed Onboarding Contracts Network Topology SLA Impact Account Migration DR Plan Assess The Impact Invoke DR Action Service Restoration DR Test DR Checklist Troubleshooting Check Snowflake Availability Check For Snowflake Email Has Snowflake Released a New Version? Has a Change Bundle Been Enabled? Failure Scope Summary Index This project-oriented book gives you a hands-on approach to designing, developing, and templating your Snowflake platform delivery. Written by seasoned Snowflake practitioners, the book is full of practical guidance and advice to accelerate and mature your Snowflake journey. Working through the examples helps you develop the skill, knowledge, and expertise to expand your organization’s core Snowflake capability and prepare for later incorporation of additional Snowflake features as they become available. Your Snowflake platform will be resilient, fit for purpose, extensible, and guarantee rapid, consistent, and repeatable, pattern-based deployments ready for application delivery. When a Snowflake account is delivered there are no controls, guard rails, external monitoring, nor governance mechanisms baked in. From a large organization perspective, this book explains how to deliver your core Snowflake platform in the form of a Landing Zone, a consistent, templated approach that assumes familiarity with Snowflake core concepts and principles. The book also covers Snowflake from a governance perspective and addresses the “who can see what?” question, satisfying requirements to know for certain that your Snowflake accounts properly adhere to your organization’s data usage policies. The book provides a proven pathway to success by equipping you with skill, knowledge, and expertise to accelerate Snowflake adoption within your organization. The patterns delivered within this book are used for production deployment, and are proven in real-world use. Examples in the book help you succeed in an environment in which governance policies, processes, and procedures oversee and control every aspect of your Snowflake platform development and delivery life cycle. Your environment may not be so exacting, but you’ll still benefit from the rigorous and demanding perspective this book’s authors bring to the table. The book shows you how to leverage what you already know and adds what you don’t know, all applied to deliver your Snowflake accounts. You will know how to position your organization to deliver consistent Snowflake accounts that are prepared and ready for immediate application development. What You Will Learn Create a common, consistent deployment framework for Snowflake in your organization Enable rapid up-skill and adoption of Snowflake, leveraging the benefits of cloud platforms Develop a deep understanding of Snowflake administration and configuration Implement consistent, approved design patterns that reduce account provisioning times Manage data consumption by monitoring and controlling access to datasets Who This Book Is For Systems administrators charged with delivering a common implementation pattern for all Snowflake accounts within an organization; senior managers looking to simplify the delivery of complex technology into their existing infrastructure; developers seeking to understand guard rails, monitoring, and controls to ensure that Snowflake meets their organization's requirements; sales executives needing to understand how their data usage can be monitored and gain insights into how their data is being consumed; governance colleagues wanting to know who can see each data set, and wanting to identify toxic role combinations, and have confidence that their Snowflake accounts are properly provisioned