Planetary Stock Trading

Graph data closes the gap between the way humans and computers view the world. While computers rely on static rows and columns of data, people navigate and reason about life through relationships. This practical guide demonstrates how graph data brings these two approaches together. By working with concepts from graph theory, database schema, distributed systems, and data analysis, you’ll arrive at a unique intersection known as graph thinking. Authors Denise Koessler Gosnell and Matthias Broecheler show data engineers, data scientists, and data analysts how to solve complex problems with graph databases. You’ll explore templates for building with graph technology, along with examples that demonstrate how teams think about graph data within an application. • Build an example application architecture with relational and graph technologies • Use graph technology to build a Customer 360 application, the most popular graph data pattern today • Dive into hierarchical data and troubleshoot a new paradigm that comes from working with graph data • Find paths in graph data and learn why your trust in different paths motivates and informs your preferences • Use collaborative filtering to design a Netflix-inspired recommendation system Cover......Page 1 Copyright......Page 4 Table of Contents......Page 5 Preface......Page 13 Who Should Read This Book......Page 14 Navigating This Book......Page 15 Conventions Used in This Book......Page 16 O’Reilly Online Learning......Page 17 Acknowledgments......Page 18 Chapter 1. Graph Thinking......Page 21 Why Now? Putting Database Technologies in Context......Page 22 1960s–1980s: Hierarchical Data......Page 23 1980s–2000s: Entity-Relationship......Page 24 2000s–2020s: NoSQL......Page 25 2020s–?: Graph......Page 27 What Is Graph Thinking?......Page 29 Complex Problems in Business......Page 30 Making Technology Decisions to Solve Complex Problems......Page 32 So You Have Graph Data. What’s Next?......Page 35 Seeing the Bigger Picture......Page 39 Getting Started on Your Journey with Graph Thinking......Page 40 Chapter Preview: Translating Relational Concepts to Graph Terminology......Page 41 Relational Versus Graph: What’s the Difference?......Page 42 Data for Our Running Example......Page 43 Relational Data Modeling......Page 45 Entities and Attributes......Page 46 Building Up to an ERD......Page 47 Fundamental Elements of a Graph......Page 48 Adjacency......Page 49 Distance......Page 50 Degree......Page 51 Vertex Labels and Edge Labels......Page 53 Properties......Page 54 Edge Direction......Page 55 Multiplicity of Your Graph......Page 58 Full Example Graph Model......Page 61 Understanding Graph Data......Page 63 Summary......Page 64 Chapter 3. Getting Started: A Simple Customer 360......Page 67 The Foundational Use Case for Graph Data: C360......Page 68 Why Do Businesses Care About C360?......Page 70 Data Models......Page 71 Relational Implementation......Page 74 Example C360 Queries......Page 78 Implementing a C360 Application in a Graph System......Page 81 Data Models......Page 82 Graph Implementation......Page 83 Example C360 Queries......Page 90 Relational Versus Graph: Data Modeling......Page 95 Relational Versus Graph: Query Languages......Page 96 Relational Versus Graph: Main Points......Page 97 Summary......Page 98 Making a Technology Choice for Your C360 Application......Page 99 Chapter Preview: Building a More Realistic Customer 360......Page 101 Graph Data Modeling 101......Page 102 Should This Be a Vertex or an Edge?......Page 103 Lost Yet? Let Us Walk You Through Direction......Page 106 A Graph Has No Name: Common Mistakes in Naming......Page 109 Our Full Development Graph Model......Page 111 Before We Start Building......Page 113 Our Thoughts on the Importance of Data, Queries, and the End User......Page 114 Implementation Details for Exploring Neighborhoods in Development......Page 115 Basic Gremlin Navigation......Page 117 Advanced Gremlin: Shaping Your Query Results......Page 126 Shaping Query Results with the project(), fold(), and unfold() Steps......Page 127 Removing Data from the Results with the where(neq()) Pattern......Page 130 Planning for Robust Result Payloads with the coalesce() Step......Page 132 Moving from Development into Production......Page 135 Chapter 5. Exploring Neighborhoods in Production......Page 137 Chapter Preview: Understanding Distributed Graph Data in Apache Cassandra......Page 139 The Most Important Topic to Understand About Data Modeling: Primary Keys......Page 140 Partition Keys and Data Locality in a Distributed Environment......Page 141 Understanding Edges, Part 1: Edges in Adjacency Lists......Page 146 Understanding Edges, Part 2: Clustering Columns......Page 148 Understanding Edges, Part 3: Materialized Views for Traversals......Page 152 Graph Data Modeling 201......Page 156 Finding Indexes with an Intelligent Index Recommendation System......Page 160 Materialized Views and Adding Time onto Edges......Page 162 Our Final C360 Production Schema......Page 164 Bulk Loading Graph Data......Page 166 Updating Our Gremlin Queries to Use Time on Edges......Page 169 Our First 10 Tips to Get from Development to Production......Page 172 Chapter Preview: Navigating Trees, Hierarchical Data, and Cycles......Page 175 Hierarchical Data in a Bill of Materials......Page 176 Hierarchical Data in Self-Organizing Networks......Page 177 Why Graph Technology for Hierarchical Data?......Page 178 Trees, Roots, and Leaves......Page 179 Depth in Walks, Paths, and Cycles......Page 180 Understanding Hierarchies with Our Sensor Data......Page 182 Understand the Data......Page 183 Conceptual Model Using the GSL Notation......Page 190 Implement Schema......Page 191 Querying from Leaves to Roots in Development......Page 194 Where Has This Sensor Sent Information To?......Page 195 From This Sensor, What Was Its Path to Any Tower?......Page 198 Querying from Roots to Leaves in Development......Page 204 Setup Query: Which Tower Has the Most Sensor Connections So That We Could Explore It for Our Example?......Page 205 Which Sensors Have Connected Directly to Georgetown?......Page 206 Find All Sensors That Connected to Georgetown......Page 207 Depth Limiting in Recursion......Page 209 Going Back in Time......Page 210 Chapter Preview: Understanding Branching Factor, Depth, and Time on Edges......Page 211 Understanding Time in the Sensor Data......Page 212 Understanding Branching Factor in Our Example......Page 220 What Is Branching Factor?......Page 221 How Do We Get Around Branching Factor?......Page 222 Production Schema for Our Sensor Data......Page 223 Where Has This Sensor Sent Information to, and at What Time?......Page 225 From This Sensor, Find All Trees up to a Tower by Time......Page 226 From This Sensor, Find a Valid Tree......Page 229 Advanced Gremlin: Understanding the where().by() Pattern......Page 231 Querying from Roots to Leaves in Production......Page 233 Which Sensors Have Connected to Georgetown Directly, by Time?......Page 234 What Valid Paths Can We Find from Georgetown Down to All Sensors?......Page 235 Applying Your Queries to Tower Failure Scenarios......Page 238 Seeing the Forest for the Trees......Page 243 Chapter 8. Finding Paths in Development......Page 245 How Much Do You Trust That Open Invitation?......Page 246 How Defensible Is an Investigator’s Story?......Page 247 How Do Companies Model Package Delivery?......Page 248 Fundamental Concepts About Paths......Page 249 Shortest Paths......Page 250 Depth-First Search and Breadth-First Search......Page 252 Learning to See Application Features as Different Path Problems......Page 253 Source Data......Page 254 Creating Our Development Schema......Page 256 Loading Data......Page 257 Exploring Communities of Trust......Page 258 Which Addresses Are in the First Neighborhood?......Page 260 Which Addresses Are in the Second Neighborhood?......Page 261 Which Addresses Are in the Second Neighborhood, but Not the First?......Page 262 Evaluation Strategies with the Gremlin Query Language......Page 264 Pick a Random Address to Use for Our Example......Page 265 Shortest Path Queries......Page 266 Finding Paths of a Fixed Length......Page 267 Finding Paths of Any Length......Page 270 Augmenting Our Paths with the Trust Scores......Page 273 Do You Trust This Person?......Page 279 Chapter 9. Finding Paths in Production......Page 281 Weighted Paths and Search Algorithms......Page 282 Shortest Weighted Path Problem Definition......Page 283 Shortest Weighted Path Search Optimizations......Page 284 Normalizing the Edge Weights......Page 287 Updating Our Graph......Page 292 Exploring the Normalized Edge Weights......Page 293 Shortest Weighted Path Queries......Page 297 Building a Shortest Weighted Path Query for Production......Page 298 Weighted Paths and Trust in Production......Page 308 Chapter 10. Recommendations in Development......Page 311 How We Give Recommendations in Healthcare......Page 312 How We Experience Recommendations in Social Media......Page 313 How We Use Deeply Connected Data for Recommendations in Ecommerce......Page 314 Understanding the Problem and Domain......Page 315 Collaborative Filtering with Graph Data......Page 317 Recommendations via Item-Based Collaborative Filtering with Graph Data......Page 318 Three Different Models for Ranking Recommendations......Page 319 Data Model for Movie Recommendations......Page 323 Schema Code for Movie Recommendations......Page 325 Loading the Movie Data......Page 327 Neighborhood Queries in the Movie Data......Page 331 Tree Queries in the Movie Data......Page 334 Path Queries in the Movie Data......Page 336 Model 1: Counting Paths in the Recommendation Set......Page 338 Model 2: NPS-Inspired......Page 339 Model 3: Normalized NPS......Page 342 Choosing Your Own Adventure: Movies and Graph Problems Edition......Page 344 Chapter Preview: Merging Multiple Datasets into One Graph......Page 345 Defining a Different Complex Problem: Entity Resolution......Page 346 Seeing the Complex Problem......Page 348 MovieLens Dataset......Page 349 Kaggle Dataset......Page 356 Development Schema......Page 359 Our Matching Process......Page 360 False Positives Found in the MovieLens Dataset......Page 363 Additional Errors Discovered in the Entity Resolution Process......Page 364 Final Analysis of the Merging Process......Page 366 The Role of Graph Structure in Merging Movie Data......Page 367 Chapter 12. Recommendations in Production......Page 369 Shortcut Edges for Recommendations in Real Time......Page 370 Where Our Development Process Doesn’t Scale......Page 371 How We Fix Scaling Issues: Shortcut Edges......Page 372 Seeing What We Designed to Deliver in Production......Page 373 Pruning: Different Ways to Precompute Shortcut Edges......Page 374 Considerations for Updating Your Recommendations......Page 376 Breaking Down the Complex Problem of Precalculating Shortcut Edges......Page 377 Addressing the Elephant in the Room: Batch Computation......Page 382 Production Schema and Data Loading for Movie Recommendations......Page 383 Production Schema for Movie Recommendations......Page 384 Production Data Loading for Movie Recommendations......Page 385 Recommendation Queries with Shortcut Edges......Page 386 Confirming Our Edges Loaded Correctly......Page 387 Production Recommendations for Our User......Page 388 Understanding Response Time in Production by Counting Edge Partitions......Page 392 Final Thoughts on Reasoning About Distributed Graph Query Performance......Page 395 Chapter 13. Epilogue......Page 397 Graph Algorithms......Page 398 Distributed Graphs......Page 399 Network Theory......Page 400 Stay in Touch......Page 402 Index......Page 403 Colophon......Page 418 How Do You Apply Graph Thinking To Solve Complex Problems? With This Practical Guide, Data Scientists Will Learn How To Think About Data As A Graph And Determine If Graph Technology Is Right For Your Company. You'll Learn Techniques For Building Scalable, Real-time, And Multimodel Architectures That Solve Complex Problems With Graph Data. Authors Denise Koessler Gosnell And Matthias Broecheler Show You How Companies Today Are Successfully Applying Graph Thinking In Distributed Production Environments. You'll Also Learn The Graph Schema Language, A Set Of Terminology And Visual Illustrations To Normalize How Graph Practitioners Communicate Conceptual Graph Models, Graph Schema, And Graph Database Design.