The Art of Immutable Architecture : Theory and Practice of Data Management in Distributed Systems

This book teaches you how to evaluate a distributed system from the perspective of immutable objects. You will understand the problems in existing designs, know how to make small modifications to correct those problems, and learn to apply the principles of immutable architecture to your tools.Most software components focus on the state of objects. They store the current state of a row in a relational database. They track changes to state over time, making several basic assumptions: there is a single latest version of each object, the state of an object changes sequentially, and a system of record exists.This is a challenge when it comes to building distributed systems. Whether dealing with autonomous microservices or disconnected mobile apps, many of the problems we try to solve come down to synchronizing an ever-changing state between isolated components. Distributed systems would be a lot easier to build if objects could not change.After reading **__The Art of Immutable Architecture__**, you will come away with an understanding of the benefits of using immutable objects in your own distributed systems. You will learn a set of rules for identifying and exchanging immutable objects, and see a collection of useful theorems that emerges and ensures that the distributed systems we build are eventually consistent. Using patterns, you will find where the truth converges, see how changes are associative, rather than sequential, and come to feel comfortable understanding that there is no longer a single source of truth. Practical hands-on examples reinforce how to build software using the described patterns, techniques, and tools. By the end, you will possess the language and resources needed to analyze and construct distributed systems with confidence. The assumptions of the past were sufficient for building single-user, single-computer systems. But as we expand to multiple devices, shared experiences, and cloud computing, they work against us. It is time for a new set of assumptions. Start with immutable objects, and build better distributed systems. **What You Will Learn*** Evaluate a distributed system from the perspective of immutable objects * Recognize the problems in existing designs, and make small modifications to correct them * Start a new system from scratch, applying patterns Apply the principles of immutable architecture to your tools, including SQL databases, message queues, and the network protocols that you already use * Discover new tools that natively apply these principles **Who This Book Is For**Software architects and senior developers. It contains examples in SQL and languages such as JavaScript and C#. Past experience with distributed computing, data modeling, or business analysis is helpful. Table of Contents About the Author About the Technical Reviewer Acknowledgments Introduction Part I: Definition Chapter 1: Why Immutable Architecture The Immutability Solution The Problems with Immutability Begin a New Journey The Fallacies of Distributed Computing The Network Is Not Reliable Latency Is Not Zero Topology Doesn’t Change Changing Assumptions Immutability Changes Everything Shared Mutable State Structural Sharing The Two Generals’ Problem A Prearranged Protocol Reducing the Uncertainty An Additional Message Proof of Impossibility Relaxing Constraints Redefining the Problem Decide and Act Accept the Truth A Valid Protocol Examples of Immutable Architectures Git Blockchain Docker Chapter 2: Forms of Immutable Architecture Deriving State from History Historical Records Building Upon the Past Evolution of Understanding Mutable Objects Identity Evolution of State Projections Two Kinds of State Projecting Objects Event Sourcing Generating Events CQRS DDD Taking a Functional View Commutative and Idempotent Events Asynchronous Model View Update The Update Loop Unidirectional Data Flow Immutable App Architecture Historical Modeling Partial Order Predecessors Successors Immutable Graphs Collaboration Acyclic Graphs Timeliness Limitations of Historical Modeling No Central Authority No Real-Time Clock No Uniqueness Constraints No Aggregation Chapter 3: How to Read a Historical Model Fact Type Graphs A Chess Game Important Attributes A Chain of Facts Endgame Fact Instance Graphs The Immortal Game Collecting Moves A Brilliant Win The Factual Modeling Language Declaring Fact Types Querying the Model Jumping Levels Joining Matches Existential Quantifiers Current Value Authorization Rules A Chess Application Use Cases User Interface Actions Views Part II: Application Chapter 4: Location Independence Modeling with Immutability Synchronization Exploring Contracts Identity Auto-incremented IDs Environment Dependence Parent–Child Insertion Remote Creation URLs Location-Independent Identity Natural Keys GUIDs Timestamps Tuples Hashes Public Keys Random Numbers Causality Putting Steps in Order The Transitive Property Concurrency Partial Order The CAP Theorem Defining CAP Proving the CAP Theorem Test an Algorithm Eventual Consistency Kinds of Consistency Strong Eventual Consistency in a Relay-Based System Idempotence and Commutativity Deriving Strong Eventual Consistency The Contact Management System Replaying History Conflict-Free Replicated Data Types (CRDTs) State-Based CRDTs Partially Ordered State Causal History Vector Clocks A History of Facts Sets Partial Order Update Merge Historical Records Distinguishing Between Records Removing a Record Changing a Record Records Are Causally Related Benefits of Explicit Causality Historical Facts Conclusion Chapter 5: Analysis Use Cases From Use Case to Decision From Extension to Succession Data Identifiers Cardinality Mutation Views Finding a Place to Start Annotated Wireframes Removal from Lists Collaboration Regions Crossing Boundaries Conversations Publishing Facts Integrating Subsystems Valid Orderings Eliminating Race Conditions Responding to Different Valid Orderings Consequences Indexes Uniqueness Constraints Navigation Searching Expected Number of Results No Implicit Order Aggregates Iterations Creation Order Chapter 6: State Transitions Many Properties Shipping and Billing Introducing Back-Orders Cancellations and Returns Parallel State Machines Many Children Software Issue Tracking Child State Composite State Transition Diagrams A Declarative Function of States Conditional Validation Nullability Based on State Cycles in State Transition Collect Data During Transitions Immutable State Transitions The Question Behind State Translating a State Machine to a Historical Model Order Fulfillment Software Change Tracking Reasons for Computing State Handling the Next Action Finding Work Items Executing Compensating Transactions Single Source of Truth Orchestrators Consistent State Central Validation Convergent Histories Define Immutable Records Query for the Next Action Capture Actions Locally Define Compensating Actions Chapter 7: Security Proof of Authorship Key Pairs Digest Authorization Principal Facts Authorization Query Initial Authorization Grant of Authority Limited Authority Indefinite Authorization Transitive Authorization Revocation Authorization Upon Receipt Confidentiality Untrusted Nodes Asymmetric Encryption Asymmetric Size Limit Encrypt the Symmetric Key Encrypting Historical Facts Limit the Distribution of Confidential Facts Distribution Rules Evidence Attacks and Countermeasures Secrecy Shared Symmetric Key A Secret Discussion Channel Creating a Secret Channel Team Distribution Rules Limit the Scope of a Shared Key Cohorts Periods Chapter 8: Patterns Structural Patterns Entity Structure Example Consequences Related Patterns Ownership Structure Example Consequences Related Patterns Delete Structure Example Consequences Related Patterns Restore Structure Example Consequences Related Patterns Membership Structure Example Consequences Related Patterns Mutable Property Structure Example Consequences Related Patterns Entity Reference Structure Example Consequences Related Patterns Workflow Patterns Transaction Structure Example Consequences Related Patterns Queue Structure Example Consequences Related Patterns Period Structure Example Consequences Related Patterns Outbox Structure Journaling Random Processing Delays Rendezvous Hashing Service Failure Example Consequences Related Patterns Designing from Constraints Part III: Implementation Chapter 9: Query Inverses Mechanizing the Problem The Anatomy of a Query A Sequence of Steps Filter by Existential Condition The Affected Set Computing the Affected Set Inverting Longer Queries Unsatisfiable Inverses Walking Backward Proof of Completeness New Results Forward Optimization Existential Conditions Recursive Inversion Tail Conditions Removing Results When Removal Isn’t Removal Nested Subqueries Tautological Conditions Proof of Completeness Continued Potential vs. Actual Change Removing Absent Results Caches Are Sets Query Inversion in Practice Chapter 10: SQL Databases Identity Content-Addressed Storage Advantages Hash Collisions Avoid Hashes As Primary Keys Table Structure Relationships Inserting Successors Optional Predecessors Many Predecessors Canonical Hash of a Set Inserting Many Predecessors Queries Joins Correlated Subqueries Derived Tables Selecting Results Optimization Spurious Joins Covering Indexes Where Not Exists Mutable Properties Deletion Queues Integration Legacy Application Integration Scanners Triggers Change Data Capture Reporting Databases Application-Agnostic Stores A Generic Fact Table Predecessor Relationships Versioning Avoid Sequential Version Numbers Structural Versioning Chapter 11: Communication Delivery Guarantees Best Effort Confirmation Safe Methods Idempotent Methods Non-idempotent Methods Retry Within a Connection Durable Protocols Queues Topics Message Processing Most Protocols Are Asynchronous HTTP Is Usually Synchronous Data Synchronization Within an Organization Pivots Multiple Subscribers Responses Notifications Between Organizations Async over HTTP Webhooks Emulating REST Occasionally Connected Clients Client-Side Queue Client-Side Bookmark Choosing a Subset Avoiding Redundant Downloads Chapter 12: Generated Behaviors Projections Defining Projections Projection Pipelines Interest Interest in Deleted Entities Interest in Past Periods Sharing Interest Losing Interest Immutable Runtimes Model Generation Query Execution Testing User Interaction Collaboration Immutable Organizations Decision Substrate Globally Distributed Systems Index This book teaches you how to evaluate a distributed system from the perspective of immutable objects. You will understand the problems in existing designs, know how to make small modifications to correct those problems, and learn to apply the principles of immutable architecture to your tools. Most software components focus on the state of objects. They store the current state of a row in a relational database. They track changes to state over time, making several basic assumptions: there is a single latest version of each object, the state of an object changes sequentially, and a system of record exists. This is a challenge when it comes to building distributed systems. Whether dealing with autonomous microservices or disconnected mobile apps, many of the problems we try to solve come down to synchronizing an ever-changing state between isolated components. Distributed systems would be a lot easier to build if objects could not change. After reading The Art of Immutable Architecture, you will come away with an understanding of the benefits of using immutable objects in your own distributed systems. You will learn a set of rules for identifying and exchanging immutable objects, and see a collection of useful theorems that emerges and ensures that the distributed systems we build are eventually consistent. Using patterns, you will find where the truth converges, see how changes are associative, rather than sequential, and come to feel comfortable understanding that there is no longer a single source of truth. Practical hands-on examples reinforce how to build software using the described patterns, techniques, and tools. By the end, you will possess the language and resources needed to analyze and construct distributed systems with confidence. The assumptions of the past were sufficient for building single-user, single-computer systems. But as we expand to multiple devices, shared experiences, and cloud computing, they work against us. It is time for a new set of assu mptions. Start with immutable objects, and build better distributed systems. What You Will Learn: Evaluate a distributed system from the perspective of immutable objects Recognize the problems in existing designs, and make small modifications to correct them Start a new system from scratch, applying patterns Apply the principles of immutable architecture to your tools, including SQL databases, message queues, and the network protocols that you already use Discover new tools that natively apply these principles This book is for software architects and senior developers. It contains examples in SQL and languages such as JavaScript and C#. Past experience with distributed computing, data modeling, or business analysis is helpful. Michael L. Perry has built upon the works of mathematicians such as Bertrand Meyer, Leslie Lamport, and Donald Knuth to develop a mathematical system for software development. He has captured this system in a set of open source projects. Michael often presents on math and software at events and online. You can find out more at qedcode.com This book teaches you how to evaluate a distributed system from the perspective of immutable objects. You will understand the problems in existing designs, know how to make small modifications to correct those problems, and learn to apply the principles of immutable architecture to your tools. Most software components focus on the state of objects. They store the current state of a row in a relational database. They track changes to state over time, making several basic assumptions: there is a single latest version of each object, the state of an object changes sequentially, and a system of record exists. This is a challenge when it comes to building distributed systems. Whether dealing with autonomous microservices or disconnected mobile apps, many of the problems we try to solve come down to synchronizing an ever-changing state between isolated components. Distributed systems would be a lot easier to build if objects could not change. After reading The Art of Immutable Architecture , you will come away with an understanding of the benefits of using immutable objects in your own distributed systems. You will learn a set of rules for identifying and exchanging immutable objects, and see a collection of useful theorems that emerges and ensures that the distributed systems we build are eventually consistent. Using patterns, you will find where the truth converges, see how changes are associative, rather than sequential, and come to feel comfortable understanding that there is no longer a single source of truth. Practical hands-on examples reinforce how to build software using the described patterns, techniques, and tools. By the end, you will possess the language and resources needed to analyze and construct distributed systems with confidence. The assumptions of the past were sufficient for building single-user, single-computer systems. But as we expand to multiple devices, shared experiences, and cloud computing, they work against us. It is time for a new set of assumptions. Start with immutable objects, and build better distributed systems. What You Will Learn Evaluate a distributed system from the perspective of immutable objects Recognize the problems in existing designs, and make small modifications to correct them Start a new system from scratch, applying patterns Apply the principles of immutable architecture to your tools, including SQL databases, message queues, and the network protocols that you already use Discover new tools that natively apply these principles Who This Book Is For Software architects and senior developers. It contains examples in SQL and languages such as JavaScript and C#. Past experience with distributed computing, data modeling, or business analysis is helpful.