The Art of Immutable Architecture: Theory and Practice of Data Management in Distributed Systems, Second Edition

This book teaches you how to evaluate a distributed system from the perspective of immutable objects. You will understand the problems in existing designs, know how to make small modifications to correct those problems, and learn to apply the principles of immutable architecture to your tools. Most software components focus on the state of objects. They store the current state of a row in a relational database. They track changes to state over time, making several basic assumptions: there is a single latest version of each object, the state of an object changes sequentially, and a system of record exists. This is a challenge when it comes to building distributed systems. Whether dealing with autonomous microservices or disconnected mobile apps, many of the problems we try to solve come down to synchronizing an ever-changing state between isolated components. Distributed systems would be a lot easier to build if objects could not change. After reading The Art of Immutable Architecture, you will come away with an understanding of the benefits of using immutable objects in your own distributed systems. You will learn a set of rules for identifying and exchanging immutable objects, and see a collection of useful theorems that emerges and ensures that the distributed systems you build are eventually consistent. Using patterns, you will find where the truth converges, see how changes are associative, rather than sequential, and come to feel comfortable understanding that there is no longer a single source of truth. Practical hands-on examples reinforce how to build software using the described patterns, techniques, and tools. By the end of the book, you will possess the language and resources needed to analyze and construct distributed systems with confidence. The assumptions of the past were sufficient for building single-user, single-computer systems. But aswe expand to multiple devices, shared experiences, and cloud computing, they work against us. It is time for a new Table of Contents About the Author Acknowledgments Introduction Part I: Definition Chapter 1: Why Immutable Architecture The Immutability Solution The Problems with Immutability Redefine the Process The Fallacies of Distributed Computing The Network Is Not Reliable Latency Is Not Zero Topology Changes Changing Assumptions Immutability Changes Everything Shared Mutable State Persistent Data Structures The Two Generals’ Problem A Prearranged Protocol Reducing the Uncertainty An Additional Message Proof of Impossibility Relaxing Constraints Redefining the Problem Decide and Act Accept the Truth A Valid Protocol Examples of Immutable Architectures Git Blockchains Docker Chapter 2: Forms of Immutable Architecture Deriving State from History Historical Records Building Upon the Past Evolution of Understanding Mutable Objects Identity Evolution of State Projections Two Kinds of State Projecting Objects Event Sourcing Generating Events CQRS DDD Taking a Functional View Commutative and Idempotent Events Model View Update The Update Loop Unidirectional Data Flow Immutable App Architecture Historical Modeling Partial Order Predecessors Successors Immutable Graphs Collaboration Acyclic Graphs Timeliness Limitations of Historical Modeling No Central Authority No Real-Time Clock No Uniqueness Constraints No Aggregation Chapter 3: How to Read a Historical Model Fact Type Graphs A Chess Game Important Attributes A Chain of Facts Endgame Fact Instance Graphs The Immortal Game Collecting Moves A Brilliant Win The Factual Modeling Language Declaring Fact Types Querying the Model Changing Direction Jumping Levels Common Ancestors Multiple Unknowns Existential Conditions Projections Nested Specifications Factual in Immutable Runtimes Historical Modeling in Analysis Part II: Application Chapter 4: Analysis Historical Modeling Workshop Which Came First Complete the Process Validate Assumptions Data Identifiers Cardinality Mutation Curren t State Views Finding a Place to Start Annotated Wireframes Removal from Lists Collaboration Regions Crossing Boundaries Conversations Publishing Facts Integrating Subsystems Valid Orderings Eliminating Race Conditions Responding to Different Valid Orderings Consequences Indexes Uniqueness Constraints Navigation Searching Expected Number of Results No Implicit Order Aggregates Iterations Creation Order Chapter 5: Location Independence Modeling with Immutability Synchronization Guarantees Identity Auto-incremented IDs Environment Dependence Parent-Child Insertion Remote Creation URLs Location-Independent Identity Natural Keys GUIDs Timestamps Tuples Hashes Public Keys Random Numbers Causality Putting Steps in Order The Transitive Property Concurrency Partial Order The CAP Theorem Defining CAP Proving the CAP Theorem Test an Algorithm Eventual Consistency Kinds of Consistency Strong Eventual Consistency in a Relay-Based System Idempotence and Commutativity Deriving Strong Eventual Consistency The Contact Management System Replaying History Conflict-Free Replicated Data Types (CRDTs) State-Based CRDTs Partially Ordered State Causal History Vector Clocks A History of Facts Sets Partial Order Update Merge Historical Records Distinguishing Between Records Removing a Record Changing a Record Records Are Causally Related Benefits of Explicit Causality Accepting Constraints Historical Facts Conclusion Chapter 6: Immutable Runtimes When Architecture Depends Upon the Domain Replicators Scaling Up Traditional Infrastructure Scaling Up an Immutable Runtime Redundant Storage Legacy Integration Persistent Projections Data Firewalls Specifications Execution Inversion Communication Security Authorization Rules Distribution Rules Versioning Incremental Addition Structural Versioning One-Way Transformation Archiving Jinaga Chapter 7: Patterns Structural Patterns Entity Structure Example Consequences Related Patterns Ownership Structure Example Consequences Related Patterns Delete Structure Example Consequences Related Patterns Restore Structure Example Consequences Related Patterns Membership Structure Example Consequences Related Patterns Mutable Property Structure Example Consequences Related Patterns Entity Reference Structure Example Consequences Related Patterns Entity List Structure Example Consequences Related Patterns Application Patterns Personal Collection Structure Example Consequences Related Patterns Social Network Structure Example Consequences Related Patterns Shared Project Structure Consequences Related Patterns Enterprise Domain Structure Example Consequences Related Patterns Designing from Constraints Chapter 8: State Transitions Many Properties Shipping and Billing Introducing Back-Orders Cancellations and Returns Parallel State Machines Many Children Software Issue Tracking Child State Composite State Transition Diagrams A Declarative Function of States Conditional Validation Nullability Based on State Cycles in State Transition Collect Data During Transitions Immutable State Transitions The Question Behind State Translating a State Machine to a Historical Model Order Fulfillment Software Change Tracking Reasons for Computing State Handling the Next Action Finding Work Items Executing Compensating Transactions Single Source of Truth Orchestrators Consistent State Central Validation Convergent Histories Define Immutable Records Query for the Next Action Capture Actions Locally Define Compensating Actions Workflow Patterns Transaction Structure Example Consequences Related Patterns Queue Structure Example Consequences Related Patterns Period Structure Example Consequences Related Patterns Chapter 9: Security Proof of Authorship Key Pairs Digest Authorization Principal Facts Authorization Rules Authorization Query Initial Authorization Grant of Authority Limited Authority Indefinite Authorization Transitive Authorization Revocation Authorization Upon Receipt Confidentiality Untrusted Replicators Asymmetric Encryption Asymmetric Size Limit Encrypt the Symmetric Key Encrypting Historical Facts Limit the Distribution of Confidential Facts Distribution Rules Evidence Attacks and Countermeasures Secrecy Shared Symmetric Key A Secret Discussion Channel Creating a Secret Channel Team Distribution Rules Limit the Scope of a Shared Key Cohorts Periods Part III: Implementation Chapter 10: SQL Databases Identity Content-Addressed Storage Advantages Avoid Hashes As Primary Keys Table Structure Relationships Inserting Successors Optional Predecessors Many Predecessors Canonical Hash of a Set Inserting Many Predecessors Queries From Specification to Pipeline A Set of Unknowns A Sequence of Steps Filter by Existential Condition Child Specifications From Pipeline to SQL Given Facts Joins Correlated Subqueries Optimization Spurious Joins Covering Indexes WHERE NOT EXISTS Mutable Properties Deletion Queues Integration Legacy Application Integration Scanners Triggers Change Data Capture Reporting Databases Chapter 11: Communication Delivery Guarantees Best Effort Confirmation Safe Methods Idempotent Methods Non-idempotent Methods Retry Within a Connection Durable Protocols Queues Topics Message Processing Most Protocols Are Asynchronous HTTP Is Usually Synchronous Data Synchronization Within an Organization Pivots Multiple Subscribers Responses Notifications Between Organizations Async over HTTP Webhooks Emulating REST Occasionally Connected Clients Client-Side Queue Client-Side Bookmark Choosing a Subset Avoiding Redundant Downloads Outbox Structure Journaling Random Processing Delays Rendezvous Hashing Service Failure Example Consequences Related Patterns Chapter 12: Feeds Interest Tuples Labels Transitive Closure Generating Feeds Positive Existential Conditions Negative Existential Conditions Nested Negative Existential Conditions Projections Unused Givens Bookmarks Location-Specific Fact ID Adding Tuples to a Feed Vectors Security Losing Interest Interest in Deleted Entities Interest in Past Periods Purging Facts Implementations Chapter 13: Inversion Mechanizing the Problem The Affected Set Computing the Affected Set Increasing the Complexity Targeted Updates New Results Removed Results Modified Results Computing Inverses Tuples Rewriting Specifications Reorder the Graph Positive Existential Conditions Negative Existential Conditions Nested Existential Conditions Child Specifications Proof of Completeness Consequences of Inversion Real-Time Notification API Isolation Low-Latency Projections Collaboration Index