Cloud-native microservices with Apache Pulsar : build distributed messaging microservices / Rahul Sharma, Mohammad Atyab

Apply different enterprise integration and processing strategies available with Pulsar, Apache's multi-tenant, high-performance, cloud-native messaging and streaming platform. This book is a comprehensive guide that examines using Pulsar Java libraries to build distributed applications with message-driven architecture. You'll begin with an introduction to Apache Pulsar architecture. The first few chapters build a foundation of message-driven architecture. Next, you'll perform a setup of all the required Pulsar components. The book also covers work with Apache Pulsar client library to build producers and consumers for the discussed patterns. You'll then explore the transformation, filter, resiliency, and tracing capabilities available with Pulsar. Moving forward, the book will discuss best practices when building message schemas and demonstrate integration patterns using microservices. Security is an important aspect of any application; the book will cover authentication and authorization in Apache Pulsar such as Transport Layer Security (TLS), OAuth 2.0, and JSON Web Token (JWT). The final chapters will cover Apache Pulsar deployment in Kubernetes. You'll build microservices and serverless components such as AWS Lambda integrated with Apache Pulsar on Kubernetes. After completing the book, you'll be able to comfortably work with the large set of out-of-the-box integration options offered by Apache Pulsar. What You'll Learn Examine the important Apache Pulsar components Build applications using Apache Pulsar client libraries Use Apache Pulsar effectively with microservices Deploy Apache Pulsar to the cloud Who This Book Is For Cloud architects and software developers who build systems in the cloud-native technologies. Table of Contents 4 About the Authors 10 About the Technical Reviewer 11 Acknowledgments 12 Introduction 13 Chapter 1: Introduction to Apache Pulsar 14 Distributed Challenges 14 Availability 15 Scalability 15 Data Consistency 15 Diverse Integration 15 Event-Driven Architecture 16 Innovation 18 Resilience 18 Scalability 19 Extensible 19 Apache Pulsar 20 Unified Messaging Model 20 Efficient Data Exchange 21 Geo-Replication 21 Multi-Tenancy 21 Schema Registry 22 Infinite Storage 22 Cloud Native 22 Pulsar Architecture 22 Brokers 23 Bookies 24 Pulsar Components 25 ZooKeeper 25 Service Discovery 25 Pulsar Proxy 26 Pulsar Setup 27 Pulsar Command Line 29 Single-Node Cluster 29 Docker 30 Multi-Node Cluster 31 ZooKeeper 31 Bookies 32 Brokers 33 Verify Cluster 34 Pulsar Admin API 35 Summary 35 Chapter 2: Working with Messages 36 Types of Messages 36 Message Channel 37 Breaking up of Messages 38 Messaging Patterns 38 Point-2-Point 38 Pub/Sub 39 Message Filtering 40 Durable Subscriptions 41 Non-Durable Subscriptions 41 Pulsar Client Libraries 42 Client Setup Phase 42 Creating a Producer 43 Creating Consumers 45 Publish Messages 47 Messaging in Pulsar 48 Message Topic Format 49 Subscriptions 51 Shared 51 Key_Shared 53 Exclusive 54 Failover 55 Receiver Queues 57 Sequencing Using Acknowledgments 57 Message Retention 59 Time-to-Live (TTL) 60 Retention Policy 60 Backlog Quota 60 Batching and Chunking 61 Batching 62 Batch Index Acknowledgment 62 Batch Receiving in Consumers 63 Chunking 64 Handle Chunked Messages with One Producer and One Ordered Consumer 65 Handle Chunked Messages with Multiple Producers and One Ordered Consumer 66 Summary 67 Chapter 3: Working with Pulsar Functions 68 Functional Design 68 Pulsar Functions 71 Developing Function 72 Type Support 75 Custom Serialization 77 Pulsar Functions SDK 81 External Configuration 81 Logs 83 State Storage 84 Testing Functions 85 Deploying Functions 86 LocalRunner 87 Programmatic API 88 Cluster Mode 89 Message Delivery Modes 90 At-Most-Once 90 At-Least-Once 91 Effectively-Once 91 Summary 92 Chapter 4: Schema Registry 93 Schema-on-Write vs. Schema-on-Read 93 Schema Registry and Pulsar 94 SchemaInfo 97 Primitive Data Types 98 Complex Data Types 99 Key/Value 100 Key/Value Test Cases 101 Struct 103 Auto Schema 105 Schema Version 105 Schema Evolution and Compatibility 106 Schema Compatibility Check Strategy 106 Managing Schemas 108 Auto Update of Schema: Producer 108 Schema AutoUpdate: Consumer 109 Adjust AutoUpdate Strategy 109 Schema Validation 110 Manual Schema Management 110 Getting a Schema 111 Deleting a Schema 112 Custom Schema Storage 112 Deploying the Classes Created 113 Summary 113 Chapter 5: Building Microservices Using Pulsar 114 Scenario: Portfolio Tracker 115 Enterprise Integration 116 Fan-out 117 Fan-in 121 Router 123 Interservice Communication 124 Resiliency 124 Dead-Letter Queues 124 Retries 127 Throttling 129 Publisher Throttling 130 Subscriber Throttling 131 Dispatch Throttling 132 Backpressure 133 Backlog Quotas 133 Time-to-Live (TTL) 135 Topic Compaction 136 Circuit Breaker 138 Summary 141 Chapter 6: Pulsar Connectors 143 Working with Connectors 143 Processing Guarantees 144 Using Connectors 145 Running Sources 145 Running Sinks 147 Monitoring Connectors 147 Sample Source Connectors 148 File Source Connector 149 MySQL Source Connector 152 Sink Connectors 158 Sample Sink Connector 159 Developing IO Connectors 160 Developing a Source IO Connector 161 Developing a Sink IO Connector 162 Summary 163 Chapter 7: Pulsar Security 164 Quick Overview of TLS 164 TLS Termination 165 Prerequisites 165 Configuring the Broker 168 Configuring the Client 169 Mutual TLS 170 Configuring Mutual TLS in the Client 171 Configuring Mutual TLS in the Broker 172 JSON Web Token 173 Creating a Token 174 Configuring JWT in the Client 175 Configuring JWT in the Broker 176 Authorization 177 Role Grants 177 Configuring Broker for Authorization 178 Configuring the Client for Authorization 178 Summary 179 Chapter 8: Deploying Pulsar on Kubernetes 180 Setup 180 Prerequisites 181 Configure Helm 182 Exploring Pulsar Helm Chart 183 Installation 185 Pulsar Components 187 Pulsar Toolset 187 Pulsar Proxy 188 Pulsar Manager 190 Deployment 191 Pulsar Storage 194 Tiered Storage 197 Configure 198 Offloading 200 Pulsar Geo-replication 201 Pulsar Monitoring 203 pulsar_function_process_latency_ms 204 pulsar_storage_write_latency_le 204 bookie_journal_JOURNAL_SYNC 204 pulsar_source_written_total / pulsar_sink_written_total 204 pulsar_ml_NumberOfMessagesInBacklog 205 pulsar_storage_size 205 pulsar_replication_backlog 205 Monitoring Dashboard 206 Summary 209 Index 210 Apply different enterprise integration and processing strategies available with Pulsar, Apache's multi-tenant, high-performance, cloud-native messaging and streaming platform. This book is a comprehensive guide that examines using Pulsar Java libraries to build distributed applications with message-driven architecture. You'll begin with an introduction to Apache Pulsar architecture. The first few chapters build a foundation of message-driven architecture. Next, you'll perform a setup of all the required Pulsar components. The book also covers work with Apache Pulsar client library to build producers and consumers for the discussed patterns. You'll then explore the transformation, filter, resiliency, and tracing capabilities available with Pulsar. Moving forward, the book will discuss best practices when building message schemas and demonstrate integration patterns using microservices. Security is an important aspect of any application; the book will cover authentication and authorization in Apache Pulsar such as Transport Layer Security (TLS), OAuth 2.0, and JSON Web Token (JWT). The final chapters will cover Apache Pulsar deployment in Kubernetes. You'll build microservices and serverless components such as AWS Lambda integrated with Apache Pulsar on Kubernetes. After completing the book, you'll be able to comfortably work with the large set of out-of-the-box integration options offered by Apache Pulsar. You will: Examine the important Apache Pulsar components Build applications using Apache Pulsar client libraries Use Apache Pulsar effectively with microservices Deploy Apache Pulsar to the cloud.