Learning Apache Kafka - Second Edition

Start from scratch and learn how to administer Apache Kafka effectively for messaging About This Book Quickly set up Apache Kafka clusters and start writing message producers and consumers Write custom producers and consumers with message partition techniques Integrate Kafka with Apache Hadoop and Storm for use cases such as processing streaming data Who This Book Is For This book is for readers who want to know more about Apache Kafka at a hands-on level; the key audience is those with software development experience but no prior exposure to Apache Kafka or similar technologies. It is also useful for enterprise application developers and big data enthusiasts who have worked with other publisher-subscriber-based systems and want to explore Apache Kafka as a futuristic solution. In Detail Kafka is one of those systems that is very simple to describe at a high level but has an incredible depth of technical detail when you dig deeper. Learning Apache Kafka Second Edition provides you with step-by-step, practical examples that help you take advantage of the real power of Kafka and handle hundreds of megabytes of messages per second from multiple clients. This book teaches you everything you need to know, right from setting up Kafka clusters to understanding basic blocks like producer, broker, and consumer blocks. Once you are all set up, you will then explore additional settings and configuration changes to achieve ever more complex goals. You will also learn how Kafka is designed internally and what configurations make it more effective. Finally, you will learn how Kafka works with other tools such as Hadoop, Storm, and so on. Learning Apache Kafka Second Edition 10 Credits 12 About the Author 15 About the Reviewers 17 www.PacktPub.com 19 Support files, eBooks, discount offers, and more 21 Why subscribe? 22 Free access for Packt account holders 23 Preface 24 What this book covers 26 What you need for this book 27 Who this book is for 29 Conventions 31 Reader feedback 33 Customer support 35 Errata 37 Piracy 38 Questions 39 1. Introducing Kafka 40 Welcome to the world of Apache Kafka 42 Why do we need Kafka? 44 Kafka use cases 46 Installing Kafka 49 Installing prerequisites 51 Installing Java 1.7 or higher 52 Downloading Kafka 53 Building Kafka 54 Summary 55 2. Setting Up a Kafka Cluster 57 A single node – a single broker cluster 59 Starting the ZooKeeper server 60 Starting the Kafka broker 61 Creating a Kafka topic 62 Starting a producer to send messages 63 Starting a consumer to consume messages 65 A single node – multiple broker clusters 66 Starting ZooKeeper 68 Starting the Kafka broker 69 Creating a Kafka topic using the command line 70 Starting a producer to send messages 71 Starting a consumer to consume messages 72 Multiple nodes – multiple broker clusters 73 The Kafka broker property list 75 Summary 77 3. Kafka Design 79 Kafka design fundamentals 81 Log compaction 84 Message compression in Kafka 86 Replication in Kafka 89 Summary 93 4. Writing Producers 95 The Java producer API 98 Simple Java producers 100 Importing classes 102 Defining properties 103 Building the message and sending it 104 Creating a Java producer with custom partitioning 107 Importing classes 109 Defining properties 110 Implementing the Partitioner class 111 Building the message and sending it 112 The Kafka producer property list 115 Summary 118 5. Writing Consumers 120 Kafka consumer APIs 123 The high-level consumer API 124 The low-level consumer API 127 Simple Java consumers 129 Importing classes 131 Defining properties 132 Reading messages from a topic and printing them 133 Multithreaded Java consumers 137 Importing classes 139 Defining properties 140 Reading the message from threads and printing it 141 The Kafka consumer property list 145 Summary 148 6. Kafka Integrations 150 Kafka integration with Storm 152 Introducing Storm 153 Integrating Storm 156 Kafka integration with Hadoop 159 Introducing Hadoop 161 Integrating Hadoop 164 Hadoop producers 165 Hadoop consumers 167 Summary 169 7. Operationalizing Kafka 171 Kafka administration tools 173 Kafka cluster tools 174 Adding servers 176 Kafka topic tools 178 Kafka cluster mirroring 180 Integration with other tools 183 Summary 185 Index 187

About This Book

• Explore the new features in OpenCart by using them in your custom theme
• Get hands-on with the Global Library Methods used in OpenCart to make themes and extensions
• Packed with hands-on examples, step-by-step explanations, and illustrations to guide you through coding modules and themes

Who This Book Is For

This book is aimed at developers and designers who want to start developing their own themes and extensions and for those who want to know about the code workflow of the OpenCart theme and modules.