Spring Cloud Data Flow : Native Cloud Orchestration Services for Microservice Applications on Modern Runtimes

Work with big data applications by using Spring Cloud Data Flow as a unified, distributed, and extensible system for data ingestion and integration, real-time analytics and data processing pipelines, batch processing, and data export. With this book you will develop a foundation for creating applications that use real-time data streaming by combining different technologies and use the full power of Spring Cloud Data Flow. The first part of Spring Cloud Data Flow introduces the concepts you will need in the rest of the book. It begins with an overview of the cloud, microservices, and big data, before moving on to the Spring projects essential to modern big data applications in Java: Spring Integration, Spring Batch, Spring Cloud Stream, and Spring Cloud Task. The second part of the book covers the internals of Spring Cloud Data Flow, giving you the insights and knowledge required to build the applications you need. You'll learn how to use Spring Data Flow's DSL and how to integrate with third-party cloud platform solutions, such as Kubernetes. Finally, the book covers Spring Cloud Data Flow applications to impart practical, useful skills for real-world applications of the technologies covered throughout the rest of the book. What You Will Learn See the Spring Cloud Data Flow internals Create your own Binder using NATs as Broker Mater Spring Cloud Data Flow architecture, data processing, and DSL Integrate Spring Cloud Data Flow with Kubernetes Use Spring Cloud Data Flow local server, Docker Compose, and Kubernetes Discover the Spring Cloud Data Flow applications and how to use them Work with source, processor, sink, tasks, Spring Flo and its GUI, and analytics via the new Micrometer stack for realtime visibility with Prometheus and Grafana Who This Book Is For Those with some experience with the Spring Framework, Microservices and Cloud Native Applications. Java experience is recommended. Table of Contents About the Author About the Technical Reviewer Acknowledgments Part I: Introductions Chapter 1: Cloud and Big Data A Little Data Science The Cloud Cloud Technology and Infrastructure The Right Tools Summary Chapter 2: Spring Boot What Is Spring Framework and What Is Spring Boot? Spring Framework A Directory Application Spring Boot A Directory App with Spring Boot Beyond the Directory App Example Spring Boot Features Summary Chapter 3: Spring Integration Integration Patterns Messaging Spring Integration Movie App Specification Requirements Movie App Part I Movie App: Part II (External) Creating the Movie App: Part I Movie Application: Declarative XML Spring Integration Through Declarative XML Spring Integration Development Creating a Movie App: Part II (External) Movie App Part II with Spring Boot MVC Spring Integration Through Declarative XML: Part II Spring Integration Development: Part II Summary Chapter 4: Spring Batch Spring Batch Programming Model Spring Batch Features Movie Batch App Using Declarative XML Movie Batch App Using JavaConfig Summary Chapter 5: Spring Cloud Microservices The Twelve-Factor App Spring Cloud Spring Cloud Config Spring Cloud Config Server Cloud Config Client Movie App Movie-Web App Running Config Server, Movie, and Movie Web Microservices Changing the Logging Level Spring Cloud Netflix Service Discovery: Eureka Service Registry Eureka Server Eureka Client Service Discoverable: Movie Web Microservice Discovering Services: Movie Microservice Ribbon: Client-Side Load Balancing Movie Microservice Running All Together Circuit Breaker Hystrix: Movie Microservice About Reactive Programming Summary Chapter 6: Spring Cloud Stream Spring Cloud Stream Application Starters HTTP Source | Log-Sink Example Using Uber-Jars Using Docker Using Docker Compose Spring Cloud Stream Movie Cloud Stream Using Kafka: movie-sink-kafka Movie Cloud Stream Using RabbitMQ: movie-sink-rabbit Spring Cloud Stream Spring Cloud Stream Features Programming Model Annotation-Based Bindings Version 2.x and Below File | Transform | Log (Rabbit) Example movie-file-source-rabbit movie-transform-processor-rabbit movie-log-sink-rabbit Running All of Them Together More About Bindings Channel/Bindings Naming Conventions Bindings Version 3.x movie-source-kafka movie-processor-kafka movie-sink-kafka Running All of Them Together Binding Naming Convention Producing and Consuming @StreamListener Features More Features Mapping Method Arguments Reactive and Functions Routing Summary Chapter 7: Spring Cloud Stream Binders Binder Implementing a Binder NATS Binder Project: nats-binder NATS Client: nats-messaging NATS Binder Implementation: nats-messaging-binder Implement the Binder Interface Create the @Configuration Beans Create the META-INF/spring.binders NATS Binder Test Multiple Binders movie-file-source-nats movie-filter-processor-nats-rabbit movie-log-sink-rabbit Running Them All Together Extra Configuration Summary Chapter 8: Spring Cloud Data Flow: Introduction and Installation Spring Cloud Data Flow Features Local Installation Single Machine/Server Using RabbitMQ as a Binder and MySQL for Persistence Using Kafka as a Binder and MySQL for Persistence Using Spring Boot Config Features Spring Cloud Data Flow Dashboard Registering Cloud Stream Applications Starters Separated Server or a Proxy Using Docker Compose Kubernetes Installation Personal Computer with Docker Desktop Minikube Installing Spring Cloud Data Flow in Kubernetes Using kubectl Testing Your Installation with a Simple Stream Using Docker Compose Using Kubernetes Using the Spring Cloud Data Flow Shell Cleaning Up Summary Part II: Spring Cloud Data Flow: Internals Chapter 9: Spring Cloud Data Flow Internals Spring Cloud Data Flow Architecture Client Tools: cURL Command, Data Flow Shell, Java Client, Dashboard Using cURL, Httpie, and jq Using Spring Cloud Data Flow Shell Using the Dashboard Creating the Stream Programmatically Java DSL API: Definition Style Java DSL API: Fluent Style Summary Chapter 10: Custom Stream Apps with Spring Cloud Data Flow Custom Stream Apps: A Quick Review Custom Stream Apps in Spring Cloud Data Flow Movie Web App: movie-source Movie IMDB App: movie-processor Movie Log App: movie-sink Packaging and Deploy Stream Apps Registering Stream Apps Using the Spring Cloud Data Flow Shell to Register Custom Apps Using the Dashboard to Register Custom Apps Create and Deploy Custom Streams Summary Chapter 11: Task and Batch Apps with Spring Cloud Data Flow Spring Cloud Task Primer Simple Task Demo Spring Cloud Stream Integration Task Events Within Spring Cloud Stream Launching Tasks in Spring Cloud Data Flow Image to Dropbox Task Using the Dashboard Launching a Task Using Data Flow Shell Batch Processing Movie Batch Launching a Task/Batch with a Stream in Data Flow Movie Batch Streams Movie Details Using the Dashboard Summary Chapter 12: Monitoring Micrometer Health Checks and Monitoring Stream Apps Monitoring Task Apps Movie Stream Pipeline DSL: Putting It All Together Stream Pipeline DSL Task Creating Stream Deployment Summary Index In Pro Spring Xd, You'll Develop A Foundation For Creating Applications That Use Real-time Data Streaming Starting With Your First Spring Xd Application. Then, You'll Examine The Spring Xd Internals Such As Xd Components Such As Jobs, Taps And Even More On Streams. Additionally, You'll Understand The Spring Xd Architecture, Messaging, And Dsls. Furthermore, While Building Up Your Case Study Application, You'll Learn And Examine Spring Xd's Administration And Monitoring Tools, Development And Deployment Tools, The Spring Xd Rest Apis. Finally, You'll Learn How To Extend And Use The Available Spring Xd Modules And Extensions And Integrate With The Spring Integration Framework For A Most Robust Spring Xd Application. Pro Spring Xd Is Your Authoritative Guide To Using The Spring Xd Platform. This Integral Spring Set Of Tools Lets You Build Applications Or Application Aspects That Take Advantage Of Big Data. Spring Xd Is Essentially A Unified, Distributed, And Extensible System For Data Ingestion, Real Time Analytics, Batch Processing, And Data Export. It Also Lets You Work With Third Party Big Data Processing Engines Like The Very Popular Hadoop And More. What You’ll Learn How To Use Spring Xd And Integrate It With The Spring Platform To Build Complex Data-rich Enterprise Cloud Applications How To Use Spring Xd With The Popular Hadoop And Other Big Data Processing Engines And Frameworks How To Use Spring Xd Components: Streams, Jobs, Taps How To Use Other Spring Xd Internals Like Xd Architecture, Xd Messaging And Xd Dsl How To Use Advanced Techniques Like Spring Xd Administration And Monitoring, Development And Deployment, The Spring Xd Rest Apis How To Extend Spring Xd Using Its Modules And Extensions How To Use Spring Integration With Spring Xd And More Who This Book Is For This Book Is For Experienced Java And Enterprise Java Programmers/developers Who Have At Least Some Prior Experience With Using The Popular Spring Framework And Platform.