دورهٔ هندسه دیفرانسیل

Develop a deep understanding of Kubernetes and other orchestration systems by building your own with Go and the Docker API. Orchestration systems like Kubernetes can seem like a black box: you deploy to the cloud and it magically handles everything you need. That might seem perfect—until something goes wrong and you don’t know how to find and fix your problems. Build an Orchestrator in Go (From Scratch) reveals the inner workings of orchestration frameworks by guiding you through creating your own. In Build an Orchestrator in Go (From Scratch) you will learn how to Identify the components that make up any orchestration system Schedule containers on to worker nodes Start and stop containers using the Docker API Manage a cluster of worker nodes using a simple API Work with algorithms pioneered by Google’s Borg Demystify orchestration systems like Kubernetes and Nomad Build an Orchestrator in Go (From Scratch) explains each stage of creating an orchestrator with diagrams, step-by-step instructions, and detailed Go code samples. Don’t worry if you’re not a Go expert. The book’s code is optimized for simplicity and readability, and its key concepts are easy to implement in any language. You’ll learn the foundational principles of these frameworks, and even how to manage your orchestrator with a command line interface. About the Technology Orchestration frameworks like Kubernetes and Nomad radically simplify managing containerized applications. Building an orchestrator from the ground up gives you deep insight into deploying and scaling containers, clusters, pods, and other components of modern distributed systems. This book guides you step by step as you create your own orchestrator—from scratch. About the Book Build an Orchestrator in Go (From Scratch) gives you an inside-out perspective on orchestration frameworks and the low-level operation of distributed containerized applications. It takes you on a fascinating journey building a simple-but-useful orchestrator using the Docker API and Go SDK. As you go, you’ll get a guru-level understanding of Kubernetes, along with a pattern you can follow when you need to create your own custom orchestration solutions. What's Inside Schedule containers on worker nodes Start and stop containers using the Docker API Manage a cluster of worker nodes using a simple API Work with algorithms pioneered by Google’s Borg inside front cover Build an Orchestrator in Go (From Scratch) Copyright dedication contents Front matter preface acknowledgments about this book Who should read this book How this book is organized: A road map About the code liveBook discussion forum about the author about the cover illustration Part 1 Introduction 1 What is an orchestrator? 1.1 Why implement an orchestrator from scratch? 1.2 The (not so) good ol’ days 1.3 What is a container, and how is it different from a virtual machine? 1.4 What is an orchestrator? 1.5 The components of an orchestration system 1.5.1 The task 1.5.2 The job 1.5.3 The scheduler 1.5.4 The manager 1.5.5 The worker 1.5.6 The cluster 1.5.7 Command-line interface 1.6 Meet Cube 1.7 What tools will we use? 1.8 A word about hardware 1.9 What we won’t be implementing or discussing 1.9.1 Distributed computing 1.9.2 Service discovery 1.9.3 High availability 1.9.4 Load balancing 1.9.5 Security Summary 2 From mental model to skeleton code 2.1 The task skeleton 2.2 The worker skeleton 2.3 The manager skeleton 2.4 The scheduler skeleton 2.5 Other skeletons 2.6 Taking our skeletons for a spin Summary 3 Hanging some flesh on the task skeleton 3.1 Docker: Starting, stopping, and inspecting containers from the command line 3.2 Docker: Starting, stopping, and inspecting containers from the API 3.3 Task configuration 3.4 Starting and stopping tasks Summary Part 2 Worker 4 Workers of the Cube, unite! 4.1 The Cube worker 4.2 Tasks and Docker 4.3 The role of the queue 4.4 The role of the DB 4.5 Counting tasks 4.6 Implementing the worker’s methods 4.6.1 Implementing the StopTask method 4.6.2 Implementing the StartTask method 4.6.3 An interlude on task state 4.6.4 Implementing the RunTask method 4.7 Putting it all together Summary 5 An API for the worker 5.1 Overview of the worker API 5.2 Data format, requests, and responses 5.3 The API struct 5.4 Handling requests 5.5 Serving the API 5.6 Putting it all together Summary 6 Metrics 6.1 What metrics should we collect? 6.2 Metrics available from the /proc filesystem 6.3 Collecting metrics with goprocinfo 6.4 Exposing the metrics on the API 6.5 Putting it all together Summary Part 3 Manager 7 The manager enters the room 7.1 The Cube manager 7.1.1 The components that make up the manager 7.2 The Manager struct 7.3 Implementing the manager’s methods 7.3.1 Implementing the SelectWorker method 7.3.2 Implementing the SendWork method 7.3.3 Implementing the UpdateTasks method 7.3.4 Adding a task to the manager 7.3.5 Creating a manager 7.4 An interlude on failures and resiliency 7.5 Putting it all together Summary 8 An API for the manager 8.1 Overview of the manager API 8.2 Routes 8.3 Data format, requests, and responses 8.4 The API struct 8.5 Handling requests 8.6 Serving the API 8.7 A few refactorings to make our lives easier 8.8 Putting it all together Summary 9 What could possibly go wrong? 9.1 Overview of our new scenario 9.2 Failure scenarios 9.2.1 Application startup failure 9.2.2 Application bugs 9.2.3 Task startup failures due to resource problems 9.2.4 Task failures due to Docker daemon crashes and restarts 9.2.5 Task failures due to machine crashes and restarts 9.2.6 Worker failures 9.2.7 Manager failures 9.3 Recovery options 9.3.1 Recovery from application failures 9.3.2 Recovering from environmental failures 9.3.3 Recovering from task-level failures 9.3.4 Recovering from worker failures 9.3.5 Recovering from manager failures 9.4 Implementing health checks 9.4.1 Inspecting a task on the worker 9.4.2 Implementing task updates on the worker 9.4.3 Healthchecks and restarts 9.5 Putting it all together Summary Part 4 Refactorings 10 Implementing a more sophisticated scheduler 10.1 The scheduling problem 10.2 Scheduling considerations 10.3 Scheduler interface 10.4 Adapting the round-robin scheduler to the scheduler interface 10.5 Using the new scheduler interface 10.5.1 Adding new fields to the Manager struct 10.5.2 Modifying the New helper function 10.6 Did you notice the bug? 10.7 Putting it all together 10.8 The E-PVM scheduler 10.8.1 The theory 10.8.2 In practice 10.9 Completing the Node implementation 10.10 Using the E-PVM scheduler Summary 11 Implementing persistent storage for tasks 11.1 The storage problem 11.2 The Store interface 11.3 Implementing an in-memory store for tasks 11.4 Implementing an in-memory store for task events 11.5 Refactoring the manager to use the new in-memory stores 11.6 Refactoring the worker 11.7 Putting it all together 11.8 Introducing BoltDB 11.9 Implementing a persistent task store 11.10 Implementing a persistent task event store 11.11 Switching out the in-memory stores for permanent ones Summary Part 5 CLI 12 Building a command-line interface 12.1 The core components of CLIs 12.2 Introducing the Cobra framework 12.3 Setting up our Cobra application 12.4 Understanding the new main.go 12.5 Understanding root.go 12.6 Implementing the worker command 12.7 Implementing the manager command 12.8 Implementing the run command 12.9 Implementing the stop command 12.10 Implementing the status command 12.11 Implementing the node command Summary 13 Now what? 13.1 Working on Kubernetes and related tooling 13.2 Manager-worker pattern and workflow systems 13.3 Manager-worker pattern and integration systems 13.4 In closing Appendix. Environment setup A.1 Installing Go A.1.1 Installing on Linux A.2 Project structure and initialization index inside back cover Develop a deep understanding of Kubernetes and other orchestration systems by building your own with Go and the Docker API.Orchestration systems like Kubernetes can seem like a black box: you deploy to the cloud and it magically handles everything you need. That might seem perfect—until something goes wrong and you don't know how to find and fix your problems. Build an Orchestrator in Go (From Scratch) reveals the inner workings of orchestration frameworks by guiding you through creating your own. In Build an Orchestrator in Go (From Scratch) you will learn how to: Identify the components that make up any orchestration system Schedule containers on to worker nodes Start and stop containers using the Docker API Manage a cluster of worker nodes using a simple API Work with algorithms pioneered by Google's Borg Demystify orchestration systems like Kubernetes and Nomad Build an Orchestrator in Go (From Scratch) explains each stage of creating an orchestrator with diagrams, step-by-step instructions, and detailed Go code samples. Don't worry if you're not a Go expert. The book's code is optimized for simplicity and readability, and its key concepts are easy to implement in any language. You'll learn the foundational principles of these frameworks, and even how to manage your orchestrator with a command line interface. About the technology Orchestration frameworks like Kubernetes and Nomad radically simplify managing containerized applications. Building an orchestrator from the ground up gives you deep insight into deploying and scaling containers, clusters, pods, and other components of modern distributed systems. This book guides you step by step as you create your own orchestrator—from scratch. About the book Build an Orchestrator in Go (From Scratch) gives you an inside-out perspective on orchestration frameworks and the low-level operation of distributed containerized applications. It takes you on a fascinating journey building a simple-but-useful orchestrator using the Docker API and Go SDK. As you go, you'll get a guru-level understanding of Kubernetes, along with a pattern you can follow when you need to create your own custom orchestration solutions. What's inside Schedule containers on worker nodes Start and stop containers using the Docker API Manage a cluster of worker nodes using a simple API Work with algorithms pioneered by Google's Borg About the reader For software engineers, operations professionals, and SREs. This book's simple Go code is accessible to all programmers. About the author Tim Boring has 20+ years of experience in software engineering. For most of that time he has worked with orchestration systems, including Borg, Kubernetes, and Nomad. Table of Contents PART 1 INTRODUCTION 1 What is an orchestrator? 2 From mental model to skeleton code 3 Hanging some flesh on the task skeleton PART 2 WORKER 4 Workers of the Cube, unite! 5 An API for the worker 6 Metrics PART 3 MANAGER 7 The manager enters the room 8 An API for the manager 9 What could possibly go wrong? PART 4 REFACTORINGS 10 Implementing a more sophisticated scheduler 11 Implementing persistent storage for tasks PART 5 CLI 12 Building a command-line interface 13 Now what? Understand Kubernetes and other orchestration systems deeply by building your own using Go and the Docker API. In Build an Orchestrator in Go (From Scratch) you will learn how Orchestration systems like Kubernetes coordinate other software subsystems and services to create a complete organized system. Although orchestration tools have a reputation for complexity, theyre designed around few important patterns that apply across many aspects of software development. Build an Orchestrator in Go (From Scratch) reveals the inner workings of orchestration frameworks by guiding you as you design and implement your own using the Go SDK. As you create your own orchestration framework, youll improve your understanding of Kubernetes and its role in distributed system design. Youll also build the skills required to design custom orchestration solutions for those times when an out-of-the-box solution isnt a good fit. Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. About the technology Orchestration systems provide the management framework for software and infrastructure thats distributed across multiple machines and services. By managing the many individual components and containers in a large application, they ensure web apps are resilient and reliable, automatically switching between resources in response to crashes and outages. A properly designed orchestration system can seamlessly scale to handle traffic loads, and reduce time-consuming manual work for sysadmin and site reliability engineers. About the book Build an Orchestrator in Go (From Scratch) teaches you to implement an orchestrator from scratch. Youll discover the components that make up all orchestration systems, and use the Docker API and Go SDK to build layers of functionality from tasks, to workers, to the manager. Learn how to save on costs by maximising the usage of a cluster, or spread tasks among workers to avoid overload and downtime. Once youve built your working system, youll even implement a command line user interface to easily manage your orchestrator. About the reader For software engineers, operations professionals, and SREs who are familiar with Docker and the basics of Go. About the author Tim Boring is a staff engineer at Golioth. He has twenty years of experience in technology organizations ranging from small business to global enterprises. His career spans roles in technical support to site reliability and software engineering. Tim is most interested in the design of software systems and distributed systems in particular.