Python microservices development : build, test, deploy, and scale microservices in Python

Key Features A very useful guide for Python developers who are shifting to the new microservices-based development A concise, up-to-date guide to building efficient and lightweight microservices in Python using Flask, Tox, and other tools Learn to use Docker containers, CoreOS, and Amazon Web Services to deploy your services Book DescriptionWe often deploy our web applications into the cloud, and our code needs to interact with many third-party services. An efficient way to build applications to do this is through microservices architecture. But, in practice, it's hard to get this right due to the complexity of all the pieces interacting with each other.This book will teach you how to overcome these issues and craft applications that are built as small standard units, using all the proven best practices and avoiding the usual traps. It's a practical book: you'll build everything using Python 3 and its amazing tooling ecosystem. You will understand the principles of TDD and apply them.You will use Flask, Tox, and other tools to build your services using best practices. You will learn how to secure connections between services, and how to script Nginx using Lua to build web application firewall features such as rate limiting. You will also familiarize yourself with Docker's role in microservices, and use Docker containers, CoreOS, and Amazon Web Services to deploy your services.This book will take you on a journey, ending with the creation of a complete Python application based on microservices. By the end of the book, you will be well versed with the fundamentals of building, designing, testing, and deploying your Python microservices.What you will learn Explore what microservices are and how to design them Use Python 3, Flask, Tox, and other tools to build your services using best practices Learn how to use a TDD approach Discover how to document your microservices Configure and package your code in the best way Interact with other services Secure, monitor, and scale your services Deploy your services in Docker containers, CoreOS, and Amazon Web Services About the AuthorTarek Ziadeis a Python developer, located in the countryside near Dijon, France. He works at Mozilla in the services team. He founded a French Python user group called Afpy, and he has written several books about Python in French and English. When he is not hacking on his computer or hanging out with his family, he's spending time between his two passions, running and playing the trumpet.You can visit his personal blog (Fetchez le Python) and follow him on Twitter (@tarek\_ziade). You can also take a look at one of his books on Amazon, Expert Python Programming, published by Packt.Table of Contents Understanding Microservices Working with flask Test-driven development and documentation, the virtuous cycle Designing Forrest Interacting with other services Monitoring your services Securing your services Consuming the Microservices Packaging Forrest Dockerizing your services Deploying on AWS Thinking ahead Preface......Page 19 What this book covers......Page 20 What you need for this book......Page 22 Who this book is for......Page 23 Conventions......Page 24 Reader feedback......Page 26 Customer support......Page 27 Downloading the example code......Page 28 Errata......Page 29 Piracy......Page 30 Questions......Page 31 Understanding Microservices......Page 32 Origins of Service-Oriented Architecture......Page 34 The monolithic approach......Page 36 The microservice approach......Page 41 Microservice benefits......Page 44 Separation of concerns......Page 45 Smaller projects......Page 46 Scaling and deployment......Page 47 Microservices pitfalls......Page 48 Illogical splitting......Page 49 More network interactions......Page 50 Data storing and sharing......Page 51 Compatibility issues......Page 52 Testing......Page 53 Implementing microservices with Python......Page 54 The WSGI standard......Page 56 Greenlet and Gevent......Page 59 Twisted and Tornado......Page 62 asyncio......Page 64 Language performances......Page 67 Summary......Page 71 Discovering Flask......Page 72 Which Python?......Page 74 How Flask handles requests......Page 75 Routing......Page 81 Variables and converters......Page 84 The url_for function......Page 89 Request......Page 90 Response......Page 92 Flask built-in features......Page 95 The session object......Page 96 Globals......Page 97 Signals......Page 99 Extensions and middlewares......Page 101 Templates......Page 104 Configuration......Page 107 Blueprints......Page 110 Error handling and debugging......Page 112 Custom error handler......Page 113 The debug mode......Page 117 A microservice skeleton......Page 119 Summary......Page 122 Coding, Testing, and Documenting - the Virtuous Cycle......Page 123 Different kinds of tests......Page 126 Unit tests......Page 127 Functional tests......Page 132 Integration tests......Page 136 Load tests......Page 138 End-to-end tests......Page 142 Using WebTest......Page 144 Using pytest and Tox......Page 147 Developer documentation......Page 152 Continuous Integration......Page 159 Travis-CI......Page 160 ReadTheDocs......Page 162 Coveralls......Page 163 Summary......Page 166 Designing Runnerly......Page 167 The Runnerly application......Page 168 User stories......Page 169 Monolithic design......Page 171 Model......Page 172 View and Template......Page 174 Background tasks......Page 179 Strava token......Page 184 Authentication and authorization......Page 186 Putting together the monolithic design......Page 191 Splitting the monolith......Page 193 Data Service......Page 195 Using Open API 2.0......Page 197 More splitting......Page 201 Summary......Page 203 Interacting with Other Services......Page 204 Synchronous calls......Page 206 Using Session in a Flask app......Page 208 Connection pooling......Page 214 HTTP cache headers......Page 216 Improving data transfer......Page 222 GZIP compression......Page 223 Binary payloads......Page 226 Putting it together......Page 231 Asynchronous calls......Page 232 Task queues......Page 233 Topic queues......Page 234 Publish/subscribe......Page 242 RPC over AMQP......Page 243 Putting it together......Page 244 Testing......Page 245 Mocking synchronous calls......Page 246 Mocking asynchronous calls......Page 248 Mocking Celery......Page 249 Mocking other asynchronous calls......Page 252 Summary......Page 253 Monitoring Your Services......Page 254 Centralizing logs......Page 255 Setting up Graylog......Page 257 Sending logs to Graylog......Page 261 Adding extra fields......Page 264 Performance metrics......Page 266 System metrics......Page 267 Code metrics......Page 269 Web server metrics......Page 272 Summary......Page 274 Securing Your Services......Page 275 The OAuth2 protocol......Page 277 Token-based authentication......Page 279 The JWT standard......Page 280 PyJWT......Page 283 X.509 certificate-based authentication......Page 285 The TokenDealer microservice......Page 288 The POST/oauth/token implementation......Page 290 Using TokenDealer......Page 293 Web application firewall......Page 296 OpenResty - Lua and nginx......Page 299 Rate and concurrency limiting......Page 303 Other OpenResty features......Page 306 Securing your code......Page 307 Asserting incoming data......Page 308 Limiting your application scope......Page 312 Using Bandit linter......Page 314 Bringing It All Together......Page 318 Building a ReactJS dashboard......Page 320 The JSX syntax......Page 321 React components......Page 323 ReactJS and Flask......Page 329 Using Bower, npm, and Babel......Page 331 Cross-origin resource sharing......Page 337 Authentication and authorization......Page 342 Interacting with Data Service......Page 343 Getting the Strava token......Page 345 JavaScript authentication......Page 348 Packaging and Running Runnerly......Page 352 The packaging toolchain......Page 354 A few definitions......Page 356 Packaging......Page 357 The setup.py file......Page 358 The requirements.txt file......Page 365 The MANIFEST.in file......Page 370 Versioning......Page 372 Releasing......Page 375 Distributing......Page 379 Running all microservices......Page 383 Process management......Page 388 Summary......Page 393 Containerized Services......Page 394 What is Docker?......Page 396 Docker 101......Page 399 Running Flask in Docker......Page 403 The full stack - OpenResty, Circus and Flask......Page 406 OpenResty......Page 408 Circus......Page 412 Docker-based deployments......Page 418 Docker Compose......Page 421 Introduction to Clustering and Provisioning......Page 425 Summary......Page 428 Deploying on AWS......Page 429 AWS overview......Page 431 Routing - Route53, ELB, and AutoScaling......Page 433 Execution - EC2 and Lambda......Page 434 Storage - EBS, S3, RDS, ElasticCache, and CloudFront......Page 436 Messaging - SES, SQS, and SNS......Page 438 Simple Email Service (SES)......Page 439 Simple Queue Service (SQS)......Page 440 Simple Notification Service (SNS)......Page 441 Provisioning and deployment - CloudFormation and ECS......Page 442 Deploying on AWS - the basics......Page 443 Setting up your AWS account......Page 444 Deploying on EC2 with CoreOS......Page 447 Deploying with ECS......Page 454 Route53......Page 460 Summary......Page 462 What Next?......Page 463 Iterators and generators......Page 465 Coroutines......Page 469 The asyncio library......Page 471 The aiohttp framework......Page 473 Sanic......Page 474 Asynchronous versus synchronous......Page 476 A practical approach to conquering the complexities of Microservices using the Python tooling ecosystemAbout This Book* A very useful guide for Python developers who are shifting to the new microservices-based development* A concise, up-to-date guide to building efficient and lightweight microservices in Python using Flask, Tox, and other tools* Learn to use Docker containers, CoreOS, and Amazon Web Services to deploy your servicesWho This Book Is ForThis book is for developers who have basic knowledge of Python, the command line, and HTTP-based application principles, and those who want to learn how to build, test, scale, and manage Python 3 microservices. No prior experience of writing microservices in Python is assumed. What You Will Learn* Explore what microservices are and how to design them* Use Python 3, Flask, Tox, and other tools to build your services using best practices* Learn how to use a TDD approach* Discover how to document your microservices* Configure and package your code in the best way* Interact with other services* Secure, monitor, and scale your services* Deploy your services in Docker containers, CoreOS, and Amazon Web ServicesIn DetailWe often deploy our web applications into the cloud, and our code needs to interact with many third-party services. An efficient way to build applications to do this is through microservices architecture. But, in practice, it's hard to get this right due to the complexity of all the pieces interacting with each other. This book will teach you how to overcome these issues and craft applications that are built as small standard units, using all the proven best practices and avoiding the usual traps. It's a practical book: you'll build everything using Python 3 and its amazing tooling ecosystem. You will understand the principles of TDD and apply them. You will use Flask, Tox, and other tools to build your services using best practices. You will learn how to secure connections between services, and how to script Nginx using Lua to build web application firewall features such as rate limiting. You will also familiarize yourself with Docker's role in microservices, and use Docker containers, CoreOS, and Amazon Web Services to deploy your services. This book will take you on a journey, ending with the creation of a complete Python application based on microservices. By the end of the book, you will be well versed with the fundamentals of building, designing, testing, and deploying your Python microservices. Style and approachThis book is an linear, easy-to-follow guide on how to best design, write, test, and deploy your microservices. It includes real-world examples that will help Python developers create their own Python microservice using the most efficient methods A practical approach to conquering the complexities of Microservices using the Python tooling ecosystem About This Book A very useful guide for Python developers who are shifting to the new microservices-based development A concise, up-to-date guide to building efficient and lightweight microservices in Python using Flask, Tox, and other tools Learn to use Docker containers, CoreOS, and Amazon Web Services to deploy your services Who This Book Is For This book is for developers who have basic knowledge of Python, the command line, and HTTP-based application principles, and those who want to learn how to build, test, scale, and manage Python 3 microservices. No prior experience of writing microservices in Python is assumed. What You Will Learn Explore what microservices are and how to design them Use Python 3, Flask, Tox, and other tools to build your services using best practices Learn how to use a TDD approach Discover how to document your microservices Configure and package your code in the best way Interact with other services Secure, monitor, and scale your services Deploy your services in Docker containers, CoreOS, and Amazon Web Services In Detail We often deploy our web applications into the cloud, and our code needs to interact with many third-party services. An efficient way to build applications to do this is through microservices architecture. But, in practice, it's hard to get this right due to the complexity of all the pieces interacting with each other. This book will teach you how to overcome these issues and craft applications that are built as small standard units, using all the proven best practices and avoiding the usual traps. It's a practical book: you'll build everything using Python 3 and its amazing tooling ecosystem. You will understand the principles of TDD and apply them. You will use Flask, Tox, and other tools to build your services using best practices. You will learn how to secure connections between services, and how to script Nginx using Lua to build web application firewall features such as rate limiting. You will also familiarize yourself with Docker's role in microservices, and use Docker containers, CoreOS, and Amazon Web Services to deploy your services. This book will take you on a journey, ending with the creation of a complete Python application based on microservices. By the end of the book, you will be well versed with the fundamentals of building, designing, testing, and deploying your Python microse..