Python high performance : build robust application by implementing concurrent and distributed processing techniques

Cover -- Copyright -- Credits -- About the Author -- About the Reviewer -- www.PacktPub.com -- Customer Feedback -- Table of Contents -- Preface -- Chapter 1: Benchmarking and Profiling -- Designing your application -- Writing tests and benchmarks -- Timing your benchmark -- Better tests and benchmarks with pytest-benchmark -- Finding bottlenecks with cProfile -- Profile line by line with line\_profiler -- Optimizing Read more... Abstract: Cover -- Copyright -- Credits -- About the Author -- About the Reviewer -- www.PacktPub.com -- Customer Feedback -- Table of Contents -- Preface -- Chapter 1: Benchmarking and Profiling -- Designing your application -- Writing tests and benchmarks -- Timing your benchmark -- Better tests and benchmarks with pytest-benchmark -- Finding bottlenecks with cProfile -- Profile line by line with line\_profiler -- Optimizing our code -- The dis module -- Profiling memory usage with memory\_profiler -- Summary -- Chapter 2: Pure Python Optimizations -- Useful algorithms and data structures -- Lists and deques -- Dictionaries -- Building an in-memory search index using a hash map -- Sets -- Heaps -- Tries -- Caching and memoization -- Joblib -- Comprehensions and generators -- Summary -- Chapter 3: Fast Array Operations with NumPy and Pandas -- Getting started with NumPy -- Creating arrays -- Accessing arrays -- Broadcasting -- Mathematical operations -- Calculating the norm -- Rewriting the particle simulator in NumPy -- Reaching optimal performance with numexpr -- Pandas -- Pandas fundamentals -- Indexing Series and DataFrame objects -- Database-style operations with Pandas -- Mapping -- Grouping, aggregations, and transforms -- Joining -- Summary -- Chapter 4: C Performance with Cython -- Compiling Cython extensions -- Adding static types -- Variables -- Functions -- Classes -- Sharing declarations -- Working with arrays -- C arrays and pointers -- NumPy arrays -- Typed memoryviews -- Particle simulator in Cython -- Profiling Cython -- Using Cython with Jupyter -- Summary -- Chapter 5: Exploring Compilers -- Numba -- First steps with Numba -- Type specializations -- Object mode versus native mode -- Numba and NumPy -- Universal functions with Numba -- Generalized universal functions -- JIT classes -- Limitations in Numba -- The PyPy project Cover......Page 1 Copyright......Page 2 Credits......Page 5 About the Author......Page 6 About the Reviewer......Page 7 www.PacktPub.com......Page 8 Customer Feedback......Page 9 Table of Contents......Page 10 Preface......Page 14 Chapter 1: Benchmarking and Profiling......Page 20 Designing your application......Page 21 Writing tests and benchmarks......Page 27 Timing your benchmark......Page 29 Better tests and benchmarks with pytest-benchmark......Page 32 Finding bottlenecks with cProfile......Page 35 Profile line by line with line_profiler......Page 40 Optimizing our code......Page 42 The dis module......Page 44 Profiling memory usage with memory_profiler......Page 45 Summary......Page 48 Chapter 2: Pure Python Optimizations......Page 49 Useful algorithms and data structures......Page 50 Lists and deques......Page 51 Dictionaries......Page 53 Building an in-memory search index using a hash map......Page 55 Sets......Page 57 Heaps......Page 58 Tries......Page 60 Caching and memoization......Page 62 Joblib......Page 65 Comprehensions and generators......Page 66 Summary......Page 68 Chapter 3: Fast Array Operations with NumPy and Pandas......Page 69 Creating arrays......Page 70 Accessing arrays......Page 72 Broadcasting......Page 77 Mathematical operations......Page 80 Rewriting the particle simulator in NumPy......Page 81 Reaching optimal performance with numexpr......Page 85 Pandas fundamentals......Page 87 Indexing Series and DataFrame objects......Page 89 Database-style operations with Pandas......Page 91 Mapping......Page 92 Grouping, aggregations, and transforms......Page 94 Joining......Page 96 Summary......Page 97 Compiling Cython extensions......Page 99 Variables......Page 102 Functions......Page 104 Classes......Page 105 Sharing declarations......Page 107 C arrays and pointers......Page 108 NumPy arrays......Page 111 Typed memoryviews......Page 112 Particle simulator in Cython......Page 114 Profiling Cython......Page 118 Using Cython with Jupyter......Page 122 Summary......Page 125 Chapter 5: Exploring Compilers......Page 127 First steps with Numba......Page 128 Type specializations......Page 130 Object mode versus native mode......Page 131 Universal functions with Numba......Page 134 Generalized universal functions......Page 136 JIT classes......Page 139 Limitations in Numba......Page 142 The PyPy project......Page 143 Setting up PyPy......Page 144 Running a particle simulator in PyPy......Page 145 Other interesting projects......Page 146 Summary......Page 147 Chapter 6: Implementing Concurrency......Page 148 Waiting for I/O......Page 149 Concurrency......Page 150 Callbacks......Page 152 Futures......Page 155 Event loops......Page 157 The asyncio framework......Page 160 Coroutines......Page 161 Converting blocking code into non-blocking code......Page 165 Observables......Page 167 Useful operators......Page 170 Hot and cold observables......Page 174 Building a CPU monitor......Page 177 Summary......Page 180 Chapter 7: Parallel Processing......Page 181 Introduction to parallel programming......Page 182 Graphic processing units......Page 184 Using multiple processes......Page 185 The Process and Pool classes......Page 186 The Executor interface......Page 188 Monte Carlo approximation of pi......Page 189 Synchronization and locks......Page 192 Parallel Cython with OpenMP......Page 195 Automatic parallelism......Page 197 Getting started with Theano......Page 198 Profiling Theano......Page 203 Tensorflow......Page 205 Running code on a GPU......Page 207 Summary......Page 211 Introduction to distributed computing......Page 212 An introduction to MapReduce......Page 213 Dask......Page 216 Directed Acyclic Graphs......Page 217 Dask arrays......Page 218 Dask Bag and DataFrame......Page 221 Dask distributed......Page 225 Manual cluster setup......Page 227 Using PySpark......Page 229 Setting up Spark and PySpark......Page 230 Spark architecture......Page 231 Resilient Distributed Datasets......Page 233 Spark DataFrame......Page 238 Scientific computing with mpi4py......Page 240 Summary......Page 243 Choosing a suitable strategy......Page 244 Generic applications......Page 245 Numerical code......Page 246 Big data......Page 248 Organizing your source code......Page 249 Using conda environments......Page 251 Virtualization and Containers......Page 253 Creating docker images......Page 256 Continuous integration......Page 258 Summary......Page 259 Index......Page 261 Index......Page 0 Learn how to use Python to create efficient applications About This Book • Identify the bottlenecks in your applications and solve them using the best profiling techniques • Write efficient numerical code in NumPy, Cython, and Pandas • Adapt your programs to run on multiple processors and machines with parallel programming Who This Book Is For The book is aimed at Python developers who want to improve the performance of their application. Basic knowledge of Python is expected What You Will Learn • Write efficient numerical code with the NumPy and Pandas libraries • Use Cython and Numba to achieve native performance • Find bottlenecks in your Python code using profilers • Write asynchronous code using Asyncio and RxPy • Use Tensorflow and Theano for automatic parallelism in Python • Set up and run distributed algorithms on a cluster using Dask and PySpark In Detail Python is a versatile language that has found applications in many industries. The clean syntax, rich standard library, and vast selection of third-party libraries make Python a wildly popular language. Python High Performance is a practical guide that shows how to leverage the power of both native and third-party Python libraries to build robust applications. The book explains how to use various profilers to find performance bottlenecks and apply the correct algorithm to fix them. The reader will learn how to effectively use NumPy and Cython to speed up numerical code. The book explains concepts of concurrent programming and how to implement robust and responsive applications using Reactive programming. Readers will learn how to write code for parallel architectures using Tensorflow and Theano, and use a cluster of computers for large-scale computations using technologies such as Dask and PySpark. By the end of the book, readers will have learned to achieve performance and scale from their Python applications. Style and approach A step-by-step practical guide filled with real-world use cases and examples