Advanced Guide to Python 3 Programming, 2nd

Advanced Guide to Python 3 Programming 2nd Edition delves deeply into a host of subjects that you need to understand if you are to develop sophisticated real-world programs. Each topic is preceded by an introduction followed by more advanced topics, along with numerous examples, that take you to an advanced level. This second edition has been significantly updated with two new sections on advanced Python language concepts and data analytics and machine learning. The GUI chapters have been rewritten to use the Tkinter UI library and a chapter on performance monitoring and profiling has been added. In total there are 18 new chapters, and all remaining chapters have been updated for the latest version of Python as well as for any of the libraries they use. There are eleven sections within the book covering Python Language Concepts, Computer Graphics (including GUIs), Games, Testing, File Input and Output, Databases Access, Logging, Concurrency and Parallelism, Reactive Programming, Networking and Data Analytics. Each section is self-contained and can either be read on its own or as part of the book as a whole. It is aimed at those who have learnt the basics of the Python 3 language but wish to delve deeper into Python’s eco system of additional libraries and modules. Preface to the Second Edition Preface to the First Edition What You Need Conventions Example Code and Sample Solutions Contents 1 Introduction 1.1 Introduction 1.2 Useful Python Resources Part I Advanced Language Features 2 Python Type Hints 2.1 Introduction 2.2 Pythons Type System 2.3 The Challenge for Python Developers 2.4 Static Typing 2.5 Python Type Hints 2.6 Type Hint Layout 2.7 Type Hints for Multiple Types 2.8 The Self Type 2.9 The Benefits of Type Hints 2.10 Summary 2.11 Online Resources 3 Class Slots 3.1 Introduction 3.2 The Problem 3.3 Slots to the Rescue 3.4 Performance Benefits 3.5 Why Not Use Slots? 3.6 Online Resource 4 Weak References 4.1 Introduction 4.2 How Garbage Collection Works: Reference Counting 4.3 Weak References 4.4 When to Use Weak References 4.5 The Weakref Module 4.6 Creating Weak References 4.7 Retrieving Objects from Weak References 4.8 The WeakValueDicitonary 4.9 WeakKeyDictionary 4.10 Proxy Objects 4.11 Online Resources 5 Data Classes 5.1 Introduction 5.2 A Traditional Class 5.3 Defining Data Classes 5.4 Defining Additional Behaviour 5.5 The Dataclass Decorator 5.6 Custom Factory for Fields 5.7 Immutable Dataclasses 5.8 Data Classes and Inheritance 5.9 Post Initialisation 5.10 Initialisation Variables 5.11 Positional Attributes 5.12 Exercise 6 Structural Pattern Matching 6.1 Introduction 6.2 The Match Statement 6.3 Matching Classes with Positional Attributes 6.4 Matching Against Standard Classes 6.5 Online Resource 7 Working with pprint 7.1 Introduction 7.2 The pprint Data Printer Module 7.3 Basic pprint Usage 7.4 Changing the Width 7.5 Changing the Depth 7.6 Managing the Indentation Level 7.7 Reducing Line Breaks Using Compact 7.8 The pformat Function 7.9 The saferepr() Function 7.10 Using the PrettyPrinter Class 7.11 Online Resource 8 Shallow v Deep Copy 8.1 Introduction 8.2 Copying a List of Lists 8.3 The Problem with Copying 8.4 The Copy Module to the Rescue 8.5 Using the deepcopy() Function 8.6 Online Resource 9 The __init__ Versus __new__ and __call 9.1 Introduction 9.2 The __new__ and __init__ Methods 9.3 The __new__ Method 9.4 When to Use the __new__ Method 9.5 Using __new__ to Create a Singleton Object 9.6 The __init__ Method 9.7 Can __new__ and __init__ Be Used Together? 9.8 The __call__ Method 9.9 Summary 10 Python Metaclasses and Meta Programming 10.1 Introduction 10.2 Metaprogramming 10.3 Decorators as a Form of Metaprogramming 10.4 Metaclasses for Metaprogramming 10.4.1 Singleton Metaclass 10.5 Exec and Eval for Metaprogramming 10.5.1 The exec() Function 10.5.2 The eval() Function 10.5.3 eval Versus exec() Part II Computer Graphics and GUIs 11 Introduction to Computer Graphics 11.1 Introduction 11.2 Background 11.3 The Graphical Computer Era 11.4 Interactive and Non Interactive Graphics 11.5 Pixels 11.6 Bit Map Versus Vector Graphics 11.7 Buffering 11.8 Python and Computer Graphics 11.9 References 11.10 Online Resources 12 Python Turtle Graphics 12.1 Introduction 12.2 The Turtle Graphics Library 12.2.1 The Turtle Module 12.2.2 Basic Turtle Graphics 12.2.3 Drawing Shapes 12.2.4 Filling Shapes 12.3 Other Graphics Libraries 12.4 3D Graphics 12.4.1 PyOpenGL 12.5 Online Resources 12.6 Exercises 13 Computer Generated Art 13.1 Creating Computer Art 13.2 A Computer Art Generator 13.3 Fractals in Python 13.4 The Koch Snowflake 13.5 Mandelbrot Set 13.6 Online Resources 13.7 Exercises 14 Introduction to Matplotlib 14.1 Introduction 14.2 Matplotlib 14.3 Plot Components 14.4 Matplotlib Architecture 14.4.1 Backend Layer 14.4.2 The Artist Layer 14.4.3 The Scripting Layer 14.5 Online Resources 15 Graphing with Matplotlib Pyplot 15.1 Introduction 15.2 The pyplot API 15.3 Line Graphs 15.3.1 Coded Format Strings 15.4 Scatter Graph 15.4.1 When to Use Scatter Graphs 15.5 Pie Charts 15.5.1 Expanding Segments 15.5.2 When to Use Pie Charts 15.6 Bar Charts 15.6.1 Horizontal Bar Charts 15.6.2 Coloured Bars 15.6.3 Stacked Bar Charts 15.6.4 Grouped Bar Charts 15.7 Figures and Subplots 15.8 3D Graphs 15.9 Exercises 16 Graphical User Interfaces 16.1 Introduction 16.2 GUIs and WIMPS 16.3 Windowing Frameworks for Python 16.3.1 Platform-Independent GUI Libraries 16.3.2 Platform-Specific GUI Libraries 16.4 Online Resources 17 Tkinter GUI Library 17.1 Introduction 17.2 Tkinter 17.3 Windows as Objects 17.4 Key Concepts 17.4.1 The Tk Class 17.4.2 TK Widgets 17.4.3 The TopLevel Class 17.4.4 The Frame Class 17.4.5 Dialogs 17.4.6 The Canvas Class 17.5 The Class Inheritance Hierarchy 17.5.1 Layout Management 17.6 A Simple Example 17.7 Tkinter Installation 17.7.1 Mac Installation 17.7.2 Windows Installation 17.8 GUI Builders for Tkinter 17.9 Online Resources 17.10 Exercises 18 Events in Tkinter User Interfaces 18.1 Introduction 18.2 Event Handling 18.3 What is Event Handling? 18.4 What Are Event Handlers? 18.5 Event Binders 18.6 Virtual Events 18.7 Event Definitions 18.8 What Types of Event Are There? 18.9 Binding an Event to an Event Handler 18.10 Implementing Event Handling 18.11 An Interactive GUI Application 18.12 Online Resources 18.13 Exercises 19 PyDraw Tkinter Example Application 19.1 Introduction 19.2 The PyDraw Application 19.3 The Structure of the Application 19.3.1 Model, View and Controller Architecture 19.3.2 PyDraw MVC Architecture 19.3.3 Additional Classes 19.3.4 Object Relationships 19.4 The Interactions Between Objects 19.4.1 The PyDrawApp 19.5 The PyDrawView Constructor 19.5.1 Changing the Application Mode 19.5.2 Adding a Graphic Object 19.6 The Classes 19.6.1 The PyDrawConstants Class 19.6.2 The PyDrawView Class 19.6.3 The PyDrawMenuBar Class 19.6.4 The PyDrawController Class 19.6.5 The DrawingModel Class 19.6.6 The DrawingView Class 19.6.7 The DrawingController Class 19.6.8 The Figure Class 19.6.9 The Square Class 19.6.10 The Circle Class 19.6.11 The Line Class 19.6.12 The Text Class 19.7 Reference 19.8 Exercises Part III Computer Games 20 Introduction to Games Programming 20.1 Introduction 20.2 Games Frameworks and Libraries 20.3 Python Games Development 20.4 Using Pygame 20.5 Online Resources 21 Building Games with Pygame 21.1 Introduction 21.2 The Display Surface 21.3 Events 21.3.1 Event Types 21.3.2 Event Information 21.3.3 The Event Queue 21.4 A First pygame Application 21.5 Further Concepts 21.6 A More Interactive pygame Application 21.7 Alternative Approach to Processing Input Devices 21.8 pygame Modules 21.9 Online Resources 22 StarshipMeteors Pygame 22.1 Introduction 22.2 Creating a Spaceship Game 22.3 The Main Game Class 22.4 The GameObject Class 22.5 Displaying the Starship 22.6 Moving the Spaceship 22.7 Adding a Meteor Class 22.8 Moving the Meteors 22.9 Identifying a Collision 22.10 Identifying a Win 22.11 Increasing the Number of Meteors 22.12 Pausing the Game 22.13 Displaying the Game Over Message 22.14 The StarshipMeteors Game 22.15 Online Resources 22.16 Exercises Part IV Testing 23 Introduction to Testing 23.1 Introduction 23.2 Types of Testing 23.3 What Should Be Tested? 23.4 Types of Testing 23.4.1 Unit Testing 23.4.2 Integration Testing 23.4.3 System Testing 23.4.4 Installation/Upgrade Testing 23.4.5 Smoke Tests 23.5 Automating Testing 23.6 Test-Driven Development 23.6.1 The TDD Cycle 23.6.2 Test Complexity 23.6.3 Refactoring 23.7 Design for Testability 23.7.1 Testability Rules of Thumb 23.8 Online Resources 23.9 Book Resources 24 PyTest Testing Framework 24.1 Introduction 24.2 What is PyTest? 24.3 Setting up PyTest 24.4 A Simple PyTest Example 24.5 Working with PyTest 24.6 Parameterised Tests 24.7 Online Resources 24.8 Exercises 25 Mocking for Testing 25.1 Introduction 25.2 Why Mock? 25.3 What is Mocking? 25.4 Common Mocking Framework Concepts 25.5 Mocking Frameworks for Python 25.6 The Unittest.Mock Library 25.6.1 Mock and Magic Mock Classes 25.6.2 The Patchers 25.6.3 Mocking Returned Objects 25.6.4 Validating Mocks Have Been Called 25.7 Mock and MagicMock Usage 25.7.1 Naming Your Mocks 25.7.2 Mock Classes 25.7.3 Attributes on Mock Classes 25.7.4 Mocking Constants 25.7.5 Mocking Properties 25.7.6 Raising Exceptions with Mocks 25.7.7 Applying Patch to Every Test Method 25.7.8 Using Patch as a Context Manager 25.8 Mock Where You Use It 25.9 Patch Order Issues 25.10 How Many Mocks? 25.11 Mocking Considerations 25.12 Online Resources 25.13 Exercises Part V File Input/Output 26 Introduction to Files, Paths and IO 26.1 Introduction 26.2 File Attributes 26.3 Paths 26.4 File Input/Output 26.5 Sequential Access versus Random Access 26.6 Files and I/O in Python 26.7 Online Resources 27 Reading and Writing Files 27.1 Introduction 27.2 Obtaining References to Files 27.3 Reading Files 27.4 File Contents Iteration 27.5 Writing Data to Files 27.6 Using Files and with Statements 27.7 The Fileinput Module 27.8 Renaming Files 27.9 Deleting Files 27.10 Random Access Files 27.11 Directories 27.12 Temporary Files 27.13 Working with Paths 27.14 Online Resources 27.15 Exercise 28 Stream IO 28.1 Introduction 28.2 What is a Stream? 28.3 Python Streams 28.4 IOBase 28.5 Raw IO/UnBuffered IO Classes 28.6 Binary IO/Buffered IO Classes 28.7 Text Stream Classes 28.8 Stream Properties 28.9 Closing Streams 28.10 Returning to the Open() Function 28.11 Online Resource 28.12 Exercise 29 Working with CSV Files 29.1 Introduction 29.2 CSV Files 29.2.1 The CSV Writer Class 29.2.2 The CSV Reader Class 29.2.3 The CSV DictWriter Class 29.2.4 The CSV DictReader Class 29.3 Online Resources 29.4 Exercises 30 Working with Excel Files 30.1 Introduction 30.2 Excel Files 30.3 The Openpyxl. Workbook Class 30.4 The Openpyxl. WorkSheet Objects 30.5 Working with Cells 30.6 Sample Excel File Creation Application 30.7 Loading a Workbook from an Excel File 30.8 Online Resources 30.9 Exercises 31 Regular Expressions in Python 31.1 Introduction 31.2 What Are Regular Expressions? 31.3 Regular Expression Patterns 31.3.1 Pattern Metacharacters 31.3.2 Special Sequences 31.3.3 Sets 31.4 The Python re Module 31.5 Working with Python Regular Expressions 31.5.1 Using Raw Strings 31.5.2 Simple Example 31.5.3 The Match Object 31.5.4 The search() Function 31.5.5 The match() Function 31.5.6 The Difference Between Matching and Searching 31.5.7 The finadall() Function 31.5.8 The finditer() Function 31.5.9 The split() Function 31.5.10 The sub() Function 31.5.11 The compile() Function 31.6 Online Resources 31.7 Exercises Part VI Database Access 32 Introduction to Databases 32.1 Introduction 32.2 What Is a Database? 32.2.1 Data Relationships 32.2.2 The Database Schema 32.3 SQL and Databases 32.4 Data Manipulation Language 32.5 Transactions in Databases 32.6 Further Reading 33 Python DB-API 33.1 Accessing a Database from Python 33.2 The DB-API 33.2.1 The Connect Function 33.2.2 The Connection Object 33.2.3 The Cursor Object 33.2.4 Mappings from Database Types to Python Types 33.2.5 Generating Errors 33.2.6 Row Descriptions 33.3 Transactions in PyMySQL 33.4 Online Resources 34 PyMySQL Module 34.1 The PyMySQL Module 34.2 Working with the PyMySQL Module 34.2.1 Importing the Module 34.2.2 Connect to the Database 34.2.3 Obtaining the Cursor Object 34.2.4 Using the Cursor Object 34.2.5 Obtaining Information About the Results 34.2.6 Fetching Results 34.2.7 Close the Connection 34.3 Complete PyMySQL Query Example 34.4 Inserting Data to the Database 34.5 Updating Data in the Database 34.6 Deleting Data in the Database 34.7 Creating Tables 34.8 Online Resources 34.9 Exercises Part VII Logging 35 Introduction to Logging 35.1 Introduction 35.2 Why Log? 35.3 What is the Purpose of Logging? 35.4 What Should You Log? 35.5 What not to Log 35.6 Why not Just Use Print? 35.7 Online Resources 36 Logging in Python 36.1 The Logging Module 36.2 The Logger 36.3 Controlling the Amount of Information Logged 36.4 Logger Methods 36.5 Default Logger 36.6 Module Level Loggers 36.7 Logger Hierarchy 36.8 Formatters 36.8.1 Formatting Log Messages 36.8.2 Formatting Log Output 36.9 Online Resources 36.10 Exercises 37 Advanced Logging 37.1 Introduction 37.2 Handlers 37.2.1 Setting the Root Output Handler 37.2.2 Programmatically Setting the Handler 37.2.3 Multiple Handlers 37.3 Filters 37.4 Logger Configuration 37.5 Performance Considerations 37.6 Exercises Part VIII Concurrency and Parallelism 38 Introduction to Concurrency and Parallelism 38.1 Introduction 38.2 Concurrency 38.3 Parallelism 38.4 Distribution 38.5 Grid Computing 38.6 Concurrency and Synchronisation 38.7 Object Orientation and Concurrency 38.8 Threads V Processes 38.9 Some Terminology 38.10 Online Resources 39 Threading 39.1 Introduction 39.2 Threads 39.2.1 Thread States 39.2.2 Creating a Thread 39.2.3 Instantiating the Thread Class 39.3 The Thread Class 39.4 The Threading Module Functions 39.5 Passing Arguments to a Thread 39.6 Extending the Thread Class 39.7 Daemon Threads 39.8 Naming Threads 39.9 Thread Local Data 39.10 Timers 39.11 The Global Interpreter Lock 39.12 Online Resources 39.13 Exercise 40 MultiProcessing 40.1 Introduction 40.2 The Process Class 40.3 Working with the Process Class 40.4 Alternative Ways to Start a Process 40.5 Using a Pool 40.6 Exchanging Data Between Processes 40.7 Sharing State Between Processes 40.7.1 Process Shared Memory 40.8 Online Resources 40.9 Exercises 41 Inter Thread/Process Synchronisation 41.1 Introduction 41.2 Using a Barrier 41.3 Event Signalling 41.4 Synchronising Concurrent Code 41.5 Python Locks 41.6 Python Conditions 41.7 Python Semaphores 41.8 The Concurrent Queue Class 41.9 Online Resources 41.10 Exercises 42 Futures 42.1 Introduction 42.2 The Need for a Future 42.3 Futures in Python 42.3.1 Future Creation 42.3.2 Simple Example Future 42.4 Running Multiple Futures 42.4.1 Waiting for All Futures to Complete 42.4.2 Processing Results as Completed 42.5 Processing Future Results Using a Callback 42.6 Online Resources 42.7 Exercises 43 Concurrency with AsyncIO 43.1 Introduction 43.2 Asynchronous IO 43.3 Async IO Event Loop 43.4 The Async and Await Keywords 43.4.1 Using Async and Await 43.5 Async IO Tasks 43.6 Running Multiple Tasks 43.6.1 Collating Results from Multiple Tasks 43.6.2 Handling Task Results as They Are Made Available 43.7 Online Resources 43.8 Exercises 44 Performance Monitoring and Profiling 44.1 Introduction 44.2 Why Monitor Performance and Memory? 44.3 Performance Monitoring and Profiling 44.4 Performance Monitoring 44.4.1 The Time Module 44.4.2 The Timeit Module 44.4.3 The Psutil Module 44.5 Python Profiling 44.5.1 The cProfile Module 44.5.2 The Line_Profiler Module 44.5.3 The Memory_Profiler Module 44.5.4 Additional Third-Party Libraries 44.6 Profiling with cProfile 44.7 Memory Profiling 44.8 Online Resources Part IX Reactive Programming 45 Reactive Programming Introduction 45.1 Introduction 45.2 What Is a Reactive Application? 45.3 The ReactiveX Project 45.4 The Observer Pattern 45.5 Hot and Cold Observables 45.6 Differences Between Event Driven Programming and Reactive Programming 45.7 Advantages of Reactive Programming 45.8 Disadvantages of Reactive Programming 45.9 The RxPy Reactive Programming Framework 45.10 Online Resources 46 RxPy Observables, Observers and Subjects 46.1 Introduction 46.2 RxPy Library 46.3 Observables in RxPy 46.4 Observers in RxPy 46.5 Multiple Subscribers/Observers 46.6 Subjects in RxPy 46.7 Observer Concurrency 46.7.1 Available Schedulers 46.8 Online Resources 46.9 Exercises 47 RxPy Operators 47.1 Introduction 47.2 Reactive Programming Operators 47.3 Piping Operators 47.4 Creational Operators 47.5 Transformational Operators 47.6 Combinatorial Operators 47.7 Filtering Operators 47.8 Mathematical Operators 47.9 Chaining Operators 47.10 Online Resources 47.11 Exercises Part X Network Programming 48 Introduction to Sockets and Web Services 48.1 Introduction 48.2 Sockets 48.3 Web Services 48.4 Addressing Services 48.5 Localhost 48.6 Port Numbers 48.7 IPv4 Versus IPv6 48.8 Sockets and Web Services in Python 48.9 Online Resources 49 Sockets in Python 49.1 Introduction 49.2 Socket to Socket Communication 49.3 Setting up a Connection 49.4 An Example Client Server Application 49.4.1 The System Structure 49.4.2 Implementing the Server Application 49.4.3 Socket Types and Domains 49.4.4 Implementing the Client Application 49.5 The Socketserver Module 49.6 Http Server 49.7 Online Resources 49.8 Exercises 50 Web Services in Python 50.1 Introduction 50.2 RESTful Services 50.3 A RESTful API 50.4 Python Web Frameworks 50.5 Online Resources 51 Flask Web Services 51.1 Introduction 51.2 Flask 51.3 Hello World in Flask 51.3.1 Using JSON 51.4 Implementing a Flask Web Service 51.4.1 A Simple Service 51.4.2 Providing Routing Information 51.5 Running the Service 51.6 Invoking the RESTFul Service 51.6.1 The Final Solution 51.7 Online Resources 52 Flask Bookshop Web Service 52.1 Introduction 52.2 Building a Flask Bookshop Service 52.3 The Design 52.4 The Domain Model 52.5 Encoding Books into JSON 52.6 Setting Up the GET Services 52.7 Deleting a Book 52.8 Adding a New Book 52.9 Updating a Book 52.10 What Happens if We Get It Wrong? 52.11 Bookshop Services Listing 52.12 Exercises Part XI Data Science: Data Analytics and Machine Learning 53 Introduction to Data Science 53.1 Introduction 53.2 Data Science 53.3 Data Science Tools and Techniques 53.4 Data Analytics Process 53.5 Python and Data Science 53.6 Machine Learning for Data Science 53.7 Online Resources 54 Pandas and Data Analytics 54.1 Introduction 54.2 The Data 54.2.1 The UK Government COVID Data Set 54.2.2 The Google Mobility Data Set 54.3 Python Pandas 54.3.1 Pandas Series and DataFrames 54.4 Loading and Analysing UK COVID Data Set 54.5 Loading the Google Mobility Data Set 54.6 Merging Two DataFrames 54.7 Analysing the Combined Data 54.8 Summary 55 Alternatives to Pandas 55.1 Introduction 55.2 Comparing Pandas 2.0.0 55.3 Pandas 1.x v 2.x 55.4 Pandas Versus Other Libraries and Tools 55.5 Online Resources 56 Machine Learning in Python 56.1 Introduction 56.2 The Data 56.3 SciKitLearn 56.4 The Problem 56.5 Using Regression Supervised Learning Systems 56.6 K-Nearest Neighbour Regressor 56.7 Decision Tree Regressor 56.8 Random Forest Regressor 56.9 Summary of Metrics Obtained 56.10 Creating the Regressor Object 56.11 Online Resources 57 Pip and Conda Virtual Environments 57.1 Introduction 57.2 Virtual Environments 57.3 Working with Pip 57.3.1 Activating a Pip Environment 57.3.2 Installing Modules Using Pip 57.3.3 Deactivating a Pip Environment 57.3.4 Check Version of Pip 57.3.5 Installing Modules into a Pip Environment 57.3.6 Freezing Modules 57.4 Conda 57.5 Anaconda 57.5.1 Installing Anaconda 57.6 Working with Anaconda 57.6.1 Checking the Conda Version 57.6.2 Updating Conda 57.6.3 Creating a Conda Environment 57.6.4 Listing Available Conda Environments 57.6.5 Activating a Conda Environment 57.6.6 Deactivating a Conda Environment 57.6.7 Listing the Modules Loaded into a Conda Environment 57.6.8 Removing an Anaconda Environment 57.6.9 Installing a Module into a Conda Environment 57.7 Anaconda in PyCharm 57.8 Online Resources