Hands-On Software Engineering with Python : Move Beyond Basic Programming and Construct Reliable and Efficient Software with Complex Code

Learn how to apply real-world Software Engineering disciplines by seeing them applied to a Python system, from fundamentals through high-end systems.Key Features Learn what skills, technical and otherwise, are expected from Senior Developers Understand how the skills are applied to a realistically-scoped project Release the power of general and Python development best practices Book DescriptionSoftware Engineering is about more than just writing code — it includes a host of “soft” skills that apply to almost any development-effort, no matter the language, development-methodology or scope of the project. Being a Senior Developer all but requires awareness of how those skills, along with their expected technical counterparts, mesh together through a project’s life-cycle.This book walks you through that discovery by walking through the entire life-cycle of a multi-tier system and its related software projects. You’ll see what happens before any development takes place, and what impact the decisions and designs made at each step have on the development process. The development of the entire project, over the course of several iterations based on real-world Agile iterations, will be implemented, sometimes starting with nothing, in one of the fastest-growing languages in the world — Python. The details of, and advantages behind a number industry best practices will show why they are best practices. Application of those practices in Python will be laid out, and some Python-specific capabilities that are often overlooked. Finally, the book will, from first principles to complete implementation, implement a High Performance Computing solution, and show the application of Metaprogramming concepts to the book’s project.What you will learn Learn what happens over the course of a system’s life (SDLC). Learn what to expect from the pre-development life-cycle steps. Find out how the development-specific phases of the SDLC affect development. Uncover what a real-world development process might be like, in an Agile way. Find out how to do more than just write the code. Learn project-independent best practices exist, and how to use them. Find out how to design and implement a High Performance Computing process. Learn what Metaprogramming is, can do, and what risks come with it Who This Book Is ForThis book is for existing developers who have basic understanding of Programming and Programming paradigms and would like to skill up as a Senior Programmer. Basic Python knowledge is assumedAbout the AuthorBrian Allbeehas been a developer professionally for two decades. His experience includes web and system applications in several languages, for industries as varied as organization membership services, consumer health-care reporting, advertising, and cloud automation. He fell in love with Python just before 2.6 was available, and has focused his development efforts as much on Python as time and professional obligations allowed since then. He lives and codes in the Denver area. Cover......Page 1 Title Page......Page 2 Copyright and Credits......Page 3 Packt Upsell......Page 4 Contributors......Page 5 Table of Contents......Page 7 Preface......Page 14 Chapter 1: Programming versus Software Engineering......Page 21 The bigger picture......Page 22 Asking questions......Page 23 Summary......Page 31 Chapter 2: The Software Development Life Cycle......Page 32 Pre-development phases of the SDLC......Page 33 Initial concept/vision......Page 34 Concept development......Page 36 Project management planning......Page 38 System architecture and design......Page 40 System integration, testing, and acceptance......Page 41 Post-development phases of the SDLC......Page 42 Summary......Page 44 Architecture, both logical and physical......Page 45 Logical architecture......Page 46 Physical architecture......Page 48 Use cases (business processes and rules)......Page 50 Data structure and flow......Page 52 Interprocess communication......Page 54 System scope and scale......Page 57 Summary......Page 59 Chapter 4: Methodologies, Paradigms, and Practices......Page 60 Process methodologies......Page 61 Waterfall......Page 62 Agile (in general)......Page 66 Scrum......Page 69 Scrum and the phases of the SDLC model......Page 76 Kanban......Page 77 Kanban and the phases of the SDLC model......Page 79 Extreme programming......Page 80 Feature-driven development......Page 81 Test-driven design......Page 82 Development paradigms......Page 83 Object-oriented programming......Page 84 Functional programming......Page 87 Development practices......Page 89 Continuous integration......Page 90 Continuous delivery or deployment......Page 91 Summary......Page 92 Goals for the system......Page 93 What's known/designed before development starts......Page 95 What the iteration chapters will look like......Page 98 Iteration goals and stories......Page 99 Post-development considerations and impact......Page 100 Summary......Page 101 Development tools......Page 102 Integrated Development Environment (IDE) options......Page 103 IDLE......Page 105 Geany......Page 107 Eclipse variations + PyDev......Page 109 Others......Page 112 Source Code Management......Page 113 Typical SCM activities......Page 115 Git......Page 117 Basic workflows for Git and SVN compared......Page 118 Best practices......Page 120 Standards for code......Page 121 PEP-8......Page 122 Code organization in modules......Page 125 Structure and standards for classes......Page 129 Function and method annotation (hinting)......Page 135 Process standards......Page 138 Unit testing......Page 139 Repeatable build processes......Page 145 Integrating unit tests and build processes......Page 150 Defining package structures for Python code......Page 151 Packages in a project's context......Page 153 Using Python virtual environments......Page 154 Summary......Page 158 Chapter 7: Setting Up Projects and Processes......Page 160 Assembly of stories and tasks......Page 161 Setting Up SCM......Page 162 Component project analysis......Page 169 Component project setup......Page 170 Packaging and build process......Page 171 Python virtual environments......Page 176 Basic unit testing......Page 184 Identifying missing test case classes......Page 186 Identifying missing test methods......Page 190 Creating reusable module code coverage tests......Page 195 The property and method testing decorators......Page 203 Creating unit test template files......Page 206 Integrating tests with the build process......Page 211 Summary......Page 213 Chapter 8: Creating Business Objects......Page 214 Iteration goals......Page 215 Assembly of stories and tasks......Page 216 A quick review of classes......Page 218 Address......Page 219 BaseArtisan......Page 229 OO principles – composition over inheritance......Page 231 Implementing BaseArtisan's properties......Page 234 Implementing BaseArtisan's methods......Page 244 BaseCustomer......Page 249 BaseOrder......Page 250 BaseProduct......Page 252 Dealing with duplicated code – HasProducts......Page 259 Summary......Page 266 Chapter 9: Testing Business Objects......Page 267 Starting the unit testing process......Page 268 Unit testing the Address class......Page 274 Unit testing HasProducts......Page 287 Unit testing BaseProduct......Page 290 Unit testing BaseOrder......Page 294 Unit-testing BaseCustomer......Page 296 Unit testing BaseArtisan......Page 298 Unit testing patterns established so far......Page 300 Distribution and installation considerations......Page 301 Quality assurance and acceptance......Page 304 Operation/use, maintenance, and decommissioning considerations......Page 305 Summary......Page 306 Chapter 10: Thinking About Business Object Data Persistence......Page 307 Iterations are (somewhat) flexible......Page 308 Data storage options......Page 310 Relational databases......Page 311 Advantages and drawbacks......Page 312 MySQL/MariaDB......Page 313 NoSQL databases......Page 314 Advantages and drawbacks......Page 317 Other NoSQL options......Page 318 Other data storage options......Page 319 Selecting a data storage option......Page 320 Polymorphism (and programming to an interface)......Page 322 Data access design strategies......Page 323 Why start from scratch?......Page 330 Summary......Page 331 Chapter 11: Data Persistence and BaseDataObject......Page 332 The BaseDataObject ABC......Page 333 Unit testing BaseDataObject......Page 350 Summary......Page 359 Chapter 12: Persisting Object Data to Files......Page 360 Setting up the hms_artisan project......Page 361 Creating a local file system data store......Page 362 Implementing JSONFileDataObject......Page 364 The concrete business objects of hms_artisan......Page 378 Dealing with is_dirty and properties......Page 379 hms_artisan.Artisan......Page 381 hms_artisan.Product......Page 387 hms_artisan.Order......Page 390 Summary......Page 402 Chapter 13: Persisting Data to a Database......Page 403 The Artisan Gateway and Central Office application objects......Page 404 Picking out a backend datastore engine......Page 405 The data access strategy for the Central Office projects......Page 410 Supporting objects for data persistence......Page 411 RDBMS implementations......Page 431 The concrete business objects of the Central Office projects......Page 439 hms_core.co_objects.Artisan......Page 440 hms_core.co_objects.Product......Page 447 Other hms_core.co_objects classes......Page 453 Accounting for the other CRUD operations......Page 454 Summary......Page 456 Chapter 14: Testing Data Persistence......Page 457 Testing hms_artisan.data_storage......Page 458 Testing hms_artisan.artisan_objects......Page 468 Testing the new hms_core Classes......Page 484 Unit testing hms_core.data_storage.py......Page 485 Building/distribution, demonstration, and acceptance......Page 494 Summary......Page 498 Chapter 15: Anatomy of a Service......Page 499 Service structure......Page 500 Configuration......Page 504 Windows-style .ini files......Page 505 JSON files......Page 507 YAML files......Page 508 Logging service activities......Page 509 Handling requests and generating responses......Page 512 Filesystem – based......Page 513 HTTP- or web-based......Page 514 Message- queue-based......Page 516 Other request types......Page 517 Request and response formats......Page 518 A generic service design......Page 519 The BaseDaemon ABC......Page 521 The BaseRequestHandler and BaseResponseFormatter ABCs......Page 536 Running a service using systemctl (Linux)......Page 544 Running a service using NSSM (Windows)......Page 546 macOS, launchd, and launchctl......Page 548 Managing services on other systems......Page 549 Summary......Page 550 Chapter 16: The Artisan Gateway Service......Page 551 Overview and goal......Page 552 Iteration stories......Page 555 Messages......Page 556 Deciding on a message-transmission mechanism......Page 566 Message-queue implementation with RabbitMQ......Page 568 Handling messages......Page 580 Queues and related Artisan properties......Page 585 Requirements for a web-service-based daemon......Page 591 Traffic to and from the service......Page 593 Impacts on testing and deployment......Page 598 Summary......Page 599 Chapter 17: Handling Service Transactions......Page 600 Remaining stories......Page 601 A bit of reorganization......Page 603 Preparation for object transactions......Page 604 Product object transactions......Page 606 Artisan – creating a product......Page 607 Central Office – approving/listing a product......Page 609 Central Office – altering product data......Page 611 Artisan – updating product data......Page 613 Artisan object transactions......Page 614 Central Office – creating an artisan......Page 617 Central Office – updating artisan data......Page 618 Artisan – updating Artisan data......Page 619 Order object transactions......Page 620 Customer – relaying order items to artisans......Page 621 Customer – canceling an order......Page 631 Artisan – fulfilling an item in an order......Page 633 When do messages get sent?......Page 635 Summary......Page 643 The challenges of testing services......Page 644 The overall testing strategy......Page 647 Unit testing variations of note......Page 648 Testing Artisan transactions......Page 653 Demonstrating the service......Page 668 Packaging and deploying the service......Page 673 Common considerations across all operating systems......Page 675 Linux (systemd) execution......Page 677 Windows (NSSM) execution......Page 678 Code review, refactoring, and cleanup......Page 681 Developing a UI......Page 683 Summary......Page 685 Chapter 19: Multiprocessing and HPC in Python......Page 686 Common factors to consider......Page 687 A simple but expensive algorithm......Page 689 Some testing setup......Page 690 Local parallel processing......Page 691 Threads......Page 700 Parallelizing across multiple machines......Page 701 Common functionality......Page 703 The Worker nodes......Page 705 The Orchestrator......Page 707 The Dispatcher......Page 708 Integrating Python with large-scale, cluster computing frameworks......Page 709 Python, Hadoop, and Spark......Page 712 Summary ......Page 713 Other Books You May Enjoy......Page 714 Index......Page 717 Explore various verticals in software engineering through high-end systems using Python Key FeaturesMaster the tools and techniques used in software engineeringEvaluates available database options and selects one for the final Central Office system-componentsExperience the iterations software go through and craft enterprise-grade systemsBook DescriptionSoftware Engineering is about more than just writing codeit includes a host of soft skills that apply to almost any development effort, no matter what the language, development methodology, or scope of the project. Being a senior developer all but requires awareness of how those skills, along with their expected technical counterparts, mesh together through a project's life cycle. This book walks you through that discovery by going over the entire life cycle of a multi-tier system and its related software projects. You'll see what happens before any development takes place, and what impact the decisions and designs made at each step have on the development process. The development of the entire project, over the course of several iterations based on real-world Agile iterations, will be executed, sometimes starting from nothing, in one of the fastest growing languages in the worldPython. Application of practices in Python will be laid out, along with a number of Python-specific capabilities that are often overlooked. Finally, the book will implement a high-performance computing solution, from first principles through complete foundation. What you will learnUnderstand what happens over the course of a system's life (SDLC)Establish what to expect from the pre-development life cycle stepsFind out how the development-specific phases of the SDLC affect developmentUncover what a real-world development process might be like, in an Agile wayFind out how to do more than just write the codeIdentify the existence of project-independent best practices and how to use themFind out how to design and implement a high-performance computing processWho this book is forHands-On Software Engineering with Python is for you if you are a developer having basic understanding of programming and its paradigms and want to skill up as a senior programmer. It is assumed that you have basic Python knowledge. Table of ContentsProgramming vs. Software Engineering The Software Development LifeCycle System ModelingMethodologies, Paradigms and PracticesThe hms_sys System-Project Development Tools and Best PracticesSetting up Projects and ProcessesCreating the Business-ObjectsTesting the Business-ObjectsThinking About Business-Object Data-PersistenceData Persistence and BaseDataObjectPersisting Object Data to FilesPersisting Data to a DatabaseTesting Data PersistenceAnatomy of a ServiceThe Artisan Gateway ServiceHandling Service TransactionsTesting and Deploying ServicesHigh Performance Computing in Python Explore various verticals in software engineering through high-end systems using Python Key Features Master the tools and techniques used in software engineering Evaluates available database options and selects one for the final Central Office system-components Experience the iterations software go through and craft enterprise-grade systems Book Description Software Engineering is about more than just writing code-it includes a host of soft skills that apply to almost any development effort, no matter what the language, development methodology, or scope of the project. Being a senior developer all but requires awareness of how those skills, along with their expected technical counterparts, mesh together through a project's life cycle. This book walks you through that discovery by going over the entire life cycle of a multi-tier system and its related software projects. You'll see what happens before any development takes place, and what impact the decisions and designs made at each step have on the development process. The development of the entire project, over the course of several iterations based on real-world Agile iterations, will be executed, sometimes starting from nothing, in one of the fastest growing languages in the world-Python. Application of practices in Python will be laid out, along with a number of Python-specific capabilities that are often overlooked. Finally, the book will implement a high-performance computing solution, from first principles through complete foundation. What you will learn Understand what happens over the course of a system's life (SDLC) Establish what to expect from the pre-development life cycle steps Find out how the development-specific phases of the SDLC affect development Uncover what a real-world development process might be like, in an Agile way Find out how to do more than just write the code Identify the existence of project-independent best practices and how to use them Find out how to design and implement a high-performance computing process Who this book is for Hands-On Software Engineering with Python is for you if you are a developer having basic understanding of programming and its paradigms and want to skill up as a senior programmer. It is assumed that you have basic Python knowledge. Downloading the example code for this book You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere .. Explore various verticals in software engineering through high-end systems using Python Key Features Master the tools and techniques used in software engineering Evaluates available database options and selects one for the final Central Office system-components Experience the iterations software go through and craft enterprise-grade systems Book Description Software Engineering is about more than just writing code - it includes a host of soft skills that apply to almost any development effort, no matter what the language, development methodology, or scope of the project. Being a senior developer all but requires awareness of how those skills, along with their expected technical counterparts, mesh together through a project's life cycle. This book walks you through that discovery by going over the entire life cycle of a multi-tier system and its related software projects. You'll see what happens before any development takes place, and what impact the decisions and designs made at each step have on the development process. The development of the entire project, over the course of several iterations based on real-world Agile iterations, will be executed, sometimes starting from nothing, in one of the fastest growing languages in the world - Python. Application of practices in Python will be laid out, along with a number of Python-specific capabilities that are often overlooked. Finally, the book will implement a high-performance computing solution, from first principles through complete foundation. What you will learn Understand what happens over the course of a system's life (SDLC) Establish what to expect from the pre-development life cycle steps Find out how the development-specific phases of the SDLC affect development Uncover what a real-world development process might be like, in an Agile way Find out how to do more than just write the code Identify the existence of project-independent best practices and how to use them Find out how to design and implement a high-performance computing process Who this book is for Hands-On Software Engineering with Python is for you if you are a developer having basic understanding of programming and its paradigms and want to skill up as a senior programmer. It is assumed that you have basic Python knowledge