LEAN SOFTWARE SYSTEMS ENGINEERING FOR DEVELOPERS : managing requirements, complexity, teams, and... change like a champ

Graduate to the next level of your software development career, learning the tools you need to successfully manage the complexity of modern software systems. Whether you are a developer at a small software company, or one of many developers at a large enterprise, your success directly correlates to the ability of your development team to rapidly respond to change. What makes this task challenging in today's world, is that the technical challenges we as developers strive to overcome are becoming increasingly more complex. We have to consider many more options when it comes to things like requirements, solution hosting, support, pace of change, and generally with less time and warning. A good developer knows that it is critical to manage every aspect of software development from soup to nuts, and understands that when details and decisions are left to chance, outcomes can be negatively impacted. Poor planning can result in increased errors, substandard quality, budget and schedule overruns, and result in the ultimate business failure, dissatisfied customers, and stakeholders. This book will help you put on the lenses of a software engineer. You will come away with an understanding of how to view the entire spectrum of the software development process, learn valuable concepts, and apply these principles through meaningful examples, case studies, and source code. What You Will Learn Move beyond being a programmer to being a professional software engineer Spend more time doing software development; minimize time spent dealing with ineffective or inadequate processes Reduce errors in judgment and provide predictable outcomes, while still maintaining agility and responsiveness using Lean and Agile practices Know the steps you can take to ensure a shared understanding among stakeholders Discover tools to validate user experience early and often to minimize costly re-work Develop software designs and architectures that enable long-term business agility Implement patterns and processes that result in "falling into the pit of success" instead of into the "pit of failure" Adopt processes and patterns that will result in pervasive "institutionalized" quality Understand the necessity of redefining the essential role of technical leadership to ensure team maturity and growth Who This Book Is For Software developers and team leaders who have struggled to implement design and development best practices due to lack of team resources, in-depth knowledge, or experience, and want a book designed to provide the confidence and foundational skills needed to achieve success Table of Contents 5 About the Authors 10 About the Technical Reviewer 12 Acknowledgments 13 Foreword 14 Introduction 17 Chapter 1: Focusing on Software Development Outcomes Instead of Outputs 22 Introduction 22 Software Engineering vs. Software Development 27 Coding Is Not Software Engineering 28 Defining Software Engineering 30 Why Does Software Engineering Matter? 33 Errors in Judgment 36 Managing the Dimensions of Complexity in Software Development 40 Objective Complexity 40 Requirements Complexity 41 Solution Complexity 42 Agility Comes from Managing Both Requirements and Solution Complexity 43 Our Responsibility 45 A Path Forward 46 Summary 48 Key Takeaways 48 Chapter 2: Gaining a Shared Understanding Throughout the Project 50 Introduction 50 Sources of Defects and Rework 52 The Danger of Incomplete Pictures 53 Why Developing a Shared Understanding Early Is Important 58 Adding Structured Processes 60 Leveraging Lean Approaches to Backlog Development 61 Journey Mapping and Story Mapping 63 Using Critical Thought to Decompose Requirements 65 How Other Engineers Tackle Big Problems 66 Estimation Scales 67 Estimating Our “Submit Order” Story 68 Summary 70 Key Takeaways 70 Chapter 3: Validation of User Experience 72 Introduction 72 The Evolution of UI Design 72 Challenges with Modern UI Design 73 Scenario 1: Technology 74 Constraints of the Modern Web Frameworks 75 Business Logic Bleeding into the UI Layer 77 Scenario 2: Process 78 Missing Artifacts 80 No Accommodation for UI Design Time 81 Failure to Communicate 81 Strategy for Effective UI Validation 82 Generate Good Requirements 82 Develop Acceptance Criteria 83 Develop Screen Flows and Wireframes 84 Review the Preliminary UI Design 85 Develop Mockups 85 Review the Mockups 86 Design the UI Components 88 Define the Client/Server Interactions 89 Build the UI Using Mock Data 89 Review the Working UI 90 Integrate Working UI with the Real Back End 90 Summary 90 Key Takeaways 91 Chapter 4: Designing Software Systems That Age Well and Adapt to Change 93 Introduction 93 Reality Check 93 Errors in Judgment Revisited 96 Case Study: Notification Subsystem 97 Change 1: Supplier Order Email 101 Change 2: Order Shipped Notices 102 Change 3: New SMTP Service 104 Change 4: Order Notices for Shipping 105 Change 5: Abandoned Cart Notices 106 Change 6: The Change That Forces the Redesign Conversations 108 The Redesign 110 What Critical Thought Might Have Uncovered 113 Tactical vs. Strategic Development 113 Case Study Wrap-Up: Common Errors in Judgment 115 Lack of a Design Identity and a Defined/Disciplined Methodology 116 Decomposing Systems Based upon Requirements Instead of Change 116 Thinking Tactically Instead of Strategically 117 Lack of Understanding of Core Concepts and Principles and the Impacts of Design Decision Trade-offs 117 Allowing Material Design Decisions to Be Made by Anyone 118 Stagnant Mental Models for Software Systems 118 Core Concepts and Principles of Modern System Design 119 Coupling and Cohesion 119 Information Hiding 120 Trade-offs Are the Norm 125 Information Hiding Hierarchies 126 Why Designing for Change Is Important 128 Design Stamina 128 Increased Comprehensibility 130 Meeting the Demands of Modern Applications and Business Agility 132 Managing Complexity in Other Systems 133 Choosing a Design Methodology 134 Summary 136 Key Takeaways 136 Chapter 5: Developers “Falling into the Pit of Success” 138 Introduction 138 Reality Check 138 A Developer’s Life 140 A Bad Week 140 Sound Familiar? 144 Creating a Pit of Success for Developers 145 Positive/Can-Do Team Attitude 147 Ensuring Requirements and Acceptance Criteria Are Understood 147 What Is My Understanding of the Requirements? What Is the Primary Goal or Objective? 150 What Are the Secondary Goals/Objectives? 150 What Are the Known Design Constraints? 150 What Are the Acceptance Criteria for the Solution (Including Non-functional)? 150 What Assumptions Am I Making About These Requirements and the System? 150 What Are the Unknowns? (i.e., What Information Is Not Currently Available?) 151 What Are the Trade-offs I Am Deliberately Making? 151 What Areas/Features Are Most Likely to Change over Time? 151 What Are Anticipated/Possible Failure Scenarios, and How Should They Be Handled? 151 Are There Any Special Considerations Related to Security? 152 What Are the Existing Areas Impacted? 152 How Will the Design/Architecture Need to Change? 152 How Can I Encapsulate Current and Future Change? 152 How Should I Test and Validate the System? 153 What Risks Am I Aware Of? 153 What Concerns Do I Have? 153 What Are the Specific Steps to Implemention, and What Is Their Level of Effort? 154 UI Design Artifacts Before Development 154 Design Identity 155 Service-Based Systems 156 Reviewing and Verifying the Design 157 Testability 158 Deciding What to Unit Test 159 Isolating Developer Environments 161 Maker Schedule 162 Mentor 164 Establishing Clear Developer Expectations 164 Pit of Success Checklist 166 Summary 167 Key Takeaways 167 Chapter 6: Institutionalized Quality 169 Introduction 169 Reality Check 169 Moving Fast vs. Maintaining a Rapid Pace 170 Toward an Integrated, Layered View 171 Quality Targets 172 Defect Detection Rate 172 Percentage of Rework 173 Quality Mindset 174 Quality Practices Review 175 During Requirements Analysis 176 Shared Understanding 176 Validation of the UI 176 During Design 177 Validation of the UI 177 Designing Systems That Age Well 177 Designing How to Execute the Project 178 During Development 178 Establishing Clear Developer Expectations 179 Reviewing the Requirements and Design Artifacts 179 Ensuring Requirements and Acceptance Criteria Are Understood 179 Enabling a Testable System 179 Implementing Continuous Integration with Automated Integration and Unit Tests 180 Using Structured Code Reviews and Pull Requests 180 Pull Request Code Review Checklist 181 Review Before Submitting 181 Testing 182 Architecture 182 General Code Review 182 New Projects 183 Database Changes 183 UI Changes 183 Cloud Architecture Changes 184 During Acceptance Testing and QA Testing 184 Summary 184 Key Takeaways 185 Chapter 7: The Role of Chief Engineer 186 Introduction 186 Revisiting the Football Analogy 186 Chief Engineer 188 Responsibilities of a Chief Engineer 188 Maintaining the Big Picture 189 Maintaining Conceptual Integrity 189 Seeing Around Corners 191 Coaching and Mentoring 192 Building Trusted Relationships 192 Being a Proactive Communicator 193 Ensuring Adequate Testing 193 Achieving Technical Success of the System 194 Summary 194 Key Takeaways 195 Chapter 8: Bringing It All Together – Creating an Action Plan 196 Introduction 196 What It Will Mean for Software Development to Be an Engineering Discipline 196 Starting Your Transformation 197 Understanding the Maturity of Your Team 199 Lightweight Organizational Maturity Assessment 200 Leadership Roles 200 Managing Requirements Complexity 200 Solution Complexity 200 Operational Effectiveness 201 The Steps Toward Transformation 201 Secure Your Platform 202 Minimize Uncertainty 202 Automate Your Builds 203 Protect Your Developer Time 203 Maintain Conceptual Integrity 203 Improve Your Productivity by Improving Quality 203 Reduce Rework 204 Improve Predictability and Visibility of Results 204 Achieving the Ultimate “-ility”... Funability 204 What Contributes to Funability 206 Frequent Delivery of Value to Customers 206 Being Part of a Team 206 Maintainability of Systems 207 Effective Management of Technical Debt 207 Consistent Quality of Product Releases 207 Productivity and Efficiency of the Developers 208 Sound Software Design 208 Now What? 209 Key Takeaways 209 Index 210 Graduate to the next level of your software development career, learning the tools you need to successfully manage the complexity of modern software systems. Whether you are a developer at a small software company, or one of many developers at a large enterprise, your success directly correlates to the ability of your development team to rapidly respond to change. What makes this task challenging in today's world, is that the technical challenges we as developers strive to overcome are becoming increasingly more complex. We have to consider many more options when it comes to things like requirements, solution hosting, support, pace of change, and generally with less time and warning. A good developer knows that it is critical to manage every aspect of software development from soup to nuts, and understands that when details and decisions are left to chance, outcomes can be negatively impacted. Poor planning can result in increased errors, substandard quality, budget and schedule overruns, and result in the ultimate business failure, dissatisfied customers, and stakeholders. This book will help you put on the lenses of a software engineer. You will come away with an understanding of how to view the entire spectrum of the software development process, learn valuable concepts, and apply these principles through meaningful examples, case studies, and source code. What You Will Learn Move beyond being a programmer to being a professional software engineer Spend more time doing software development; minimize time spent dealing with ineffective or inadequate processes Reduce errors in judgment and provide predictable outcomes, while still maintaining agility and responsiveness using Lean and Agile practices Know the steps you can take to ensure a shared understanding among stakeholders Discover tools to validate user experience early and often to minimize costly re-work Develop software designs and architectures that enable long-term business agility Implement patterns and processes that result in "falling into the pit of success" instead of into the "pit of failure" Adopt processes and patterns that will result in pervasive "institutionalized" quality Understand the necessity of redefining the essential role of technical leadership to ensure team maturity and growth This book is for software developers and team leaders who have struggled to implement design and development best practices due to lack of team resources, in-depth knowledge, or experience, and want a book designed to provide the confidence and foundational skills needed to achieve success. Doug Durham is CEO of Don't Panic Labs, a firm that helps companies innovate through the design and development of software technologies. He is also the co-founder of Nebraska Global (parent company of Don't Panic Labs), a pioneer in the startup landscape in Nebraska. Doug has more than three decades of software engineering and development experience in aerospace and defense, healthcare, manufacturing, ecommerce, consumer web applications, and Internet network services. He is passionate about the process of solving problems through software, and the application of sound engineering principles and patterns to these efforts. Doug has taught at the University of Nebraska-Lincoln Raikes School of Computer Science, and serves on a College of Engineering advisory board. He often speaks at industry conferences on the topic of software engineering, and is a frequent guest lecturer at the University of Nebraska-Lincoln. Chad Michel is Lead Software Architect for Don't Panic Labs and has more than 20 years of software development and engineering experience. He helps clients solve problems through innovative software solutions. He graduated from the University of Nebraska-Lincoln with a bachelor's degree in computer engineering and a master's degree in computer science. Chad has worked for several companies in Lincoln, where he helped build a practice management application for lawyers, developed key features for an ecommerce application, and helped wrangle a content delivery network into a stable platform. He often speaks at technical meetups hosted by Don't Panic Labs, as well as at many other conferences and technical groups. He regularly contributes to the Don't Panic Labs blog. Chad is a fourth-degree black belt in Tae Kwon Do.