Foundations of Linux debugging, disassembling, and reversing : analyze binary code, understand stack memory usage, and reconstruct C/C++ code with Intel x64

Review topics ranging from Intel x64 assembly language instructions and writing programs in assembly language, to pointers, live debugging, and static binary analysis of compiled C and C++ code. This book is ideal for Linux desktop and cloud developers. Using the latest version of Debian, you’ll focus on the foundations of the diagnostics of core memory dumps, live and postmortem debugging of Linux applications, services, and systems, memory forensics, malware, and vulnerability analysis. This requires an understanding of x64 Intel assembly language and how C and C++ compilers generate code, including memory layout and pointers. This book provides the back­ground knowledge and practical foundations you’ll need in order to master internal Linux program structure and behavior. It consists of practical step-by-step exercises of increasing complexity with explanations and ample diagrams. You’ll also work with the GDB debugger and use it for disassembly and reversing. By the end of the book, you will have a solid understanding of how Linux C and C++ compilers generate binary code. In addition, you will be able to analyze such code confidently, understand stack memory usage, and reconstruct original C/C++ code. Foundations of Linux Debugging, Disassembling, and Reversing is the perfect companion to Foundations of ARM64 Linux Debugging, Disassembling, and Reversing for readers interested in the cloud or cybersecurity. What You'll Learn Review the basics of x64 assembly language Examine the essential GDB debugger commands for debugging and binary analysis Study C and C++ compiler code generation with and without compiler optimizations Look at binary code disassembly and reversing patterns See how pointers in C and C++ are implemented and used Who This Book Is For Software support and escalation engineers, cloud security engineers, site reliability engineers, DevSecOps, platform engineers, software testers, Linux C/C++ software engineers and security researchers without Intel x64 assembly language background, beginners learning Linux software reverse engineering techniques, and engineers coming from non-Linux environments. Table of Contents 4 About the Author 10 About the Technical Reviewer 12 Preface 13 Chapter 1: Memory, Registers, and Simple Arithmetic 15 Memory and Registers Inside an Idealized Computer 15 Memory and Registers Inside Intel 64-Bit PC 16 “Arithmetic” Project: Memory Layout and Registers 17 “Arithmetic” Project: A Computer Program 19 “Arithmetic” Project: Assigning Numbers to Memory Locations 19 Assigning Numbers to Registers 22 “Arithmetic” Project: Adding Numbers to Memory Cells 22 Incrementing/Decrementing Numbers in Memory and Registers 25 Multiplying Numbers 28 Summary 31 Chapter 2: Code Optimization 32 “Arithmetic” Project: C/C++ Program 32 Downloading GDB 33 GDB Disassembly Output – No Optimization 33 GDB Disassembly Output – Optimization 38 Summary 39 Chapter 3: Number Representations 40 Numbers and Their Representations 40 Decimal Representation (Base Ten) 41 Ternary Representation (Base Three) 42 Binary Representation (Base Two) 42 Hexadecimal Representation (Base Sixteen) 43 Why Are Hexadecimals Used? 43 Summary 45 Chapter 4: Pointers 46 A Definition 46 “Pointers” Project: Memory Layout and Registers 47 “Pointers” Project: Calculations 49 Using Pointers to Assign Numbers to Memory Cells 49 Adding Numbers Using Pointers 55 Incrementing Numbers Using Pointers 58 Multiplying Numbers Using Pointers 61 Summary 64 Chapter 5: Bytes, Words, Double, and Quad Words 65 Using Hexadecimal Numbers 65 Byte Granularity 65 Bit Granularity 66 Memory Layout 67 Summary 70 Chapter 6: Pointers to Memory 71 Pointers Revisited 71 Addressing Types 71 Registers Revisited 77 NULL Pointers 77 Invalid Pointers 77 Variables As Pointers 78 Pointer Initialization 79 Initialized and Uninitialized Data 79 More Pseudo Notation 80 “MemoryPointers” Project: Memory Layout 80 Summary 91 Chapter 7: Logical Instructions and RIP 92 Instruction Format 92 Logical Shift Instructions 93 Logical Operations 93 Zeroing Memory or Registers 94 Instruction Pointer 95 Code Section 96 Summary 97 Chapter 8: Reconstructing a Program with Pointers 98 Example of Disassembly Output: No Optimization 98 Reconstructing C/C++ Code: Part 1 101 Reconstructing C/C++ Code: Part 2 103 Reconstructing C/C++ Code: Part 3 104 Reconstructing C/C++ Code: C/C++ Program 105 Example of Disassembly Output: Optimized Program 106 Summary 107 Chapter 9: Memory and Stacks 108 Stack: A Definition 108 Stack Implementation in Memory 109 Things to Remember 111 PUSH Instruction 112 POP Instruction 112 Register Review 113 Application Memory Simplified 116 Stack Overflow 116 Jumps 117 Calls 119 Call Stack 121 Exploring Stack in GDB 123 Summary 126 Chapter 10: Frame Pointer and Local Variables 127 Stack Usage 127 Register Review 128 Addressing Array Elements 128 Stack Structure (No Function Parameters) 129 Function Prolog 131 Raw Stack (No Local Variables and Function Parameters) 131 Function Epilog 133 “Local Variables” Project 134 Disassembly of Optimized Executable 137 Summary 138 Chapter 11: Function Parameters 139 “FunctionParameters” Project 139 Stack Structure 140 Function Prolog and Epilog 142 Project Disassembled Code with Comments 143 Parameter Mismatch Problem 147 Summary 148 Chapter 12: More Instructions 149 CPU Flags Register 149 The Fast Way to Fill Memory 150 Testing for 0 151 TEST – Logical Compare 152 CMP – Compare Two Operands 153 TEST or CMP? 154 Conditional Jumps 154 The Structure of Registers 155 Function Return Value 156 Using Byte Registers 156 Summary 157 Chapter 13: Function Pointer Parameters 158 “FunctionPointerParameters” Project 158 Commented Disassembly 159 Summary 168 Chapter 14: Summary of Code Disassembly Patterns 169 Function Prolog/Epilog 169 LEA (Load Effective Address) 172 Passing Parameters 172 Accessing Parameters and Local Variables 173 Summary 174 Index 175