Foundations of ARM64 Linux debugging, disassembling, and reversing : analyze code, understand stack memory usage, and reconstruct original C/C++ code with ARM64

Gain a solid understanding of how Linux C and C++ compilers generate binary code. This book explains the reversing and binary analysis of ARM64 architecture now used by major Linux cloud providers and covers topics ranging from writing programs in assembly language, live debugging, and static binary analysis of compiled C and C++ code. It is ideal for those working with embedded devices, including mobile phones and tablets. Using the latest version of Red Hat, you'll look closely at the foundations of diagnostics of core memory dumps, live and postmortem debugging of Linux applications, services, and systems. You'll also work with the GDB debugger and use it for disassembly and reversing. This book uses practical step-by-step exercises of increasing complexity with explanations and many diagrams, including some necessary background topics. In addition, you will be able to analyze such code confidently, understand stack memory usage, and reconstruct original C/C++ code. And as you'll see, memory forensics, malware, and vulnerability analysis, require an understanding of ARM64 assembly language and how C and C++ compilers generate code, including memory layout and pointers. This book provides the background knowledge and practical foundations you’ll need to understand internal Linux program structure and behavior. Foundations of ARM64 Linux Debugging, Disassembling, and Reversing is the perfect companion to Foundations of Linux Debugging, Disassembling, and Reversing for readers interested in the cloud or cybersecurity. What You'll Learn Review the basics of ARM64 assembly language Examine the essential GDB debugger commands for debugging and binary analysis Study C and C++ compiler code generation with and without compiler optimizations Look at binary code disassembly and reversing patterns See how pointers in C and C++ are implemented and used Who This Book Is For Software support and escalation engineers, cloud security engineers, site reliability engineers, DevSecOps, platform engineers, software testers, Linux C/C++ software engineers and security researchers without ARM64 assembly language background, and beginners learning Linux software reverse engineering techniques. Table of Contents 4 About the Author 10 About the Technical Reviewer 12 Preface 13 Chapter 1: Memory, Registers, and Simple Arithmetic 15 Memory and Registers Inside an Idealized Computer 15 Memory and Registers Inside ARM 64-Bit Computer 16 “Arithmetic” Project: Memory Layout and Registers 17 “Arithmetic” Project: A Computer Program 19 “Arithmetic” Project: Assigning Numbers to Memory Locations 19 Assigning Numbers to Registers 22 “Arithmetic” Project: Adding Numbers to Memory Cells 23 Incrementing/Decrementing Numbers in Memory and Registers 26 Multiplying Numbers 29 Summary 32 Chapter 2: Code Optimization 33 “Arithmetic” Project: C/C++ Program 33 Downloading GDB 34 GDB Disassembly Output – No Optimization 35 GDB Disassembly Output – Optimization 41 Summary 42 Chapter 3: Number Representations 43 Numbers and Their Representations 43 Decimal Representation (Base Ten) 44 Ternary Representation (Base Three) 44 Binary Representation (Base Two) 45 Hexadecimal Representation (Base Sixteen) 46 Why Are Hexadecimals Used? 46 Summary 48 Chapter 4: Pointers 49 A Definition 49 “Pointers” Project: Memory Layout and Registers 50 “Pointers” Project: Calculations 52 Using Pointers to Assign Numbers to Memory Cells 53 Adding Numbers Using Pointers 60 Incrementing Numbers Using Pointers 65 Multiplying Numbers Using Pointers 68 Summary 72 Chapter 5: Bytes, Halfwords, Words, and Doublewords 73 Using Hexadecimal Numbers 73 Byte Granularity 74 Bit Granularity 74 Memory Layout 75 Summary 78 Chapter 6: Pointers to Memory 79 Pointers Revisited 79 Addressing Types 79 Registers Revisited 84 NULL Pointers 84 Invalid Pointers 84 Variables As Pointers 85 Pointer Initialization 85 Initialized and Uninitialized Data 86 More Pseudo Notation 86 “MemoryPointers” Project: Memory Layout 87 Summary 101 Chapter 7: Logical Instructions and PC 102 Instruction Format 102 Logical Shift Instructions 103 Logical Operations 103 Zeroing Memory or Registers 104 Program Counter 105 Code Section 106 Summary 107 Chapter 8: Reconstructing a Program with Pointers 108 Example of Disassembly Output: No Optimization 108 Reconstructing C/C++ Code: Part 1 111 Reconstructing C/C++ Code: Part 2 113 Reconstructing C/C++ Code: Part 3 115 Reconstructing C/C++ Code: C/C++ Program 116 Example of Disassembly Output: Optimized Program 117 Summary 119 Chapter 9: Memory and Stacks 120 Stack: A Definition 120 Stack Implementation in Memory 121 Things to Remember 123 Stack Push Implementation 124 Stack Pop Implementation 124 Register Review 124 Application Memory Simplified 125 Stack Overflow 126 Jumps 127 Calls 128 Call Stack 129 Exploring Stack in GDB 131 Summary 134 Chapter 10: Frame Pointer and Local Variables 135 Stack Usage 135 Register Review 136 Addressing Array Elements 136 Stack Structure (No Function Parameters) 138 Function Prolog 139 Raw Stack (No Local Variables and Function Parameters) 139 Function Epilog 141 “Local Variables” Project 142 Disassembly of Optimized Executable 145 Summary 146 Chapter 11: Function Parameters 147 “FunctionParameters” Project 147 Stack Structure 148 Function Prolog and Epilog 150 Project Disassembled Code with Comments 151 Parameter Mismatch Problem 156 Summary 157 Chapter 12: More Instructions 158 PSTATE Flags 158 Testing for 0 158 TST – Logical Compare 159 CMP – Compare Two Operands 160 TST or CMP? 161 Conditional Jumps 161 Function Return Value 162 Summary 163 Chapter 13: Function Pointer Parameters 164 “FunctionPointerParameters” Project 164 Commented Disassembly 165 Summary 172 Chapter 14: Summary of Code Disassembly Patterns 173 Function Prolog/Epilog 173 ADR (Address) 174 Passing Parameters 174 Accessing Saved Parameters and Local Variables 175 Summary 176 Index 177