Malware Analysis and Detection Engineering : A Comprehensive Approach to Detect and Analyze Modern Malware

Discover how the internals of malware work and how you can analyze and detect it. You will learn not only how to analyze and reverse malware, but also how to classify and categorize it, giving you insight into the intent of the malware. Malware Analysis and Detection Engineering is a one-stop guide to malware analysis that simplifies the topic by teaching you undocumented tricks used by analysts in the industry. You will be able to extend your expertise to analyze and reverse the challenges that malicious software throws at you. The book starts with an introduction to malware analysis and reverse engineering to provide insight on the different types of malware and also the terminology used in the anti-malware industry. You will know how to set up an isolated lab environment to safely execute and analyze malware. You will learn about malware packing, code injection, and process hollowing plus how to analyze, reverse, classify, and categorize malware using static and dynamic tools. You will be able to automate your malware analysis process by exploring detection tools to modify and trace malware programs, including sandboxes, IDS/IPS, anti-virus, and Windows binary instrumentation. The book provides comprehensive content in combination with hands-on exercises to help you dig into the details of malware dissection, giving you the confidence to tackle malware that enters your environment. What You Will Learn Analyze, dissect, reverse engineer, and classify malware Effectively handle malware with custom packers and compilers Unpack complex malware to locate vital malware components and decipher their intent Use various static and dynamic malware analysis tools Leverage the internals of various detection engineering tools to improve your workflow Write Snort rules and learn to use them with Suricata IDS Who This Book Is For Security professionals, malware analysts, SOC analysts, incident responders, detection engineers, reverse engineers, and network security engineers "This book is a beast! If you're looking to master the ever-widening field of malware analysis, look no further. This is the definitive guide for you." Pedram Amini, CTO Inquest; Founder OpenRCE.org and ZeroDayInitiative Table of Contents......Page 4 About the Authors......Page 22 About the Technical Reviewer......Page 23 About the Foreword Author......Page 24 Acknowledgments......Page 25 Foreword......Page 26 Introduction......Page 28 Part I: Introduction......Page 32 Chapter 1: Introduction......Page 33 Types of Malware......Page 35 Platform Diversity......Page 37 The Cyber Kill Chain......Page 38 Malware Attack Life Cycle......Page 40 Development Phase......Page 41 The Adaptive and Deceptive Nature of Malware......Page 42 Distribution Phase: The Diverse Transport System......Page 43 The Malware Business Model......Page 44 The War Against Malware......Page 45 Blogs, Feeds, and Other Shared Sources......Page 46 Incident Responders and Forensic Analysts......Page 47 Detection Teams......Page 48 Firewalls, IDS/IPS, and Network Security Products......Page 49 Terminologies......Page 50 Summary......Page 54 Chapter 2: Malware Analysis Lab Setup......Page 55 Host System Requirements......Page 56 Network Requirements......Page 57 Creating the Malware Analysis VM......Page 58 Disable Hidden Extensions......Page 59 Show Hidden Files and Folders......Page 60 Disable ASLR......Page 62 Disable Windows Defender (or Any Antivirus)......Page 63 Mimic an End-User System......Page 64 Snapshots......Page 65 Tools......Page 66 Hashing Tools: HashMyFiles and Others......Page 68 File Type Identification Tools......Page 69 Autoruns......Page 70 FakeNet......Page 71 YARA......Page 72 Microsoft Network Monitor......Page 73 Malzilla......Page 74 GMER......Page 75 Registry Viewer......Page 76 Cuckoo Sandbox......Page 77 OllyDumpEx......Page 78 x64dbg and Immunity Debugger......Page 79 Summary......Page 80 Part II: OS and System Fundamentals......Page 81 Visualizing a File in Its Native Hex Form......Page 82 Hash: Unique File Fingerprint......Page 84 Identifying Files......Page 86 File Extension......Page 87 File Association: How an OS Uses File Extensions......Page 89 Extension Faking......Page 90 Thumbnail Faking......Page 91 Well-Known File Extensions......Page 92 File Format: The Real File Extension......Page 93 TriD and TriDNet......Page 95 Manual Identification of File Formats......Page 97 Summary......Page 98 Process Creation......Page 99 Executing the Program......Page 100 Exploring the Process with Process Hacker......Page 102 Virtual Memory......Page 104 Memory Pages......Page 107 Demand Paging......Page 109 Page Table......Page 110 Division of Virtual Memory Address Space......Page 111 Inspecting Pages Using Process Hacker......Page 112 Types of Pages......Page 114 States of a Page......Page 115 Page Permissions......Page 116 Strings in Virtual Memory......Page 117 Using Virtual Memory Against Malware......Page 120 Portable Executable File......Page 121 Exploring Windows Executable......Page 122 Endianness......Page 125 Image Base......Page 126 The Catch......Page 127 Relative Virtual Address (RVA)......Page 128 Important PE Headers and Fields......Page 130 File Header......Page 131 NumberOfSections......Page 132 Optional Header......Page 133 Section Data and Section Headers......Page 134 Raw Address......Page 135 Characteristics......Page 136 Windows Loader: Section Data—Virtual Memory......Page 138 Dynamic-Link Library (DLL)......Page 139 Dependencies and Import Tables......Page 142 Dependency Chaining......Page 144 Import Address Table......Page 146 Summary......Page 150 Win32 API......Page 151 Win32 DLLs......Page 152 Studying Win32 API and MSDN Docs......Page 154 Parameters......Page 156 ASCII and Unicode Versions of API......Page 157 Native (NT) Version of the APIs......Page 158 The Undocumented APIs......Page 159 Important APIs to Remember......Page 160 Behavior Identification with APIs......Page 162 Using Handle to Identify Sequences......Page 163 Logical View of Registry......Page 164 Registry Hives......Page 166 Adding Storage to the Registry......Page 167 Altering Registry Information......Page 168 system32......Page 169 User Document and Settings......Page 170 What to Look for as a Malware Analyst......Page 171 Attributes of a Process and Malware Anomalies......Page 172 Sessions (Session ID)......Page 175 Parent Process......Page 176 Number of Instances in a System Process......Page 177 Windows Services......Page 178 Executable Service Under SVCHOST.EXE......Page 180 DLL Services under Svchost......Page 181 Malware as Windows Services......Page 184 Syscall......Page 187 Mutants/Mutex......Page 188 Summary......Page 190 Part III: Malware Components and Analysis......Page 191 Malware Components......Page 192 Payload......Page 193 Packer......Page 194 Communication......Page 195 Armoring......Page 196 Distribution Mechanisms......Page 197 Exploits and Exploit Kits......Page 199 Patches: Fixing Vulnerabilities......Page 200 How Attackers (Mis)Use Exploits......Page 201 Exploit Kit......Page 202 Exploit Kit Components......Page 203 Exploit Kit Flow......Page 204 Exploit Kit as Malware Delivery Mechanism......Page 206 Exploit Kit Case Study......Page 207 Spam......Page 208 Infected Storage Devices......Page 210 Drive-by Download......Page 211 Direct Login via Weak Authentication......Page 212 Shared Folders......Page 213 Summary......Page 215 Encryption and Compression......Page 216 How Packers Work......Page 218 Installers......Page 220 Let’s Pack......Page 221 Comparing Packed and Unpacked Samples......Page 222 Identifying Packed Samples......Page 224 Entropy......Page 225 Static Observation of Strings in a File......Page 226 Dynamic Observation of Strings in Memory......Page 229 Case-Study with Malware......Page 232 Identifying Packers......Page 233 PEiD Tool......Page 234 Code at the Entry Point......Page 235 Section Names......Page 236 Custom Packers......Page 237 Summary......Page 238 Resources Used for Persistence......Page 239 Autoruns......Page 240 ProcMon......Page 241 Startup Shell Directories......Page 243 Registry RUN......Page 247 Services......Page 251 File Infection......Page 254 DLL Hijacking......Page 255 Winlogon......Page 256 Task Scheduler......Page 257 Image File Execution Option (IFEO)......Page 259 SilentProcessExit......Page 260 Summary......Page 262 Why Communicate?......Page 263 CnC Servers, Relays, Proxies, and Malware Networks......Page 266 Fixed Domain Names......Page 268 Domain Flux and DGA......Page 269 Identifying DGA......Page 271 HTTP......Page 273 A Love Affair with HTTPS......Page 276 IRC......Page 277 Other Methods......Page 279 Reconnaissance......Page 280 Stealing Credentials and Weak Passwords......Page 282 Misconfiguration......Page 283 SMB, PsExec, and Others......Page 284 Networking APIs and API Logs with APIMiner......Page 285 IP and Domain Reputation......Page 288 Static Signatures: IDS and Firewalls......Page 289 Anomaly Baselines......Page 290 Summary......Page 291 What Is Code Injection?......Page 292 Hiding......Page 293 Process Piggybacking......Page 294 Altering Functionality......Page 295 Code Injection Target......Page 296 Steps for Code Injection......Page 297 Steps for Process User-Mode Code Injection......Page 298 Step 1: Locating the Target Process......Page 300 Step 2: Allocating Memory in a Remote Target Process......Page 302 Step 3: Writing into Remote Target Memory......Page 306 Section Object and Section Views......Page 308 Remote Thread Creation API......Page 311 Asynchronous Procedure Call (APC) Queues......Page 314 Altering the Thread Context......Page 315 Classical DLL Injection......Page 316 Process Hollowing......Page 322 Classical Shellcode Injection......Page 328 Important APIs to Remember......Page 330 Code/API Hooking......Page 332 Identify Hooking Point/Target......Page 333 Placing Hooks in User Space......Page 335 Inline Hooking......Page 336 Self-Protection......Page 338 Intercept Network Communication......Page 339 Man in Browser Attacks: The Banking Malware......Page 341 Hook Scanning Tools......Page 343 Case Study: DeleteFile() Hooked......Page 346 Case Study: Internet Explorer Hooked......Page 349 APIMiner......Page 352 Summary......Page 354 Chapter 11: Stealth and Rootkits......Page 355 File Properties and Permissions......Page 356 Exercise 1......Page 359 Exercise 2......Page 361 Thumbnail Faking......Page 363 Filename Faking and Extension Faking......Page 365 The Psycholinguistic Technique......Page 366 Hiding Process Window......Page 368 Code Injection......Page 369 User Mode Rootkits......Page 370 Kernel Mode Rootkits......Page 374 Request Flow from User to Kernel......Page 375 Injecting Code into Kernel Space......Page 376 Viewing Loaded Kernel Modules and Drivers......Page 377 SSDT and How to View It......Page 378 Drivers and IRP......Page 380 How to Insert Kernel Modules and Driver?......Page 381 SSDT Rootkits and SSDT Table Hooking......Page 382 SSDT Rootkit Exercise......Page 384 DKOM Rootkits and Kernel Object Manipulation......Page 388 Process Hiding Using DKOM In-Depth......Page 389 DKOM Rootkit Exercise......Page 391 Rootkits Using IRP Filtering or Filter Driver......Page 394 Summary......Page 397 Part IV: Malware Analysis and Classification......Page 398 Why Static Analysis?......Page 399 Sample Hash for Information Xchange......Page 400 Hash Generation......Page 401 Internet, Blogs, and Analysis Reports......Page 402 VirusTotal and Other Analysis Platforms......Page 403 They Say It’s Clean! Is It?......Page 405 Figuring Out the File Format......Page 406 Obtain Full Infection Context......Page 407 Filename Faking and Extension Faking......Page 408 File Thumbnail Faking......Page 410 File Type and File Extension Mismatch......Page 412 Version Information/Details......Page 413 Code Signer Information......Page 414 String Analysis Statically......Page 416 Strings That Indicate Maliciousness......Page 419 YARA......Page 421 Static Fail: Feeder for Dynamic Analysis......Page 423 Summary......Page 424 Keep Your Base Snapshot Handy......Page 425 First Run: A Bird’s-Eye View......Page 426 Whoa! The Sample Refuses to Run!......Page 427 Case Study 1......Page 428 Case Study 2......Page 430 Case Study 3......Page 432 APIMiner: API Log Behavior Identification......Page 434 Classify the Malware Family......Page 437 String Analysis Dynamically......Page 438 Version Information/Details......Page 439 Entropy Check with PEiD......Page 440 Static Observation of Strings in File......Page 441 Dynamic Observation of Strings in Memory......Page 442 ProcMon: Behavior Events Analysis......Page 446 AutoRuns......Page 448 Detecting Code Injection......Page 450 Other Malicious Behavior......Page 451 Disappearing Executable......Page 452 Summary......Page 453 What Are Memory Forensics?......Page 454 Why Another Technique?......Page 455 Memory Acquisition......Page 457 Sample-14-1.mem......Page 458 Sample-14-2.mem......Page 459 Memory Analysis/Forensics......Page 460 Volatility Command Format......Page 464 Image Information......Page 465 pslist......Page 466 pstree......Page 468 psxview......Page 469 Virtual Memory Inspection......Page 470 vadinfo......Page 471 vaddump......Page 473 dlllist......Page 474 ldrmodules......Page 476 Listing Handles......Page 477 mutant......Page 479 Scanning Registry......Page 481 hivelist......Page 482 dumpregistry......Page 483 printkey......Page 487 The malfind Plugin......Page 488 Detecting API Hooks......Page 491 Scanning Kernel Modules......Page 492 Scanning SSDT......Page 494 Network Communication......Page 495 Summary......Page 497 Chapter 15: Malware Payload Dissection and Classification......Page 498 Malware Type, Family, Variant, and Clustering......Page 499 Nomenclature......Page 500 Correct Remediation......Page 502 Intention and Scope of Attack......Page 503 Classification Basis......Page 504 Hooking Keyboard Messages......Page 509 Getting Keyboard Status......Page 511 Information Stealers (PWS)......Page 513 Dynamic Events and API Logs......Page 514 String Analysis of Info Stealers......Page 515 API Logs and Hook Scanners......Page 517 String Analysis on Banking Trojans......Page 519 How POS Devices Work......Page 522 How POS Malware Work......Page 524 Identifying and Classifying POS......Page 525 Strings In POS Malware......Page 526 ATM Malware......Page 529 RATs......Page 530 Identifying RATs......Page 531 Strings in RAT Malware......Page 532 Ransomware......Page 533 Identifying Ransomware......Page 534 Strings in Ransomware......Page 537 Cryptominer......Page 538 Virus (File Infectors)......Page 540 Summary......Page 542 Part V: Malware Reverse Engineering......Page 543 Chapter 16: Debuggers and Assembly Language......Page 544 Reversing and Disassemblers: Source ➤ Assembly ➤ Back......Page 545 PE and Machine Code......Page 546 x86 Assembly Language......Page 548 Instruction: The Format......Page 549 Operand Types and Addressing Mode......Page 552 Endianness......Page 554 Data Registers......Page 555 Pointer Registers......Page 557 Index Registers......Page 558 Flags (Status) Register......Page 559 Stack Operations......Page 561 Basic Arithmetic Instructions......Page 563 Increment and Decrement Instructions......Page 564 Logical Instructions......Page 565 Shift Instructions......Page 566 Comparison Instructions......Page 567 Unconditional Branch Instructions......Page 568 Conditional Branch Instructions......Page 570 Address Loading Instructions......Page 571 Data Movement Instructions......Page 572 String Related Data Movement Instructions......Page 573 MOVS......Page 574 SCAS......Page 575 INT......Page 576 Debuggers and Disassembly......Page 577 OllyDbg vs. IDA Pro......Page 578 Exploring OllyDbg......Page 579 Basic Debugging Steps......Page 583 Stepping Into and Stepping Over......Page 584 Execute Till Return......Page 587 Breakpoint......Page 588 Software Breakpoints......Page 589 Hardware Breakpoints......Page 590 Memory Breakpoint......Page 591 Exploring IDA Debugger......Page 594 Notations in OllyDbg and IDA......Page 600 Local Variable and Parameter Names......Page 601 Undoing Debugger Analysis......Page 602 Identifying The Stack Frame......Page 604 EBP Based Stack Frames......Page 606 Identifying a Function Epilogue and Prologue......Page 607 Identifying Local Variables......Page 608 Identifying Pointers......Page 611 Identifying Global Variables......Page 613 Identifying Array on Stack......Page 615 Identifying Structures on Stack......Page 618 Function Call Parameter Identification......Page 621 Identifying Branch Conditions......Page 624 Identifying Loops......Page 626 Labels and Comments......Page 629 Tracking Variables......Page 631 Skipping Compiler Stub and Library Code......Page 632 Condensing Instructions With Algebra......Page 633 Using Decompilers......Page 634 Blocks and Flowcharts......Page 636 References (XREF)......Page 638 References to API calls......Page 643 Advance Usage of Debuggers......Page 645 Observing API Calls and Parameters......Page 646 Breaking on Win32 APIs......Page 647 Conditional Breakpoints......Page 650 Debugger Events......Page 651 Patching......Page 653 Call Stack......Page 655 Summary......Page 656 Chapter 17: Debugging Tricks for Unpacking Malware......Page 658 OEP and Payload......Page 659 Execution of a Packed Binary......Page 660 Memory Allocation......Page 662 Decompression......Page 663 Import Resolution......Page 664 Jump to OEP......Page 666 The Payload Execution......Page 667 Fast Unpacking Using API Logs and APIMiner......Page 668 Debugging Tricks for Known Packers......Page 670 OllyDumpEx to Dump Payloads......Page 675 Compiler Stubs to Identify OEP......Page 677 Back Tracing......Page 678 Variations in Unpacking Techniques......Page 682 Summary......Page 683 API Logs and Breakpoints......Page 684 IEP: Injection Entry Point......Page 685 Locating IEP with CreateRemoteThread......Page 686 Locating IEP with Thread Context......Page 694 The EBFE Trick......Page 705 Summary......Page 708 Anti-Static Analysis......Page 709 Anti-Dynamic Analysis......Page 710 Identifying Analysis Environment......Page 711 Analysis Tool Identification......Page 713 Virtual Machine Identification......Page 717 Detecting Processor Type......Page 719 Communication Port......Page 721 Anti-Debugging......Page 722 PEB-Based Debugging Detection......Page 723 Using Windows API to Detect Debugger......Page 725 Detect Debugging by Identifying Breakpoints......Page 728 Detect Debugging by Identifying Code Stepping......Page 729 Anti-Disassembly Using Garbage Code......Page 730 Antivirus Evasion......Page 731 Network Security Evasion......Page 732 User Interaction......Page 733 Detecting Agents......Page 734 Timing Attacks......Page 735 Fooling Malware Armoring......Page 736 Open Source Anti-Projects......Page 737 Summary......Page 738 Windows Scripting Environment......Page 739 Obfuscation......Page 741 Hex Equivalents......Page 742 Splits and Joins......Page 743 Inserting Junk......Page 744 Expression Evaluation with eval......Page 745 Encryption Algorithms......Page 746 Deobfuscation......Page 747 Static Deobfuscation......Page 748 Dynamic Deobfuscation......Page 750 Embedded Script Debuggers......Page 752 Downloaders and Droppers......Page 756 Exploits......Page 758 VBScript Malware......Page 759 OLE File Format......Page 761 Dissecting the OLE Format......Page 763 Extracting Streams......Page 765 Automatic Macros......Page 767 Macro Extraction......Page 768 Macro Deobfuscation Using Debugging......Page 770 Windows Management Instrumentation (WMI)......Page 775 Cmdlets and Aliases......Page 779 In-Memory Attacks......Page 782 Summary......Page 784 Part VI: Detection Engineering......Page 786 Linux VM......Page 787 Suricata Setup......Page 789 Windows VM......Page 792 Visual Studio Installation......Page 793 Cygwin Installation......Page 796 Cygwin + Visual Studio......Page 798 Other Tools......Page 799 Summary......Page 800 Main Components of Antiviruses......Page 801 Signatures and Signature Module......Page 803 Signature Categories......Page 805 Hash-Based Signatures......Page 806 Demerits of Hash-Based Detection......Page 807 Hash Signatures Generation Process......Page 808 Generic Signatures......Page 809 Signatures on Disassembly Code......Page 814 Caveats......Page 818 Signature Optimization......Page 819 Risk Minimization......Page 822 File Scanner......Page 823 Unpacker Module......Page 825 Memory Scanner......Page 827 Hook and Rootkit Detection Modules......Page 828 Viral Polymorphism and Emulators......Page 830 Next-Gen Antiviruses......Page 831 Summary......Page 833 Chapter 23: IDS/IPS and Snort/Suricata Rule Writing......Page 834 North-South Traffic......Page 835 Network Traffic Analysis......Page 836 IDS vs. IPS......Page 837 SPAN......Page 838 TAP......Page 839 IPS Traffic Feed......Page 840 Peer Mode or Bridging......Page 841 Deployment Quirks for IDPS Sensors......Page 842 IDPS Components......Page 843 Packet Capture Module......Page 844 Packet Layer Decoding......Page 846 App Layer Parsing......Page 847 Detection Engine......Page 848 Rule Language......Page 849 Yaml Config......Page 850 Running Suricata in PCAP File Mode......Page 852 Rule Writing with Suricata......Page 853 ACTION......Page 854 PROTOCOL......Page 855 SRC_IP and DST_IP......Page 856 DIRECTION......Page 857 Keywords......Page 858 Exercise 1: IP-Only Rule......Page 859 Exercise 2: Content Keyword......Page 860 Exercise 3: Case Matters and Keyword Modifiers......Page 861 Exercise 4: Matching on App Layer Buffers......Page 862 Other Keywords......Page 864 Summary......Page 865 What Is a Malware Sandbox?......Page 866 Why Malware Sandbox?......Page 868 Sandbox In Your Security Architecture......Page 869 Sandbox Design......Page 871 Sample Analysis Workflow......Page 872 Guest......Page 873 Guest VM Mimicking End-User Systems......Page 874 Host Agent......Page 876 Guest Agent......Page 878 API Logger......Page 880 ProcMon and ETW......Page 881 Communication Channel Host Guest......Page 882 Writing Detection on Sandbox Results......Page 883 Machine Learning Using Sandbox......Page 884 Summary......Page 885 What Is Binary Instrumentation?......Page 886 DBI: Terminologies and Internals......Page 889 Inserting Instrumentation Code......Page 891 DBI for Malware Analysis......Page 893 Tool Writing Using DBI......Page 894 Setting up PIN......Page 895 Tool 1: Logging All Instructions......Page 899 Tool 2: Win32 API Logging......Page 903 Tool 3: Code Modification and Branch Bypass......Page 906 Summary......Page 910 Index......Page 911 Discover how the internals of malware work and how you can analyze and detect it. You will learn not only how to analyze and reverse malware, but also how to classify and categorize it, giving you insight into the intent of the malware. Malware Analysis and Detection Engineering is a one-stop guide to malware analysis that simplifies the topic by teaching you undocumented tricks used by analysts in the industry. You will be able to extend your expertise to analyze and reverse the challenges that malicious software throws at you. The book starts with an introduction to malware analysis and reverse engineering to provide insight on the different types of malware and also the terminology used in the anti-malware industry. You will know how to set up an isolated lab environment to safely execute and analyze malware. You will learn about malware packing, code injection, and process hollowing plus how to analyze, reverse, classify, and categorize malware using static and dynamic tools. You will be able to automate your malware analysis process by exploring detection tools to modify and trace malware programs, including sandboxes, IDS/IPS, anti-virus, and Windows binary instrumentation. The book provides comprehensive content in combination with hands-on exercises to help you dig into the details of malware dissection, giving you the confidence to tackle malware that enters your environment. You will: Analyze, dissect, reverse engineer, and classify malware Effectively handle malware with custom packers and compilers Unpack complex malware to locate vital malware components and decipher their intent Use various static and dynamic malware analysis tools Leverage the internals of various detection engineering tools to improve your workflow Write Snort rules and learn to use them with Suricata IDS "This book is a beast! If you're looking to master the ever-widening field of malware analysis, look no further. This is the definitive guide for you." Pedram Amini, CTO Inquest; Fo under OpenRCE.org and ZeroDayInitiative