Practical Reverse Engineering x86, x64, ARM, Windows Kernel, Reversing Tools, and Obfuscation

Analyzing how hacks are done, so as to stop them in the future

Reverse engineering is the process of analyzing hardware or software and understanding it, without having access to the source code or design documents. Hackers are able to reverse engineer systems and exploit what they find with scary results. Now the good guys can use the same tools to thwart these threats. Practical Reverse Engineering goes under the hood of reverse engineering for security analysts, security engineers, and system programmers, so they can learn how to use these same processes to stop hackers in their tracks.

The book covers x86, x64, and ARM (the first book to cover all three); Windows kernel-mode code rootkits and drivers; virtual machine protection techniques; and much more. Best of all, it offers a systematic approach to the material, with plenty of hands-on exercises and real-world examples.

• Offers a systematic approach to understanding reverse engineering, with hands-on exercises and real-world examples
• Covers x86, x64, and advanced RISC machine (ARM) architectures as well as deobfuscation and virtual machine protection techniques
• Provides special coverage of Windows kernel-mode code (rootkits/drivers), a topic not often covered elsewhere, and explains how to analyze drivers step by step
• Demystifies topics that have a steep learning curve
• Includes a bonus chapter on reverse engineering tools

Practical Reverse Engineering: Using x86, x64, ARM, Windows Kernel, and Reversing Tools provides crucial, up-to-date guidance for a broad range of IT professionals.

Alexandre Gazet is a senior security researcher at QuarksLab focusing on reverse engineering and software protection.

Elias Bachaalany is a software security engineer at Microsoft.

Introduction xxiii

Chapter 1 x86 and x64 1

Register Set and Data Types 2

Instruction Set 3

Syntax 4

Data Movement 5

Exercise 11

Arithmetic Operations 11

Stack Operations and Function Invocation 13

Exercises 17

Control Flow 17

System Mechanism 25

Address Translation 26

Interrupts and Exceptions 27

Walk-Through 28

Exercises 35

x64 36

Register Set and Data Types 36

Data Movement 36

Canonical Address 37

Function Invocation 37

Exercises 38

Chapter 2 ARM 39

Basic Features 40

Data Types and Registers 43

System-Level Controls and Settings 45

Introduction to the Instruction Set 46

Loading and Storing Data 47

LDR and STR 47

Other Usage for LDR 51

LDM and STM 52

PUSH and POP 56

Functions and Function Invocation 57

Arithmetic Operations 60

Branching and Conditional Execution 61

Thumb State 64

Switch-Case 65

Miscellaneous 67

Just-in-Time and Self-Modifying Code 67

Synchronization Primitives 67

System Services and Mechanisms 68

Instructions 70

Walk-Through 71

Next Steps 77

Exercises 78

Chapter 3 The Windows Kernel 87

Windows Fundamentals 88

Memory Layout 88

Processor Initialization 89

System Calls 92

Interrupt Request Level 104

Pool Memory 106

Memory Descriptor Lists 106

Processes and Threads 107

Execution Context 109

Kernel Synchronization Primitives 110

Lists 111

Implementation Details 112

Walk-Through 119

Exercises 123

Asynchronous and Ad-Hoc Execution 128

System Threads 128

Work Items 129

Asynchronous Procedure Calls 131

Deferred Procedure Calls 135

Timers 140

Process and Thread Callbacks 142

Completion Routines 143

I/O Request Packets 144

Structure of a Driver 146

Entry Points 147

Driver and Device Objects 149

IRP Handling 150

A Common Mechanism for User-Kernel Communication 150

Miscellaneous System Mechanisms 153

Walk-Throughs 155

An x86 Rootkit 156

An x64 Rootkit 172

Next Steps 178

Exercises 180

Building Confidence and Solidifying Your Knowledge 180

Investigating and Extending Your Knowledge 182

Analysis of Real-Life Drivers 184

Chapter 4 Debugging and Automation 187

The Debugging Tools and Basic Commands 188

Setting the Symbol Path 189

Debugger Windows 189

Evaluating Expressions 190

Process Control and Debut Events 194

Registers, Memory, and Symbols 198

Breakpoints 208

Inspecting Processes and Modules 211

Miscellaneous Commands 214

Scripting with the Debugging Tools 216

Pseudo-Registers 216

Aliases 219

Language 226

Script Files 240

Using Scripts Like Functions 244

Example Debug Scripts 249

Using the SDK 257

Concepts 258

Writing Debugging Tools Extensions 262

Useful Extensions, Tools, and Resources 264

Chapter 5 Obfuscation 267

A Survey of Obfuscation Techniques 269

The Nature of Obfuscation: A Motivating Example 269

Data-Based Obfuscations 273

Control-Based Obfuscation 278

Simultaneous Control-Flow and Data-Flow Obfuscation 284

Achieving Security by Obscurity 288

A Survey of Deobfuscation Techniques 289

The Nature of Deobfuscation: Transformation Inversion 289

Deobfuscation Tools 295

Practical Deobfuscation 312

Case Study 328

First Impressions 328

Analyzing Handlers Semantics 330

Symbolic Execution 333

Solving the Challenge 334

Final Thoughts 336

Exercises 336

Appendix Sample Names and Corresponding SHA1 Hashes 341

Index 343

The New copy of this book will include any supplemental materials advertised. Please check the title of the book to determine if it should include any access cards, study guides, lab manuals, CDs, etc.

The Used, Rental and eBook copies of this book are not guaranteed to include any supplemental materials. Typically, only the book itself is included. This is true even if the title states it includes any access cards, study guides, lab manuals, CDs, etc.