Sanjivani Rural Education Society’s

Sanjivani College of Engineering, Kopargaon-423 603

(An Autonomous Institute, Affiliated to Savitribai Phule Pune University, Pune)
NAAC ‘A’ Grade Accredited, ISO 9001:2015 Certified

Department of Computer Engineering

(NBA Accredited)

Subject- Digital Forensics (DF) [CO 315A)]

Unit 5 :- Software Reverse Engineering

Prof. Abhijit S. Bodhe

Assistant Professor
Department of Computer Engineering
E-mail :
[email protected]
Contact No: 7709 340 570
 Unit 5:- Software Reverse Engineering
• Software Reverse Engineering Introduction
• Defend against software targets for viruses,
• worms and other malware,
• improving third party software library,
• identifying hostile codes-buffer overflow,
• provision of unexpected inputs.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 2

 Reverse engineering be used to improve the security

• Reverse engineering is the process of analyzing a system, software, or

device to understand how it works and what it does. It can be a
powerful tool for cyber security.
• Reverse engineering can be applied to understand the library's
functionality, identify security vulnerabilities, or undocumented
features that may be exploited.
• Reverse engineering covers a broad range of areas, including
decompiling and disassembling of executable files and libraries, and
analysis of system data.
• Once these vulnerabilities are identified, patches or security
enhancements can be applied.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 3

 Defend against software targets for viruses
• It means using techniques like software reverse engineering to protect
or secure software systems from being attacked, infected, or exploited
by viruses or other malicious programs.
• Defend:- Take actions to protect software from harm or attack.
• Software Targets:- Refers to software systems or applications that
hackers or viruses aim to attack.
• Viruses (Malicious programs designed to):
• Damage or steal data,
• Spread across systems.
• Disrupt normal software behavior.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 4

 Defend against software targets for viruses Involves:

1. Analyzing how viruses work (through reverse engineering).

2. Finding weak points in software that viruses can exploit.
3. Creating defenses, like:
• Antivirus signatures:-A small piece of unique code (like a fingerprint)
used by antivirus software to identify known viruses.
• Security patches.:-An update released by software developers to fix
bugs or security flaws that viruses might exploit.
• Input validation:-A technique to ensure only correct and safe input is
accepted by software.
• Behavior monitoring systems:-Software that watches how programs
behave in real time and blocks suspicious actions.
DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 5
 worms and other malware in SRE
• Worms are self-replicating malware that spread across systems without
user interaction.
• They exploit network vulnerabilities or misconfigurations to move
from one machine to another.
• E.g.SQL Slammer Worm exploited a buffer overflow in Microsoft
SQL Server and spread rapidly over the internet.
• Role of Reverse Engineering (SRE helps in:-)
1. Analyzing how worms spread.
2. Understanding how they avoid detection.
3. Identifying exploits they use (e.g., open ports, unpatched services).

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 6

 Analyze worms and viruses
• A Worm is a form of malware that replicates itself and can spread to
different computers via Network. A Virus is a malicious executable
code attached to another executable file which can be harmless or can
modify or delete data.
• method to handle and analyze worms and viruses found during a
reverse engineering process.
• Start by isolating the malware in a secure, controlled environment. Use
disassembly and debugging tools to analyze the code and understand
its functionality and propagation mechanism. Document the behavior
and develop signatures or heuristic patterns for detection. Finally,
contribute findings to security databases and communities to help
others in detection and mitigation.
DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 7
 worms and other malware in SRE
Types of Malware Studied in SRE:-
1. Viruses – Attach to files or programs; need a host to spread.
2. Worms – Spread independently over networks.
3. Trojans – Disguise as legitimate software.
4. Ransomware – Encrypts data and demands payment.
5. Spyware/Keyloggers – Secretly monitor user activity.
6. Rootkits – Hide malware by altering system processes.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 8

 Techniques Used in Reverse Engineering Malware
1.Static Analysis:-Inspect binary without executing it. Tools Used : IDA Pro.
2. Dynamic Analysis:-Run malware in a sandbox to observe real-time
behavior. Tools Used: Cuckoo Sandbox, Process Monitor.
3.Network Analysis:-Monitor outgoing/incoming connections.
Method:-Reveal/Explore command-and-control (C2) servers.
• Defense Outcomes from RE after malware detection:-
1. Signature development for antivirus.
2. Firewall rules to block worm propagation.
3. Patch identification to close exploited vulnerabilities.
4. Behavior monitoring rules to catch similar malware in future.
DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 9
 Process of identifying a buffer overflow vulnerability
• Buffer overflow is a software coding error or vulnerability that can be
exploited by hackers to gain unauthorized access to corporate systems.
• For example, an attacker may introduce extra code, sending new
instructions to the application to gain access to IT systems.
• The process involves disassembling the application code to analyze how it
handles memory, particularly how it writes data to buffers.
• By examining the assembly instructions, one can identify if bounds
checking is absent or improperly implemented, leading to buffer overflows.
• Buffer overflow vulnerability happens when data written to a buffer
exceeds its size, which may overwrite important data or execute malicious
code

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 10

 Improving third party software library
• It means analyzing, fixing, or enhancing a software library that was
created by someone else (not by you or your team), especially when
you don’t have the source code.
Improving library Means:-
1.Fixing Bugs – If the library crashes or behaves incorrectly.
2.Adding Features – If it lacks a function you need.
3.Boosting Performance – If it's too slow or inefficient.
4.Enhancing Security– If it has vulnerabilities (outdated encryption).
5.Making It Compatible – If it's not working with newer systems or
architectures.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 11

 Improving third party software library
• Why Use Reverse Engineering?:- When you don’t have the source
code,
you can use Software Reverse Engineering (SRE) to:
1. Understand what the library does.
2. Modify or patch the binary.
3. Interact with undocumented functions.
• Example::-You use a third-party image filter library in your app:
1. It’s slow with large images.
2. The source code is not available.
3. You reverse engineer it, find the bottleneck, and patch the binary to
make it faster.
DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 12
 Process to Improving Third-Party Software Libraries
Using SRE(6)
1. Understanding the Library's Functionality (Without Source Code)
• SRE helps you analyze how a third-party library works.
• Useful when source code is unavailable (e.g., closed-source, legacy software).
• Example: Reverse engineering a proprietary DLL to figure out its encryption
method.
2. Fixing Bugs or Vulnerabilities:-
• Identify logic flaws, memory leaks, or buffer overflows in the compiled binary.
• You can patch the binary to fix critical issues.
• Example: A third-party audio codec causes app crashes—using RE, you find and
patch the bug.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 13

 Process to improve TPSL using SRE
3. Performance Optimization:-Analyze slow or inefficient functions using
disassemblers or profilers.
• Replace or rewrite only the slow parts.
• Example: A decompression function in a library is slow—RE helps you optimize
it.
4. Security Enhancements:-Check if the library uses weak cryptography or stores
sensitive data insecurely.
• Modify or wrap around unsafe parts to improve security.
• Example: Replacing hard-coded credentials or weak encryption algorithms.
5. Compatibility and Integration:-RE helps you adapt old or incompatible libraries
for use in modern systems.
• Understand API behavior to build wrappers or bridges.
• Example: Adapting a 32-bit library for use in a 64-bit system.
DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 14
 Process to improve TPSL using SRE
6. Enhancing Documentation
• Many third-party libraries have poor or no documentation.
• By reverse engineering, you can discover function names, parameters, and usage.
• Example: Creating your own documentation from RE analysis of a game engine's
plugin.
• Sample Tools Used to Third Party Software Libraries:-
1. IDA Pro / Ghidra – Disassembly and analysis.
2. x64dbg / OllyDbg – Debugging and patching.
3. Frida / Radare2 – Runtime manipulation and analysis.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 15

 Identifying hostile codes-buffer overflow
• It involves understanding how such attacks work and what indicators
suggest their presence.
• A buffer overflow occurs when a program writes more data to a buffer
(a block of memory) than it can hold. If exploited, this can overwrite
adjacent memory and allow attackers to:
1. Crash programs
2. Execute arbitrary code
3. Escalate privileges

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 16

 Signs of Hostile Buffer Overflow Code
1. Memory Overwrites:-Writing user input directly to memory pointers
or structures without bounds checking. E.g., writing past array
limits.
2. Function Return Address Manipulation:- Assembly or machine code
that manipulates stack pointers or return addresses. E.g. Stack
smashing techniques:-Overwriting return address to jump to
shellcode.
3. No Input Validation:-Functions that use user-supplied input directly
with no size checks or sanitization.
4. Use of Unsafe Functions:- Look for functions that don’t perform
bounds checking.E.g. strcpy(), strcat(), gets().

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 17

 How to Detect Buffer Overflow Exploits
1. Use Static Code Analysis Tools: Flawfinder, RATS, Cppcheck,
Coverity
2. UseDynamic Analysis / Fuzzing: Send random or oversized inputs
and monitor for crashes.
3. Use Memory Protection Tools: AddressSanitizer (-fsanitize=address
in GCC/Clang), Valgrind
4. By Monitoring Logs for Crashes / Core Dumps.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 18

 Buffer flow- hostile behavior exploiting
If you're analyzing malware or suspicious code, reverse engineering
(using tools like Ghidra, IDA Pro, or Radare2) and looking for patterns
like:
1. Shellcode injection
2. Stack pointer manipulation
3. Calls to known bad APIs
• …can also help identify hostile behavior exploiting buffer overflows.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 19

 Provision of unexpected inputs
• It is a classic method for discovering vulnerabilities like buffer
overflows, logic flaws, or improper error handling in software. This
approach is a cornerstone of both fuzz testing and security auditing.
It means supplying input data that the application does not expect or
handle properly—like:
1. Data that is too large, too small, or malformed
2. Non-alphanumeric characters (e.g., \x00, \xFF)
3. Incorrect types (e.g., letters where numbers are expected)
4. Edge cases (empty strings, null pointers, maximum integers, etc.)
• These can lead to: Buffer overflows, Crashes or exceptions,
Unauthorized access or code execution.
DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 20
 Provision of unexpected inputs
• To prevent crashes or vulnerabilities from unexpected input:
1. Always validate input sizes and types.
2. Use safe string handling (strncpy, snprintf).
3. Implement input sanitation and length checks.
4. Use modern compiler protections methods.
• Tools to Automate unexpected inputs are :-
1. AFL (American Fuzzy Lop) – Generates and tests thousands of input
permutations.
2. LibFuzzer – Built-in LLVM fuzzing.
3. Peach Fuzzer, Sulley, etc.
DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 21
 Unit 5:- Software Reverse Engineering
• Software Reverse Engineering Introduction
• Defend against software targets for viruses,
• worms and other malware,
• improving third party software library,
• identifying hostile codes-buffer overflow,
• provision of unexpected inputs.

• Some more relevant point. .

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 22

 Static vs. Dynamic analysis techniques
• Static vs. Dynamic analysis techniques in identifying vulnerabilities in software:-
• Static analysis examines the code without executing it and can uncover
vulnerabilities like buffer overflows and memory leaks.
• Static analysis is a test of the internal structure of the application, rather
than functional testing.
• Dynamic analysis, on the other hand, involves running the software in
controlled conditions to observe its behavior, helping to identify runtime
issues and malicious payloads that only manifest during execution.
• Dynamic analysis adopts the opposite approach of static analysis and is
executed while a program is in operation.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 23

 Hidden malicious code
• These are fragments of code or malicious files hidden within
legitimate websites or third-party scripts or applications.
• step-by-step approach to reverse engineer an executable to find hidden
malicious code.
1. Begin by setting up a secure environment to contain potential malware.
2. Use a disassembler to convert the executable to assembly code.
3. Use static analysis tools (Eg. Fundl and RegCon) to scrutinize the code for
suspicious patterns and signatures.
4. Utilize dynamic analysis by running the program in a sandbox( keeping
potentially malicious program or unsafe code isolated from the rest of the
organization's environment) to observe behavior.
5. Use debugging tools to step through the code and inspect changes in system state
and network activity.
DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 24
 Robustness of a software application
• Software robustness refers to the ability of a software system to continue
functioning correctly and reliably even in the face of unexpected or
abnormal inputs or situations.
• unexpected inputs be used as a test method to determine the robustness of
a software application.
• By designing tests that provide invalid, unexpected, or random data as
inputs to the software, one can observe how the application handles such
inputs.
• This method helps in identifying unhandled exceptions, crashes, and
potential security vulnerabilities like buffer overflows and injection flaws.
• For example, if a system is designed to accept numerical input values between 1 and 10, a robustness
test would involve trying to input values outside of this range, such as 0, 11, or -5, to see how the
system responds
DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 25
 Obfuscation techniques to evade detection
• Obfuscation means to make something difficult to understand.
• Obfuscation is an umbrella term for a variety of processes that
transform data into another form in order to protect sensitive
information or personal data.
• Three of the most common techniques used to obfuscate data
are encryption, tokenization, and data masking.
• Malware authors may use techniques such as packing, encryption, and
polymorphism to obscure malicious code.
• Countermeasures include using advanced static analysis tools that can
deobfuscate code, and employing heuristic and behavior-based
detection systems that do not rely solely on signatures.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 26

 Ethical Implications of reverse engineering software
• This issue is largely debated and does not seem to have a clear cut
answer. Ethically, reverse engineering can be a contentious issue.
• The number one argument against reverse engineering is that
of intellectual property. If an individual or an organization produces a
product or idea, is it ok for others to "disassemble" the product in order
to discover the inner workings?
• It is often justified for finding and fixing security vulnerabilities,
ensuring interoperability, or validating intellectual property rights.
• However, it can also lead to copyright infringement, privacy violations,
and competitive harm if not bounded by legal frameworks and ethical
guidelines.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 27

 Methods to secure software
• Methods to secure software against reverse engineering attacks
• Developers can use a range of techniques to protect against reverse
engineering include using code obfuscation, implementing secure coding
practices to minimize vulnerabilities, employing anti-tamper
mechanisms,
• Techniques also using cryptographic signatures to protect code integrity,
and utilizing hardware-based security solutions like TPM or HSM.
• TPM(Trusted Platform Modules) stores keys securely within your
device, while HSM(Hardware Security Modules) offers dedicated
hardware for key storage, management, backup, and separation of access
control.
• Anti-reverse engineering methods help protect mobile apps from being
reverse-engineered.
DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 28
 Challenges of reverse engineering
• Challenges of reverse engineering in different programming languages
1. Higher-level languages like Java or C# that compile to intermediate
representations (like bytecode) can be easier to reverse engineer than
lower-level languages like C/C++ because tools can reconstruct a
form closer to the original source code.
2. In contrast, compiled binary code of lower-level languages is less
readable and requires more effort to understand.
3. But C is the mostly high-level programming language used in
reverse engineering. As the C/C++ compiler is available for most of
the OS platforms, its programs can be run on all standard platforms.
All these nature of C/C++ made this one of the good programming
languages for hackers and crackers.
DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 29
 Guideline for safely testing malware
• Guidelines should include using isolated environments (sandboxes),
ensuring all network connections are controlled or disabled to prevent
the spread of malware, employing up-to-date antivirus tools to scan and
monitor activities, and maintaining strict access controls.
• To prevent malware attacks,
1. Patch and update software.
2. Use firewalls and security software, such as antimalware and antivirus.
3. Follow email security best practices.
4. Deploy email security gateways.
5. Avoid clicking links and downloading attachments.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 30

 Methods for detecting and defending against
code injection attacks
• Code injection is the term used to describe attacks that inject code into
an application.
• That injected code is then interpreted by the application, changing the
way a program executes.
• Code injection attacks typically exploit an application vulnerability that
allows the processing of invalid data.
• Detection can involve analyzing code for improper input validation and
data handling practices.
• Defense strategies include implementing comprehensive input sanitation
and validation routines, employing least privilege principles, and using
security frameworks that inherently protect against such attacks.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 31

 Role of reverse engineering
• Mostly in the lifecycle of software development and maintenance.
• Reverse engineering can aid in understanding legacy code, recovering lost
documentation, debugging, and interoperability.
• In maintenance, it helps identify parts of the code that need refactoring or
security updates.
1. Cost Reduction: In some cases, reverse engineering can be a cost-
effective alternative to creating products or components from scratch.
2. Repairing existing products. Backwards engineering can help engineers
repair and bend existing products.
3. Competitive Analysis: In business, reverse engineering allows
companies to analyze competitors' products and gain a competitive edge.
DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 32
 Legal and Ethical hacking practices
• reverse engineering can help in legal and ethical hacking practices.
• To begin with, ethical hacking is authorized and aims to improve security.
In contrast, malicious hacking is both illegal and harmful.
• Ethical hackers have permission to test systems, while malicious hackers
exploit vulnerabilities for personal gain or damage.
• In ethical hacking, reverse engineering aids in vulnerability assessments
and penetration testing by understanding how applications handle security
mechanisms.
• Legally, it can help audit compliance with security standards and
intellectual property use.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 33

 Tools like IDA Pro and OllyDbg
• Reverse engineering tools like IDA Pro and OllyDbg enhance software
security analysis.
• These tools help in disassembling binaries to human-readable assembly
code, debugging applications to understand runtime behavior, and
analyzing executable without source code. This analysis is crucial for
uncovering hidden vulnerabilities and ensuring application robustness.
• IDA is primarily geared towards static analysis (though it does have
dynamic debugging functionality)
• OllyDbg (and x32dbg/x64dbg, Immunity Debugger, WinDbg, Cheat
Engine, etc.) is geared towards dynamic analysis.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 34

 Unit 5:- Software Reverse Engineering
• Software Reverse Engineering:
• Defend against software targets for viruses,
• worms and other malware,
• improving third party software library,
• identifying hostile codes-buffer overflow,
• provision of unexpected inputs.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 35

 Unit 6:- Computer crime and Legal issues
• Computer crime and Legal issues: Intellectual property.
• privacy issues.
• Criminal Justice system for forensic.
• audit/investigative.
• situations and digital crime procedure/standards for extraction,
preservation, and deposition of legal evidence in a court of law.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 36