3761

Security Core Lecture

05 Software Security II – Stack
and Calling Conventions
Prof. Dr. Thorsten Holz | 07.11.2023
 3761

Recap: QUIZTIME: Conditional Jump #1 (ex12.asm)

§ When is ebx set to 1?

Assembly
cmp eax, 10
jge a1
jmp a2
a1:
mov ebx, 1
a2:
nop

2
 3761

Recap: QUIZTIME: Conditional Jump #1 (ex12.asm)

§ When is ebx set to 1?

Assembly
cmp eax, 10
jge a1
jmp a2
a1:
mov ebx, 1
a2:
nop

jge = jump greater or equal

Answer B is correct
3
 3761

Recap: QUIZTIME: Conditional Jump #2 (ex12.asm)

§ What is the equivalent C code for this snippet?

(Assume rbx is variable b and rax is variable a; both are
signed 64-bit integers)

Assembly
mov rcx, 1
shl rcx, 63
test rax, rcx
jz b1
mov rbx, 1
b1:
nop

4
 3761

Recap: QUIZTIME: Conditional Jump #2 (ex12.asm)

§ What is the equivalent C code for this snippet?

(Assume rbx is variable b and rax is variable a; both are
signed 64-bit integers)

Assembly
mov rcx, 1
shl rcx, 63
test rax, rcx
jz b1
mov rbx, 1
b1:
nop

test: bit-wise AND of two values

Answer B is correct 5
 3761

Recap: QUIZTIME: Conditional Jump #3 (ex12.asm)

§ Can this code be simplified (still using the test

instruction)?

Assembly
mov rcx, 1
shl rcx, 63
test rax, rcx
jz b1
mov rbx, 1
b1:
nop

6
 3761

Recap: QUIZTIME: Conditional Jump #3 (ex12.asm)

§ Can this code be simplified (still using the test

instruction)?

Assembly
mov rcx, 1
shl rcx, 63
test rax, rcx
jz b1
mov rbx, 1
b1:
nop

We could rewrite the code to not use JZ

but Jump if (not) negative
7
 3761

This Lecture

§ Last lectures
- Control flow operations
- (un)conditional jumps
- Loops and similar code constructs
§ This lecture
- Stack and push/pop instructions
- Calling conventions for x64 / x86

8
 3761

Functions and the Stack

 3761

Stack

§ Stack: Memory in LIFO layout

- Local variables, function arguments, return values, ...
§ Two basic assembly primitives:
- push: Add data <op> to top of stack (and reduce rsp by number of bytes)
- pop: Remove data from top of stack into register <op> (and increase rsp)
§ rsp points to top of stack (i.e., to last element on stack)

10
 3761

Stack II

§ push/pop examples

Assembly
SECTION .data
myvar dq 0xcafe;

SECTION .text
lea rax, [myvar] ; rax contains address of myvar
push 0xcabe ; push constant 0x000000000000CABE on stack
push rax ; push value in rax on stack

pop rax ; pop into rax (rax := rax)

pop qword [myvar] ; pop into variable (myvar := 0xCABE)

; default push/pop size is 8B in x64, but 2B/4B can be specified

push ax ; push value in ax on stack (RSP –= 2)
push word 0xcabe ; push constant 0xCABE on stack (RSP -= 2)

11
 3761

Challenges for Calling Functions

§ One functions calls another and passes arguments to the callee - how is the
communication handled?

Code
int foo (int var1, char *var2) {
…
tmp = bar (23, 42);
…
}

12
 3761

Function Calls

§ Invoke function: call fun

- call implicitly performs these two operations:
1. Store return address on stack (next instr.):
push rip (implicitly rsp -= 8 in x64)
2. Change control flow to callee: jmp fun
§ Return to caller: ret (<op>)
- Return value stored in rax
- ret instruction implicitly performs these operations:
1. Optional: remove <op> bytes from stack (cleanup; see later)
2. Restore return address from stack: pop saved rip (implicitly rsp += 8)
3. Change control flow back to caller: jmp rip

13
 3761

Linux Calling Convention in x64

§ First 6 parameters passed via registers: rdi, rsi, rdx, rcx, r8, r9
§ Additional parameters are passed on the stack
§ Return value in rax

§ Who is responsible for saving registers? Fixed in calling convention

§ Non-volatile, callee-saved registers: rbp, rbx, r12-r15, rsp
- The called function (callee) has to save/restore registers before/after
modification
- Caller can rely on the fact that the callee does not alter the registers
§ Volatile, caller-saved registers: all other registers
- Callee may alter registers at free will
- If still required, calling function needs to store registers before call and
restore after call
14
 3761

Example

https://godbolt.org/z/9WSBds

15
 3761

Memory Layout: Stack in x64

§ Call stack (stack frame) is subroutine’s perspective on stack, includes the

following info
- arguments (parameter values) passed to routine (if any);
- return address back to the routine’s caller
- space for local variables (if any).
- Stack frame is 16B aligned in x64, 8B aligned in x86

§ Stack grows to lower addresses

- rsp: top of stack (stack pointer)
- rbp: pointer into current frame (base pointer)

16
 3761

Memory Layout: Stack in x64 II

§ Contents of specific addresses

- rbp: caller’s RBP
- rbp+8: return address
- rbp+16: 7th parameter
- Negative displacement to rbp: access to local
variable
- below rsp: ephemeral scratch space
§ Stack’s base address
- Loader determines memory location of data section
of program (see PE/ELF header)
- Usually the stack is high up, but not adjacent to
0x07FF...FF
17
 3761

QUIZTIME: Stack

§ Assembly Challenge
- Does push change rsp?

18
 3761

QUIZTIME: Stack

§ Assembly Challenge
- Does push change rsp?

Answer B is correct
19
 3761

QUIZTIME: Function Parameters

§ Assembly Challenge
- How many parameters can be
passed to a function in x64?

20
 3761

QUIZTIME: Function Parameters

§ Assembly Challenge
- How many parameters can be
passed to a function in x64?

Slide 14:
First 6 parameters passed via registers:
rdi, rsi, rdx, rcx, r8, r9
Additional parameters are passed on
the stack

Answer C is correct

21
 3761

Memory Layout: Stack in x64 (Example)

§ Stack layout after function prologue of summing()

C Code
uint64_t summing(uint64_t a, uint64_t b,
uint64_t c, uint64_t d,
uint64_t e, uint64_t f,
uint64_t g, uint64_t h) {

uint64_t x = a + b + c;
uint64_t y = d + e + f;
uint64_t z = g + h;

uint64_t s = x + y + z;
return s;
}

int main(int argc, char* argv[]) {

summing(1, 2, 3, 4, 5, 6, 7, 8);
nxt return 0; RDI=1, RSI=2, RDX=3,
} RCX=4, R8=5, R9=6
22
 3761

Memory Layout: Setting Up the Stack

§ Function prologue
1. Save old rbp: push rbp
2. Establish frame: mov rbp, rsp
3. Reserve stack space for local vars: sub rsp, <size_local_vars>

§ Function epilogue
1. Restore old frame (alternative: leave)
• mov rsp, rbp
• pop rbp
2. Return to caller (ret)

23
 3761

Memory Layout: Stack in x64 (Example)

C Code
void funA(int a, int b) {
uint64_t vA, vB;
vA = vA + vB;
}

void funB(int x, int y) {

uint64_t v1,v2,v3,v4;
funA(v1,v4+v3);
nB: v2=v1+v3+v4;
}

int main(...) {
funB(1, 2);
nM: return 0;
}

24
 3761

QUIZTIME: Function Calls

§ Assembly Challenge
- Suppose fun_A calls fun_B
- How does the stack frame (i.e., the
registers rbp and rsp) of fun_A look like
after fun_B has returned to fun_A?
- Assume x64 calling convention

25
 3761

Memory Layout: Frame Pointer Omission (FPO)

§ rbp is not mandatory

- Addressing can be done relative to rsp
- rsp-relative addressing is fairly common in x64

§ Implementation
- Flag –fomit-frame-pointer in gcc (enabled in O1-3) or /Oy in VC does
this
- All stack addresses are relative to rsp
- rbp can be used as a general-purpose register
- Function prologue omits mov rbp, rsp
- Function epilogue omits pop rbp

26
 3761

Calling Conventions in x86

§ x86 (32-bit) has many different calling conventions

§ Vary in how to pass parameters
- Parameters in registers vs. parameters in stack
- Order of parameters in stack
- Set of registers of parameters
- Set of registers that need to be saved by callee

27
 3761

Calling Conventions in x86 (Example)

§ cdecl example

C Code Assembly (int main(), caller)

int addall(int a, int b, int c) { public main
int x;
x += a + b + b + c + c + c; ...
return x; push ebp ; store ebp
} mov ebp, esp ; set ebp
push ecx ; store register ecx
int main() { push 3 ; push 3x4 = 0x0C bytes
addall(1, 2, 3); push 2
return 0; push 1
} call addall ; call addall()
add esp, 0x0C. ; clean up stack
pop ecx ; restore ecx
mov eax, 0. ; return value
...

28
 3761

Memory Layout: Stack in x86 (Example)

Assembly (int main(), caller)
§ Stack layout (cdecl)
; in main
01: push 0x3
02: call recurse
C Code 03: add esp,0x4
uint32_t recurse(uint32_t a) {
; in recurse
if (a <= 1) {
04: push ebp
return 1;
05: mov ebp,esp
} else { 06: push ebx
return a+recurse(a-1); 07: mov ebx,[ebp+0x8]
} 08: cmp ebx,0x1
fin: 09: mov eax,0x1
} 10: jbe end_recurse
11: lea eax,[ebx-0x1]
12: push eax
int main(...) {
13: call recurse
recurse(3);
14: add esp,0x4
fin: 15: add eax,ebx
} 16: end_recurse:
17: pop ebx
18: leave
19: ret
29
 3761

Memory Layout: Stack in x86 (Example) II

30
 3761

Stack Setup in x86 (32b_ex02.asm)

§ push vs. sub and mov

- push X adds X to stack
§ So does: sub esp, 4 and mov [esp], X
Assembly Code
; cdecl: sum_three_numbers(1, 2, 3)
push 3
push 2
push 1
call sum_three_numbers
add esp, 0x0c ; 3*4 = 12 bytes

; alternative using sub esp and moves instead of push

sub esp, 0x0c
mov [esp+0], dword 1
mov [esp+4], dword 2
mov [esp+8], dword 3
call sum_three_numbers
add esp, 0x0c ; 3*4 = 12 bytes 31
 3761

System Calls using the Linux x64 ABI

§ Calling convention like normal function calls

- 4th parameter is different (r10 instead of rcx), thus:
rdi, rsi, rdx, r10, r8, r9
- System call number placed in rax
- Return value in rax
§ syscall instruction initiates system call
- Usually wrapped by system libraries (e.g., libc)
- Could be used, in principle, by program itself (or by shellcode)

32
 3761

System Calls using the Linux x64 ABI II

§ Example system call

Assembly Code
; sys_exit(42)
mov rax,60 ; system call number (sys_exit)
mov rdi,42 ; system call return value
Syscall

; sys_write(stdout, buf, buflen)

mov rax,1 ; system call number (1 = sys_write)
mov rdi,1 ; arg1: fd (1 = stdout)
mov rsi,buf ; arg2: buffer
mov rdx,[buflen]. ; arg3: buflen
Syscall

§ Good overview of system calls:

- http://blog.rchapman.org/posts/Linux_System_Call_Table_for_x86_64/
- https://github.com/torvalds/linux/blob/master/arch/x86/entry/syscalls/sysc
all_64.tbl
33