IPDPS 2010 @ Atlanta

GenerOS: An Asymmetric
Operating System Kernel for
Multi-core Systems

Authors:
Qingbo Yuan, Jianbo Zhao, Mingyu Chen, Ninghui Sun
Institute of Computing Technology
Chinese Academy of Sciences
[email protected]

Speaker:
SuZhen Wu
Huazhong University of Science & Technology
Outline

1 Motivation

2 Architecture of GenerOS

3 Implementation of GenerOS

4 Evaluation of GenerOS vs Linux

5 Conclusion

GenerOS | 2
Motivation

Symmetric multithread operating system such as Linux

suffers from lock contention and cache pollution
Lock contention
 As more cores are packaged into a single chip, there are two many
cores in a system
 Each core has the ability to trap into kernel
 Too many procedures in kernel -> serious lock contention
Cache pollution
 Applications and kernel run on the same core
 Applications may kick kernel’s cache line out of cache
 And vice versa

GenerOS | 3
Motivation
---- Lock Contention @ Linux
Contention Probability = contentions / acquisitions
 acquisitions: times acquiring lock
 contentions: times encountering contention

Contention Efficiency = hold time / (hold time + wait time)

 hold time: time in critical region
 wait time: time waiting for entering critical region

GenerOS | 4
Motivation
---- Lock Contention @ Linux
Contention Probability Contention Efficiency
0.18% 100%

0.16% 90%

80%
0.14%
70%
0.12%
60%
0.10%
50%
0.08%
40%
0.06%
30%
0.04%
20%

0.02% 10%

0.00% 0%
1 4 16 64 256 1024 1 4 16 64 256 1024
Thread Number Thread Number

GenerOS | 5
Motivation
---- Cache Pollution @ Linux
100% 96%
92%

80%

60%
Kernel Proportion

60%

42%
40%

20%

0%
Dcache Icache Dcache Icache
Miss Ratio Miss Ratio Lines Evicted Lines Evicted

GenerOS | 6
Motivation

Lock Contention Solution

More Cores Decrease Cores

More Contentions in Kernel Mode
GenerOS

Cache Pollution Solution

Applications Separate Kernel

Run Together and Applications
with OS

GenerOS | 7
Outline

1 Motivation

2 Architecture of GenerOS

3 Implementation of GenerOS

4 Evaluation of GenerOS vs Linux

5 Conclusion

GenerOS | 8
Architecture

In a symmetric multiprocessing system, Linux treats all cores

as an equal which causes a lot of problems

By contrast, GenerOS partitions processing cores into

application core, kernel core and interrupt core
 All of applications run on application core
 Their system calls are executed by kernel core
 Interrupts are all bound to interrupt core

GenerOS | 9
Architecture

Most of cores are used by applications

A limited number of cores are used by kernel service
 File System
 Process
Few number of cores are used to handle interrupt

GenerOS | 10
Outline

1 Motivation

2 Architecture of GenerOS

3 Implementation of GenerOS

4 Evaluation of GenerOS vs Linux

5 Conclusion

GenerOS | 11
Implementation

GenerOS is developed based on Linux-2.6.25 @ x86_64 architecture

In system call level, several kernel servers are developed
 File system server (98 system calls)
sys_open / sys_close / sys_read / sys_write
 Network server (15 system calls)
sys_socket / sys_connect
 Signal server (12 system calls)
sys_rt_sigaction
 IPC server (12 system calls)
sys_msgget
 Process server (10 system calls)
sys_fork
 Others (141 system calls)
sys_brk

GenerOS | 12
Implementation
---- GenerOS Processing Flow Chart
Application @ Runtime @ Application core
Application core
pid_t generos_sys_getpid(void)
int main(void) {
{ req = generos_get_request();
pid_t getpid(); generos_init_request(req);
generos_send_to_kernel(req);
return 0; sleep();
} return pid;
}

Process Server @ Kernel core

while(generos_request_queue_is_not_empty(&process_queue)){
req = generos_pick_request(&process_queue);
switch(req->type){
case GETPID:
req->retvalue = sys_getpid();
break;
……
}
wake_up_process(req->task);
}

GenerOS | 13
Implementation
---- Runtime at Application Core
It replaces the system call table of Linux
const sys_call_ptr_t syscall_table [__NR_syscall_max+1] = {
[__NR_read] = &generos_sys_read,
[__NR_write] = &generos_sys_write,
……
[__NR_timerfd_gettime] = &generos_sys_timerfd_gettime;
};
The left side keeps the same meaning with Linux which
makes GenerOS compatible with Linux
The right side uses self defined function which will find a
kernel core to handle its system call

GenerOS | 14
Implementation
---- Kernel Core
Two queues
 Request queue
Receive system call requests from application core
 Wait queue
Store the being handled system calls which are waiting for
something
One schedule method
 Slim Schedule
Schedule system calls in this kernel core with almost zero overhead

GenerOS | 15
Implementation
---- Binding Interrupt Handler
Interrupt core is used to deal with most of interrupts from
network interface, disk, or local timer
In such way, both of application core and kernel core will
have a clean execution environment
GenerOS uses the method in Linux to bind interrupt handler
to some processing core

GenerOS | 16
Outline

1 Motivation

2 Architecture of GenerOS

3 Implementation of GenerOS

4 Evaluation of GenerOS vs Linux

5 Conclusion

GenerOS | 17
Evaluation
---- Platform

GenerOS | 18
Evaluation
---- Lock contention
linux generos linux generos

0.18% 100%

0.16%

80%
0.14%

0.12%
Contention Probability

Contention Efficiency
60%
0.10%

0.08%
40%
0.06%

0.04%
20%

0.02%

0.00% 0%
1 4 16 64 256 1024 1 4 16 64 256 1024
Thread Numbers Thread Numbers

GenerOS | 19
Evaluation
---- Cache Pollution
linux generos

100% 96%
92% 92%

83%
80%

60%
Kernel Proportion

60%

44% 42%
40% 37%

20%

0%
Dcache Icache Dcache Icache
Miss Ratio Miss Ratio Lines Evicted Lines Evicted

GenerOS | 20
Evaluation
---- Single System Call

GenerOS | 21
Evaluation
---- Single System Call
l-enter l-handle l-exit flyin flyout l-enter l-handle l-exit flyin flyout
g-enter g-flyin g-handle g-flyout g-exit g-enter g-flyin g-handle g-flyout g-exit
50 25

45

40 20

35

30 15

million-cycles
kilo-cycles

25

20 10

15

10 5

0 0
open close read write

GenerOS | 22
Evaluation
---- TPC-H 1GB Power
700

611
600
559 554
511 522 517
500
450 455
The bigger the better
TPC-H Power@1G

400

300

200

100

0
Linux g-8000 g-8800 g-8880 g-8888 g-c000 g-e000 g-f000

GenerOS | 23
Evaluation
---- TPC-H 1GB Power

GenerOS | 24
Evaluation
---- Httperf
450 linux g-8000-8000 g-8000-800

400

350

300
the more the better
Replies Per Second

250

200

150

100

50

0
200 400 600 800 1000
Requests Per Second

GenerOS | 25
Outline

1 Motivation

2 Architecture of GenerOS

3 Implementation of GenerOS

4 Evaluation of GenerOS vs Linux

5 Conclusion

GenerOS | 26
Conclusion

GenerOS is an asymmetric kernel which is designed to deal

with the problems faced in traditional symmetric kernel
Being compatible with Linux, GenerOS does not need to
modify, recompile, or relink applications, or libraries
Experiments with two typical workloads on 16-core AMD
machine show that GenerOS behaves better than original
Linux kernel when there are more processing cores
 19.6% for TPC-H using oracle database management system
 42.8% for httperf using apache web server

GenerOS | 27
Thank you very much!

Any question ?

Please contact the author

[email protected]
GenerOS | 28