IBMLinuxTechnologyCenter

Managing memory in variable sized chunks

Christopher Yeoh <[email protected]>

Linux.conf.au 2006

2005IBMCorporation

IBMLinuxTechnologyCenter

Outline
Current Linux/K42 memory management Fragmentation problems Proposed system K42 Architecture/Implementation Experiment results Future work

2005IBMCorporation

IBMLinuxTechnologyCenter

Current OS memory management

Physical memory split into frames
Typically OS

4Kbytes

Applications work with virtual addresses

manages
allocation of physical frames loading of frame contents virtual to physical address mapping

2005IBMCorporation

IBMLinuxTechnologyCenter

Virtual memory
Virtual to physical mapping must be very fast
Hardware

support

TLB

Effect of caches degraded as memory sizes increase

Large

pages increase effectiveness of caches

requires physically contiguous memory

2005IBMCorporation

IBMLinuxTechnologyCenter

Causes of fragmentation
Page allocation
Memory

allocated a frame at a time for a process or file

Fragmentation builds up over time as processes allocate

memory and then exit

2005IBMCorporation

IBMLinuxTechnologyCenter

Fragmentation
Process B starts

Process C starts

2005IBMCorporation

IBMLinuxTechnologyCenter

Fragmentation
Process B exits

Minimal fragmentation

2005IBMCorporation

IBMLinuxTechnologyCenter

Problems with fragmentation

Can't allocate large pages Some device drivers need physically contiguous memory More difficult to hotplug memory
virtual

machines as well as real physical memory areas

One solution
Reserved

Fixed size

Balancing pools

2005IBMCorporation

IBMLinuxTechnologyCenter

Problems with Fragmentation

Current Linux approach
3

zones
user reclaimable kernel reclaimable kernel non reclaimable

Fall-back Copy

allocation when a zone is exhausted

memory around in reclaimable areas when too

fragmented

Some device drivers have to reserve contiguous physical

memory
Module

loading can fail

2005IBMCorporation

IBMLinuxTechnologyCenter

Proposed system
Allocation of memory in chunks
Chunks

allocated for processes/files

allocation for process done from chunk when exhausted another chunk allocated

Chunks

can be of variable size

Allocate array of page descriptors which refer to a chunk

2005IBMCorporation

IBMLinuxTechnologyCenter

Chunk allocation

2005IBMCorporation

IBMLinuxTechnologyCenter

Chunk Allocation

2005IBMCorporation

IBMLinuxTechnologyCenter

Chunk Allocation

2005IBMCorporation

IBMLinuxTechnologyCenter

Chunk Allocation

2005IBMCorporation

IBMLinuxTechnologyCenter

Chunk Allocation

2005IBMCorporation

IBMLinuxTechnologyCenter

Chunk Allocation

2005IBMCorporation

IBMLinuxTechnologyCenter

Chunk Allocation

2005IBMCorporation

IBMLinuxTechnologyCenter

Chunk Allocation

2005IBMCorporation

IBMLinuxTechnologyCenter

Chunk Allocation

2005IBMCorporation

IBMLinuxTechnologyCenter

Pros/Cons
Advantages
Naturally Naturally Cache No Operate Increase don't

reduces fragmentation scalable

friendlier data structures on groups of pages potential to promote to large pages

need to continuously initialise/free page descriptor objects

Disadvantages
maybe doesn't matter on PPC (dcbz)

always allocate hot page

2005IBMCorporation

IBMLinuxTechnologyCenter

What is K42
Open source research kernel (64 bit, cache coherent
systems)

Focus on performance, scalability, customizability, Supports Linux API/ABI Uses Linux device drivers, filesystems, ... Userspace servers (NFS, socket, pipe server)
Application

maintainability

level libraries

Pageable kernel data, Userspace thread scheduling, ...

2005IBMCorporation

IBMLinuxTechnologyCenter

Why K42?
Designed for experimentation/prototyping
OO

design easy to add alternative implementations Linux API/ABI (64-bit PowerPC) occurs through per process object

Supports Has

Allocation

infrastructure for experimentation

2005IBMCorporation

IBMLinuxTechnologyCenter

K42 Memory Management System

2005IBMCorporation

IBMLinuxTechnologyCenter

K42 Memory allocation

allocations page allocator, root rep, pmleaf freeing through same places doesn't have to be from original object
2005IBMCorporation

caching of pages in PMRoot and PMRoot reps

IBMLinuxTechnologyCenter

Implementation in K42
Page Descriptor Array (PDA) Header
Page

size, Array Size, Chunk start address of free pages

Bitmap

PDA header and chunk allocated separately allocPage returns address and pointer to PDA
Page

descriptors have a field added to store PDA pointer removed frees done directly to PDA

deallocPage

Under memory pressure fully freed PDAs are freed back to

page allocator
can

re-use partially populated PDAs

2005IBMCorporation

IBMLinuxTechnologyCenter

K42 PDA memory allocation

2005IBMCorporation

IBMLinuxTechnologyCenter

Experimental results
Very preliminary
Implementation Still

still under development

debugging/optimising

Fixed (per boot) size chunks

2005IBMCorporation

IBMLinuxTechnologyCenter

Performance results
Performance
SDET

system benchmark commonly used with scalability testing

0.5% degradation UP Large degradation for SMP

have

not optimised for SMP yet obvious places to fix

some

2005IBMCorporation

IBMLinuxTechnologyCenter

Fragmentation definition
Definition of fragmentation
Measure For For

amount of free memory

a given page size, calculate number of pages that you a given page size, calculate number of pages you can

should be able to allocate allocate

Actual allocate FragmentationPageSize =1 100 Theoretical allocate

2005IBMCorporation

IBMLinuxTechnologyCenter

Example

Each block is 4kb Free memory: 28kb For 8kb pages

theoretical actual 33%

3 x 8kb pages

2 x 8kb pages

fragmentation

2005IBMCorporation

IBMLinuxTechnologyCenter

Fragmentation results
Test load
Long

lived processes

small allocations/deallocations

Short lived processes (forked from long lived ones)

Simple simulation of web server Modified kernel to dump bitmap of all pages in memory
marking free/used state

2005IBMCorporation

IBMLinuxTechnologyCenter

Fragmentation results
Table of fragmentation vs page size Reduced fragmentation for page size <= chunk size Increased fragmentation for page size > chunk size
PageSize(kb) 4 8 16 32 64 128 256 512 Fragmentation% Normal 0 1.3 3.5 7.3 11.8 16.1 17 17.3 Fragmentation PDA(256kb) 0 0.4 1 1.3 1.6 2.1 2.9 64
2005IBMCorporation

IBMLinuxTechnologyCenter

Future work #1
Further debugging/optimisations
SMP

Move more bitmaps into page descriptor array

eg

dirty pages of page descriptor arrays out entire chunks

Handling low memory conditions

Splitting Swapping

2005IBMCorporation

IBMLinuxTechnologyCenter

Future work #2
Variable sized chunks
tailor

size of chunk to process (CPO)

PDAs passed through to FCMs File allocations grouped (PMLeaf equivalent) Reverse mapping in PDAs to point to FCMs
paging/defragmentation

Move technology into Linux

2005IBMCorporation

IBMLinuxTechnologyCenter

Legal Statement

This work represents the view of the author and does not necessarily represent the view of IBM. PowerPC is a registered trademarks of International Business Machines Corporation in the United States, other countries, or both.

Linux is a registered trademark of Linus Torvalds. Other company, product and service names may be trademarks or service marks of others.

2005IBMCorporation