The Inside Story On Shared Libraries and Dynamic Loading PDF
The Inside Story On Shared Libraries and Dynamic Loading PDF
90
$ nm hello.o
00000000 T main
U printf
00000000 D x
R_386_32
R_386_PC32
SEPTEMBER/OCTOBER 2001
.rodata
printf
91
SCIENTIFIC PROGRAMMING
Cafe Dubois
The Times, They Are a Changin
Twenty years of schoolin and they put you on the day shift.
Bob Dylan
This summer marks my 25th year at Lawrence Livermore
National Laboratory, all of it on the day shift. LLNL is a
good place to work if you are someone like me who likes to
try new areas, because you can do it without moving to a
new company.
When my daughter was in the fifth grade, she came to
Take Your Daughter to Work Day, and afterwards told me,
referring to the system of community bicycles that you can
ride around on, The Lab is the greatest place in the world
to work. They have free bikes and the food at the cafeteria
is yummy! After that day she paid a lot of attention to her
math and science. Free bikes and yummy food is a lot of
motivation. Shes off to college this year, and I will miss her.
We technical types live in such a constant state of
change, and it is so hard to take the time to keep up. For
each of us, the time will come when we have learned our
last new thing, when we tell ourselves something is not
worth learning when the truth is we just cant take the pain
anymore. So, when I decide not to learn something these
days, I worry about my decision. Was that the one? Is it already too late?
Was it Java Beans? I sure hope it wasnt Java Beans. What
an ignominious end that would be.
F90 pointers
In my article on Fortran 90s space provisions, I didnt
have space to discuss pointers. One reader wrote me about
having performance problems allocating and deallocating
a lot of small objects. So, here is a simple small object
cache module that will give you the idea of how to use
pointers. In this module, one-dimensional objects of size N
or smaller can be allocated by handing out columns of a
fixed cache. The free slots are kept track of through a simple linked list. If the cache fills up, we go to the heap:
module soc
! Allocate memory of size <= N from a xed block.
private
public get, release, init_soc
92
tual library le), the static linker checks for unresolved symbols and reports errors as usual. However, rather than copying the contents of the libraries into the target executable,
the linker simply records the names of the libraries in a list
in the executable. You can view the contents of the library
dependency list with a command such as ldd:
ldd a.out
return
endif
k = rst
rst = links(k)
get => cache(1:s, k)
return
end function get
subroutine release(x)
real, pointer:: x(:)
integer i
if (size(x) > N) then
deallocate(x)
return
endif
do i = 1, M
if (associated(x, cache(1:size(x), i))) then
links(i) = rst
rst = i
return
endif
enddo
deallocate(x)
end subroutine release
end module soc
program socexample
use soc
real, pointer:: x1(:), x2(:), x3(:)
integer i
call init_soc ()
x1 => get(3)
x2 => get(3)
x3 => get(20)
x3 = (/ (i/2., i=1, 20) /)
do i = 1, 3
x1(i) = i
x2(i) = -i
enddo
print *, x1+x2
print *, x3
call release(x2)
call release(x1)
call release(x3)
end program socexample
The input queue is low just now and Id love to hear from
authors about proposed articles. Just email me at paul@
pfdubois.com. And remember, if its Java Beans you want, it
aint me youre lookin for, babe.
SEPTEMBER/OCTOBER 2001
An interesting aspect of shared libraries is that the linking process happens at each program invocation. To minimize this performance overhead, shared libraries use both
indirection tables and lazy symbol binding. That is, the location
of external symbols actually refers to table entries, which remain unbound until the application actually needs them.
This reduces startup time because most applications use
only a small subset of library functions.
To implement lazy symbol binding, the static linker creates
a jump table known as a procedure-linking table and includes it
as part of the nal executable. Next, the linker resolves all unresolved function references by making them point directly
to a specic PLT entry. So, executable programs created by
the static linker have an internal structure similar to that in
Figure 1. To make lazy symbol binding work at runtime, the
dynamic linker simply clears all the PLT entries and sets them
to point to a special symbol-binding function inside the dynamic library loader. The neat part about this trick is that as
each library function is used for the rst time, the dynamic
linker regains control of the process and performs all the necessary symbol bindings. After it locates a symbol, the linker
simply overwrites the corresponding PLT entry so that subsequent calls to the same function transfer control directly to
the function instead of calling the dynamic linker again. Figure 2 illustrates an overview of this process.
Although symbol binding is normally transparent to users,
you can watch it by setting the LD_DEBUG environment
variable to the value bindings before starting your program.
93
SCIENTIFIC PROGRAMMING
Figure 1. The internal structure of
an executable linked with shared
libraries. External library calls
point to procedure-linking table
entries, which remain unresolved
until the dynamic linker fills them
in at runtime.
Source
foo() {
...
malloc();
...
print();
...
}
Executable
Compile/link
foo() {
...
call malloc()
...
call printf
...
}
PLT
Executable
Dynamic linker
foo()
...
call malloc
...
call printf
...
ld.so.1
bindsymbol:
Shared library
libc.so
PLT
malloc: jmp x
printf: jmp ???
malloc:
94
Unresolved
Unresolved
the dynamic linker loads the libraries in the order libfoo.so, libbar.so, libsocket.so, libm.so, and so
forth. If a shared library includes additional library dependencies, those libraries are appended to the end of the library
list during loading. For example, if the library libbar.so
depends on an additional library libspam.so, that library
loads after all the other libraries on the link line (such as
those after libc.so). To be more precise, the library loading order is determined by a breadth-rst traversal of library
dependencies starting with the libraries directly linked to the
executable. Internally, the loader keeps track of the libraries
by placing them on a linked list known as a linkchain. Moreover, the loader guarantees that no library is ever loaded
more than once (previously loaded copies are used when a
library is repeated).
Because directory traversal is relatively slow, the loader does
not look at the directories in /etc/ld.so.conf every time
it runs to nd library les, but consults a cache le instead. Normally named /etc/ld.so.cache, this cache le is a table
that matches library names to full pathnames. If you add a new
shared library to a library directory listed in /etc/ld.
so.conf, you must rebuild the cache le with the ldcong
command, or the loader wont nd it. The -v option produces
verbose detail, including any new or altered libraries.
If the dynamic loader cant find a library in the cache, it
often makes a last-ditch effort with a manual search of system library directories such as /lib and /usr/lib before
it gives up and returns an error. This behavior depends on
the operating system.
You can obtain detailed information about how the dynamic linker loads libraries by setting the LD_DEBUG environment variable to libsfor example,
$ LD_DEBUG=libs a.out
and link an executable with the library, you might get the
following error when you try to run your program:
./a.out: error in loading shared libraries: libfoo.so:
cannot open shared object le: No such le or directory
SEPTEMBER/OCTOBER 2001
form certain preliminary steps. For example, if a C++ program has any statically constructed objects, they must be initialized before program startup. For example,
class Foo {
public:
Foo();
~Foo();
...
}
/* Statically initialized object */
Foo f;
So far, our discussion has focused primarily on the underlying implementation of shared libraries and how they are
organized to support runtime linking of programs. An added
feature of the dynamic linker is an API for accessing symbols
and loading new libraries at runtime (dynamic loading). The
dynamic loading mechanism is a critical part of many extensible systems, including scripting language interpreters.
Dynamic loading is usually managed by three functions
exposed by the dynamic linker: dlopen() (which loads a
new shared library), dlsym() (which looks up a specific
symbol in the library), and dlclose() (which unloads the
library and removes it from memory)for example,
void *handle;
void (*foo)(void);
95
SCIENTIFIC PROGRAMMING
Figure 3. Linkchains created by
dynamic loading. Each dynamically
loadable module goes on a new
linkchain but can still bind to symbols
defined in the primary executable.
Dynamic module 1
mod1.so
libd.so
Dynamic module 2
Executable
mod2.so
a.out
liba.so
libb.so
libc.so
Dynamic module 3
mod3.so
libe.so
Dynamic linking enables modes of linking that are not easily achieved with static libraries. One such example is the implementation of lters that let you alter the behavior of common library functions. For example, suppose you want to
track calls to dynamic memory-allocation functions malloc() and free(). One way to do this with dynamic linking is to write a simple library as in Figure 4. In the gure, we
implemented replacements for the standard memory man-
96
libf.so
agement functions. However, these replacements use dlsym() to search for the real implementation of malloc()
and free() farther down the linkchain.
To use this lter, the library can be included on the linkline
like a normal library. Alternatively, the library can be preloaded
by supplying a special environment variable to the dynamic
linker. The following command illustrates library preloading by
running Python with our memory-allocation lter enabled:
$ LD_PRELOAD=./libmlter.so python
Depending on how you tweak the linker, the user can place
lters almost anywhere on the linkchain. For instance, if the
lter functions were linked to a dynamically loadable module,
only the memory allocations performed by the module pass
through the lters; all other allocations remain unaffected.
Constructing shared libraries
/* libmlter.c */
#include <dlfcn.h>
#include <stdio.h>
#include <assert.h>
typedef void *(*malloc_t)(size_t nbytes);
void *malloc(size_t nbytes) {
void *r;
static malloc_t real_malloc = 0;
if (!real_malloc) {
real_malloc = (malloc_t) dlsym(RTLD_NEXT, malloc);
assert(real_malloc);
}
r = (*real_malloc)(nbytes);
printf(malloc %d bytes at %x\n, nbytes, r);
return r;
}
scripting language interpreters and other extensible systems, it is common to create various
types of modules for different parts of an application. However, if parts of the application are
built as static libraries (.a les), the linking
process does not do what you might expect.
Specically, when a static library is linked into
a shared module, the relevant parts of the static
library are simply copied into the resulting
typedef void (*free_t)(void *ptr);
shared library object. For multiple modules
void free(void *ptr) {
linked in this manner, each module ends up
static free_t real_free = 0;
with its own private copy of the static library.
if (!real_free) {
When these modules load, all private library
real_free = (free_t) dlsym(RTLD_NEXT, free);
copies remain isolated from each other, owing
assert(real_free);
}
to the way in which symbols are bound in dyprintf(free %x\n, ptr);
namic modules (as described earlier). The end
(*real_free)(ptr);
result is grossly erratic program behavior. For
}
instance, changes to global variables dont seem
to affect other modules, functions seem to operate on different sets of data, and so on. To x
Figure 4. Filters for malloc and free. The filters look up the real
this problem, convert static libraries to shared implementation using dlsym() and print debugging information.
libraries. Then, when multiple dynamic modules link again to the library, the dynamic linker
will guarantee that only one copy of the library loads into 3. R.A. Gingell et al., Shared Libraries in SunOS, Usenix, San Diego, Calif.,
1987.
memory.
4. J.Q. Arnold, Shared Libraries on UNIX System V, Usenix, San Diego, Calif.,
1986.
References
1. J.R. Levine, Linkers & Loaders, Morgan Kaufmann, San Francisco, 2000.
SEPTEMBER/OCTOBER 2001
97