What is Cross Compilation ? Cross Compilation Demystified by cawan (cawan[at]ieee.org or chuiyewleong[at]hotmail.

com) on 11/11/2012 Cross compilation is a very important topic in embedded linux system design. It is about to use a host machine to build a software which is going to be run in embedded system. Besides, it is also about to harness the power of open source technology for embedded applications. The compiler that is going to be used in cross compilation is known as cross compiler. So, the cross compiler will be run in a standard x86 PC as host machine, and generate binary for embedded platform such as ARM, MIPS, or PPC. Let's illustrate with example. Host:# cat hello.c #include<stdio.h> main() { printf("hello world\n"); } Host:# gcc -o hello-x86 hello.c Host:# file hello-x86 hello-x86: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.15, dynamically linked (uses shared libs), not stripped Host:# ./hello-x86 hello world Host:# Well, we have a hello.c and we compile it with default gcc to generate a x86 compliant elf executable which is able to run on x86 based platform, as shown. Nothing special here. However, for cross compilation, we are going to use platform dependent cross compiler instead of the default gcc. As demonstration here, we are going to use a MIPS based cross compiler to generate a executable which is able to run on MIPS platform. Let's start. Host:# ./mips-linux-gnu-gcc -EL -o hello-mips hello.c Host:# file hello-mips hello-mips: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1, for GNU/Linux 2.6.12, dynamically linked (uses shared libs), not stripped Host:# ./hello-mips bash: ./hello-mips: cannot execute binary file Host:# In this case, the mips-linux-gnu-gcc is the MIPS based cross compiler. The -EL flag is to tell cross compiler to generate code which is compliant to little endian MIPS architecture. Without the -EL flag, the cross compiler will generate default big endian MIPS binary. Of course, when we try to run it on our x86 based host machine, it should fail. However, if we run the executable in a MIPS platform, it should work as expected. tango3[~]# ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1): 56 data 64 bytes from 192.168.1.1: seq=0 ttl=64 64 bytes from 192.168.1.1: seq=1 ttl=64 64 bytes from 192.168.1.1: seq=2 ttl=64

bytes time=1.942 ms time=1.090 ms time=1.049 ms

--- 192.168.1.1 ping statistics --3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 1.049/1.360/1.942 ms

tango3[~]# mkdir ./cawan tango3[~]# mount -o nolock 192.168.1.1:/home/smp383/mips-4.3/bin/ ./cawan tango3[~]# cd cawan tango3[cawan]# ls -l hello-mips -rwxr-xr-x 1 root root 5870 Nov 10 2012 hello-mips* tango3[cawan]# ./hello-mips hello world tango3[cawan]# The 192.168.1.1 is host machine, when ping it from embedded system, it is alive. Then, we create a local directory in the embedded system and mount it onto a development directory at remote host machine by using NFS. The -o nolock is normally added in embedded environment for compatibility issues. From development directory of host machine, we found our hello-mips, and once we run it in MIPS console, it works well as expected. How about the hello-x86 ? Let's verify. tango3[cawan]# ./hello-x86 -sh: ./hello-x86: cannot execute binary file tango3[cawan]# Yes, it should not able to run in MIPS platform. How about big endian MIPS binary ? Host:# ./mips-linux-gnu-gcc -o hello-mips-be hello.c Host:# ls -l hello-mips-be -rwxr-xr-x 1 root root 5870 2012-11-11 14:06 hello-mips-be Host:# file hello-mips-be hello-mips-be: ELF 32-bit MSB executable, MIPS, MIPS32 rel2 version 1, for GNU/Linux 2.6.12, dynamically linked (uses shared libs), not stripped Host:# tango3[cawan]# ./hello-mips-be -sh: ./hello-mips-be: cannot execute binary file tango3[cawan]# Yes, the big endian MIPS binary should not able to run in little endian MIPS platform. Well, that is simple. But, how to cross compile an open source project in order to let it run in our MIPS platform ? For this issue, we need to know more about make tool, and we need to instruct the make tool to invoke the appropriate cross compiler instead of the default gcc. Let us demonstrate with an open source project such as Lame MP3 Encoder. Host:# pwd /home/smp383/mips-4.3/bin/lame-3.99.5 Host:# make --help Usage: make [options] [target] ... Options: -b, -m Ignored for compatibility. -B, --always-make Unconditionally make all targets. -C DIRECTORY, --directory=DIRECTORY Change to DIRECTORY before doing anything. -d Print lots of debugging information. --debug[=FLAGS] Print various types of debugging information. -e, --environment-overrides Environment variables override makefiles. -f FILE, --file=FILE, --makefile=FILE Read FILE as a makefile. -h, --help Print this message and exit. -i, --ignore-errors Ignore errors from commands. -I DIRECTORY, --include-dir=DIRECTORY

Search DIRECTORY for included makefiles. -j [N], --jobs[=N] Allow N jobs at once; infinite jobs with no arg. -k, --keep-going Keep going when some targets can't be made. -l [N], --load-average[=N], --max-load[=N] Don't start multiple jobs unless load is below N. -L, --check-symlink-times Use the latest mtime between symlinks and target. -n, --just-print, --dry-run, --recon Don't actually run any commands; just print them. -o FILE, --old-file=FILE, --assume-old=FILE Consider FILE to be very old and don't remake it. -p, --print-data-base Print make's internal database. -q, --question Run no commands; exit status says if up to date. -r, --no-builtin-rules Disable the built-in implicit rules. -R, --no-builtin-variables Disable the built-in variable settings. -s, --silent, --quiet Don't echo commands. -S, --no-keep-going, --stop Turns off -k. -t, --touch Touch targets instead of remaking them. -v, --version Print the version number of make and exit. -w, --print-directory Print the current directory. --no-print-directory Turn off -w, even if it was turned on implicitly. -W FILE, --what-if=FILE, --new-file=FILE, --assume-new=FILE Consider FILE to be infinitely new. --warn-undefined-variables Warn when an undefined variable is referenced. This program built for i486-pc-linux-gnu Report bugs to <[email protected]> Host:# Nothing related to cross compilation in make tool. But, the definition of cross compiler should be in Makefile, which is generated by configuration tool. Let's check now. Host:# ./configure --help `configure' configures lame 3.99.5 to adapt to many kinds of systems. Usage: ./configure [OPTION]... [VAR=VALUE]... To assign environment variables (e.g., CC, CFLAGS...), specify them as VAR=VALUE. See below for descriptions of some of the useful variables. Defaults for the options are specified in brackets. Configuration: -h, --help --help=short --help=recursive -V, --version -q, --quiet, --silent --cache-file=FILE -C, --config-cache -n, --no-create --srcdir=DIR

display this help and exit display options specific to this package display the short help of all the included packages display version information and exit do not print `checking ...' messages cache test results in FILE [disabled] alias for `--cache-file=config.cache' do not create output files find the sources in DIR [configure dir or `..']

Installation directories: --prefix=PREFIX install architecture-independent files in PREFIX [/usr/local] --exec-prefix=EPREFIX install architecture-dependent files in EPREFIX [PREFIX]

By default, `make install' will install all the files in `/usr/local/bin', `/usr/local/lib' etc. You can specify an installation prefix other than `/usr/local' using `--prefix', for instance `--prefix=$HOME'. For better control, use the options below. Fine tuning of the installation directories: --bindir=DIR user executables [EPREFIX/bin] --sbindir=DIR system admin executables [EPREFIX/sbin] --libexecdir=DIR program executables [EPREFIX/libexec] --sysconfdir=DIR read-only single-machine data [PREFIX/etc] --sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com] --localstatedir=DIR modifiable single-machine data [PREFIX/var] --libdir=DIR object code libraries [EPREFIX/lib] --includedir=DIR C header files [PREFIX/include] --oldincludedir=DIR C header files for non-gcc [/usr/include] --datarootdir=DIR read-only arch.-independent data root [PREFIX/share] --datadir=DIR read-only architecture-independent data [DATAROOTDIR] --infodir=DIR info documentation [DATAROOTDIR/info] --localedir=DIR locale-dependent data [DATAROOTDIR/locale] --mandir=DIR man documentation [DATAROOTDIR/man] --docdir=DIR documentation root [DATAROOTDIR/doc/lame] --htmldir=DIR html documentation [DOCDIR] --dvidir=DIR dvi documentation [DOCDIR] --pdfdir=DIR pdf documentation [DOCDIR] --psdir=DIR ps documentation [DOCDIR] Program names: --program-prefix=PREFIX --program-suffix=SUFFIX --program-transform-name=PROGRAM System types: --build=BUILD --host=HOST

prepend PREFIX to installed program names append SUFFIX to installed program names run sed PROGRAM on installed program names

configure for building on BUILD [guessed] cross-compile to build programs to run on HOST [BUILD]

Optional Features: --disable-option-checking ignore unrecognized --enable/--with options --disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no) --enable-FEATURE[=ARG] include FEATURE [ARG=yes] --enable-maintainer-mode enable make rules and dependencies not useful (and sometimes confusing) to the casual installer --disable-dependency-tracking speeds up one-time build --enable-dependency-tracking do not reject slow dependency extractors --enable-shared[=PKGS] build shared libraries [default=yes] --enable-static[=PKGS] build static libraries [default=yes] --enable-fast-install[=PKGS] optimize for fast installation [default=yes] --disable-libtool-lock avoid locking (might break parallel builds) --disable-largefile omit support for large files --enable-nasm Allow the use of nasm if available --disable-rpath do not hardcode runtime library paths --disable-cpml Do not use Compaq's fast Math Library --disable-gtktest Do not try to compile and run a test GTK program --enable-efence Use ElectricFence for malloc debugging --disable-analyzer-hooks Exclude analyzer hooks --disable-decoder Exclude mpg123 decoder --disable-frontend Do not build the lame executable default=build --enable-mp3x Build GTK frame analyzer default=no

--enable-mp3rtp Build mp3rtp default=no --enable-dynamic-frontends Link frontends against shared libraries default=no --enable-expopt=full,norm Whether to enable experimental optimizations default=no --enable-debug=alot,norm Enable debugging (disables optimizations) default=no Optional Packages: --with-PACKAGE[=ARG] --without-PACKAGE --with-pic

use PACKAGE [ARG=yes] do not use PACKAGE (same as --with-PACKAGE=no) try to use only PIC/non-PIC objects [default=use both] --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-dmalloc use dmalloc, as in http://www.dmalloc.com/dmalloc.tar.gz --with-gnu-ld assume the C compiler uses GNU ld default=no --with-libiconv-prefix[=DIR] search for libiconv in DIR/include and DIR/lib --without-libiconv-prefix don't search for libiconv in includedir and libdir --with-gtk-prefix=PFX Prefix where GTK is installed (optional) --with-gtk-exec-prefix=PFX Exec prefix where GTK is installed (optional) --with-fileio=lame Use lame's internal file io routines default =sndfile Use Erik de Castro Lopo's libsndfile (no stdin possible currently)

Some influential environment variables: CC C compiler command CFLAGS C compiler flags LDFLAGS linker flags, e.g. -L<lib dir> if you have libraries in a nonstandard directory <lib dir> LIBS libraries to pass to the linker, e.g. -l<library> CPPFLAGS (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if you have headers in a nonstandard directory <include dir> CPP C preprocessor PKG_CONFIG path to pkg-config utility PKG_CONFIG_PATH directories to add to pkg-config's search path PKG_CONFIG_LIBDIR path overriding pkg-config's built-in search path SNDFILE_CFLAGS C compiler flags for SNDFILE, overriding pkg-config SNDFILE_LIBS linker flags for SNDFILE, overriding pkg-config Use these variables to override the choices made by `configure' or to help it to find libraries and programs with nonstandard names/locations. Report bugs to <[email protected]>. Host:# Well, there are several items related to cross compilation here. 1. --host=HOST cross-compile to build programs to run on HOST [BUILD] 2. CFLAGS C compiler flags 3. --prefix=PREFIX install architecture-independent files in PREFIX The --host is about to let us to specify the prefix of the cross compiler being used. For our case here, we should specify it as "mips-linux-gnu". Regarding the CFLAGS, of course we need to specify -EL for our little endian MIPS architecture. For --prefix, it is about to redefine the installation directory which is other than /usr/local. As personal experience, I really not recommend to use this option.

The reason is the binary being generated at the path that specified in --prefix is not as "friendly" as the counterpart option in make install DESTDIR. Let us start to cross compile this Lame MP3 Encoder now. Host:# ./configure --host="mips-linux-gnu" CFLAGS="-EL" configure: WARNING: if you wanted to set the --build type, don't use --host. If a cross compiler is detected then cross compile mode will be used checking build system type... i686-pc-linux-gnu checking host system type... mips-unknown-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for mips-linux-gnu-strip... mips-linux-gnu-strip checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether to enable maintainer-specific portions of Makefiles... no checking for style of include used by make... GNU checking for mips-linux-gnu-gcc... mips-linux-gnu-gcc ... ... config.status: creating ACM/ddk/Makefile config.status: creating ACM/tinyxml/Makefile config.status: creating lame.spec config.status: creating mac/Makefile config.status: creating macosx/Makefile config.status: creating macosx/English.lproj/Makefile config.status: creating macosx/LAME.xcodeproj/Makefile config.status: creating vc_solution/Makefile config.status: creating config.h config.status: executing depfiles commands config.status: executing libtool commands Host:# make make all-recursive make[1]: Entering directory `/home/smp383/mips-4.3/bin/lame-3.99.5' Making all in mpglib ... ... collect2: ld returned 1 exit status make[3]: *** [libmp3lame.la] Error 1 make[3]: Leaving directory `/home/smp383/mips-4.3/bin/lame-3.99.5/libmp3lame' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/home/smp383/mips-4.3/bin/lame-3.99.5/libmp3lame' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/home/smp383/mips-4.3/bin/lame-3.99.5' make: *** [all] Error 2 Host:# Unfortunately, make error. It might because of some source files might be in .cpp and since we don't specify the CPPFLAGS yet, error should be incurred. Let's check for any .cpp file. Host:# find . -iname "*.cpp" ./ACM/DecodeStream.cpp ./ACM/AEncodeProperties.cpp ./ACM/tinyxml/xmltest.cpp ./ACM/tinyxml/tinyxmlparser.cpp ./ACM/tinyxml/tinyxmlerror.cpp ./ACM/tinyxml/tinyxml.cpp ./ACM/main.cpp ./ACM/ACM.cpp

./ACM/ADbg/ADbg.cpp ./ACM/ACMStream.cpp ./Dll/Example.cpp ./dshow/REG.CPP ./dshow/Mpegac.cpp ./dshow/PropPage.cpp ./dshow/aboutprp.cpp ./dshow/PropPage_adv.cpp ./dshow/Encoder.cpp Host:# Yes, there are some .cpp files in certain subdirectories. Let's specify CPPFLAGS and try again. Host:# make distclean Host:# ./configure --host="mips-linux-gnu" CFLAGS="-EL" CPPFLAGS="-EL" Host:# make However, the error still there. Now, we suspect we are missing LDFLAGS. Specify it and try again. Host:# make distclean Host:# ./configure --host="mips-linux-gnu" CFLAGS="-EL" CPPFLAGS="-EL" LDFLAGS="-EL" Host:# make Unfortunately, the error is still. So, we doubt the Makefile being generated by configure tool did not specify the options of cross compiler properly. In other words, if we redefine the CFLAGS, CPPFLAGS, or LDFLAGS, the default options might be overrode, and this might be the reason to cause the error. Let us ignore those CFLAGS, CPPFLAGS, and LDFLAGS, and we specify the -EL option in CC now. Host:# make distclean Host:# ./configure --host="mips-linux-gnu" CC="mips-linux-gnu-gcc -EL" Host:# make Excellent, it is done. Now, let us make a comparison between the cross compilation options being generated by those 2 configuration methods as aforementioned. ////////////////////////////////////////////////////////////////////////////////// Host:# ./configure --host="mips-linux-gnu" CFLAGS="-EL" CPPFLAGS="-EL" LDFLAGS="-EL" Makefile: ... ... CC = mips-linux-gnu-gcc CCDEPMODE = depmode=gcc3 CFLAGS = -Wall -pipe -EL CONFIG_DEFS = CONFIG_MATH_LIB = -lm CPP = mips-linux-gnu-gcc -E CPPFLAGS = -EL ... ... ////////////////////////////////////////////////////////////////////////////////// ////////////////////////////////////////////////////////////////////////////////// Host:# ./configure --host="mips-linux-gnu" CC="mips-linux-gnu-gcc -EL"

Makefile: ... ... CC = mips-linux-gnu-gcc -EL CCDEPMODE = depmode=gcc3 CFLAGS = -O3 -fomit-frame-pointer -ffast-math -Wall -pipe CONFIG_DEFS = CONFIG_MATH_LIB = -lm CPP = mips-linux-gnu-gcc -EL -E CPPFLAGS = ... ... ////////////////////////////////////////////////////////////////////////////////// Apparently, some default options in CFLAGS have been overrode if we redefine the CFLAGS ourself. So, it is important to note that we should avoid to redefine those compilation and link flag in doing cross compilation. Now, the binaries are ready for us, and we are not going to install it into /usr/local/. Because we are doing cross compilation now, we expect the binaries will be concentrated in a special directory in our development directory. Yes, for this purpose, we can make use of DESTDIR option in doing make install. Let's check. Host:# make install DESTDIR=$PWD/cawanlame Making install in mpglib make[1]: Entering directory `/home/smp383/mips-4.3/bin/lame-3.99.5/mpglib' make[2]: Entering directory `/home/smp383/mips-4.3/bin/lame-3.99.5/mpglib' make[2]: Nothing to be done for `install-exec-am'. make[2]: Nothing to be done for `install-data-am'. make[2]: Leaving directory `/home/smp383/mips-4.3/bin/lame-3.99.5/mpglib' make[1]: Leaving directory `/home/smp383/mips-4.3/bin/lame-3.99.5/mpglib' Making install in libmp3lame ... ... make[2]: Leaving directory `/home/smp383/mips-4.3/bin/lame-3.99.5' make[1]: Leaving directory `/home/smp383/mips-4.3/bin/lame-3.99.5' Host:# cd cawanlame Host:# ls usr Host:# cd usr Host:# ls local Host:# cd local Host:# ls bin include lib share Host:# Well, the lame executable and shared library should be in bin and lib folders, respectively. Let's verify. Host:# cd bin Host:# ls -l male ls: cannot access male: No such file or directory Host:# cd .. Host:# cd bin Host:# ls -l lame -rwxr-xr-x 1 root root 409167 2012-11-11 16:10 lame Host:# cd .. Host:# cd lib

Host:# ls -l total 768 -rw-r--r-- 1 -rwxr-xr-x 1 lrwxrwxrwx 1 lrwxrwxrwx 1 -rwxr-xr-x 1 Host:#

root root root root root

root 429192 2012-11-11 16:10 libmp3lame.a root 943 2012-11-11 16:10 libmp3lame.la root 19 2012-11-11 16:10 libmp3lame.so -> libmp3lame.so.0.0.0 root 19 2012-11-11 16:10 libmp3lame.so.0 -> libmp3lame.so.0.0.0 root 343465 2012-11-11 16:10 libmp3lame.so.0.0.0

Nice, we are at the right point now. Let us make sure the binaries are really for little endian MIPS platform. Host:# cd bin Host:# file lame lame: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1, for GNU/Linux 2.6.12, dynamically linked (uses shared libs), not stripped Host:# cd .. Host:# cd lib Host:# ls -l total 768 -rw-r--r-- 1 root root 429192 2012-11-11 16:10 libmp3lame.a -rwxr-xr-x 1 root root 943 2012-11-11 16:10 libmp3lame.la lrwxrwxrwx 1 root root 19 2012-11-11 16:10 libmp3lame.so -> libmp3lame.so.0.0.0 lrwxrwxrwx 1 root root 19 2012-11-11 16:10 libmp3lame.so.0 -> libmp3lame.so.0.0.0 -rwxr-xr-x 1 root root 343465 2012-11-11 16:10 libmp3lame.so.0.0.0 Host:# file libmp3lame.so.0.0.0 libmp3lame.so.0.0.0: ELF 32-bit LSB shared object, MIPS, MIPS32 rel2 version 1 (SYSV), dynamically linked, not stripped Host:# Yes, we got what we want. Let's run it in MIPS environment now. tango3[cawan]# cd lame-3.99.5 tango3[lame-3.99.5]# cd cawanlame/ tango3[cawanlame]# cd usr/ tango3[usr]# cd local/ tango3[local]# cd bin tango3[bin]# ./lame LAME 32bits version 3.99.5 (http://lame.sf.net) usage: ./lame [options] <infile> [outfile] <infile> and/or <outfile> can be "-", which means stdin/stdout. Try: "./lame --help" or: "./lame --preset help" or: "./lame --longhelp" or "./lame -?" tango3[bin]# Good, the lame executable running well in our MIPS platform. In order to get better experience in running executable in MIPS platform, let us use the lame executable to compress an .wav file into .mp3 file now. Of course the process might take quite a while because we are running the compression process in a resource constraint embedded platform. for a complete options list for information on suggested predefined settings for general usage information

tango3[bin]# ./lame --help LAME 32bits version 3.99.5 (http://lame.sf.net) usage: ./lame [options] <infile> [outfile] <infile> and/or <outfile> can be "-", which means stdin/stdout. RECOMMENDED: lame -V2 input.wav output.mp3 OPTIONS: -b bitrate -h -f -V n --preset type

set the bitrate, default 128 kbps higher quality, but a little slower. Recommended. fast mode (lower quality) quality setting for VBR. default n=4 0=high quality,bigger files. 9=smaller files type must be "medium", "standard", "extreme", "insane", or a value for an average desired bitrate and depending on the value specified, appropriate quality settings will be used. "--preset help" gives more info on these ID3 tagging related options full list of options print License information

--help id3 --longhelp --license

tango3[bin]# ./lame -V2 luohua.wav luohua.mp3 LAME 3.99.5 32bits (http://lame.sf.net) Using polyphase lowpass filter, transition band: 18671 Hz - 19205 Hz Encoding luohua.wav to luohua.mp3 Encoding as 44.1 kHz j-stereo MPEG-1 Layer III VBR(q=2) Frame | CPU time/estim | REAL time/estim | play/CPU | ETA 6253/6253 (100%)| 1:52/ 1:52| 5:37/ 5:37| 1.4538x| 0:00 32 [ 81] ** 40 [ 0] 48 [ 0] 56 [ 1] % 64 [ 2] % 80 [ 5] % 96 [ 2] % 112 [ 1] * 128 [ 73] %* 160 [2410] %%%%%************************************************ 192 [3126] %%%%%%%%%%%%%%%%%%%%%%%%******************************************** 224 [ 295] %%***** 256 [ 197] %%*** 320 [ 60] %* ------------------------------------------------------------------------------kbps LR MS % long switch short % 181.4 23.4 76.6 95.6 2.4 2.0 Writing LAME Tag...done ReplayGain: -8.0dB tango3[bin]# ls lame* luohua.mp3 luohua.wav Well, the luohua.mp3 is generated. Let us play it with a player. Nice, it sounds really good.