Anatomy of Cross-Compilation Toolchains: Thomas Petazzoni
Anatomy of Cross-Compilation Toolchains: Thomas Petazzoni
Anatomy of
cross-compilation toolchains
Thomas Petazzoni
free electrons
Artwork and Photography by Jason Freeny
[email protected]
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 1/1
Thomas Petazzoni
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 2/1
Disclaimer
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 3/1
What is a cross-compiling toolchain?
▶ A set of tools that allows to build source code into binary code for a target
platform different than the one where the build takes place
▶ Different CPU architecture
▶ Different ABI
▶ Different operating system
▶ Different C library
▶ Three machines involved in the build process
▶ build machine, where the build takes place
▶ host machine, where the execution takes place
▶ target machine, for which the programs generate code
▶ Native toolchain: build == host == target
▶ Cross-compilation toolchain: build == host != target
▶ Corresponds to the --build, --host and --target autoconf configure script
arguments
▶ By default, automatically guessed by autoconf to be for the current machine
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 4/1
Toolchain tuple
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 5/1
Toolchain tuple examples
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 6/1
Bare-metal vs. Linux toolchain
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 7/1
Components
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 8/1
binutils
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 10/1
Linux Kernel headers
▶ In order to build a C library, the Linux kernel headers are needed: definitions of
system call numbers, various structure types and definitions.
▶ In the kernel, headers are split between:
▶ User-space visible headers, stored in uapi directories: include/uapi/,
arch/<ARCH>/include/uapi/asm
▶ Internal kernel headers.
▶ Installation takes place using
make ARCH=.. INSTALL_HDR_PATH=... headers_install
▶ The installation includes a sanitation pass, to remove kernel-specific constructs from
the headers.
▶ As of Linux 4.8, installs 756 header files.
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 11/1
Linux Kernel headers version
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 12/1
C library
▶ Provides the implementation of the POSIX standard functions, plus several other
standards and extensions
▶ Based on the Linux system calls
▶ Several implementations available:
▶ glibc
▶ uClibc-ng (formerly uClibc)
▶ musl
▶ bionic, for Android systems
▶ A few other more special-purpose: newlib (for bare-metal), dietlibc, klibc
▶ After compilation and installation, provides:
▶ The dynamic linker, ld.so
▶ The C library itself libc.so, and its companion libraries: libm, librt, libpthread,
libutil, libnsl, libresolv, libcrypt
▶ The C library headers: stdio.h, string.h, etc.
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 13/1
C library: glibc
▶ GNU C Library
▶ De-facto standard of Linux C libraries
▶ Used in virtually all common desktop/server distributions
▶ Full-featured
▶ Supports for numerous architectures or operating systems
▶ No support for noMMU platforms
▶ No support for static linking
▶ ABI backward compatibility
▶ Almost no configurability
▶ Used to be “too big” for embedded, but no longer necessarily the case.
▶ LGPLv2.1 or later
▶ https://www.gnu.org/software/libc/
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 14/1
C library: uClibc/uClibc-ng
▶ Started in 2000
▶ High-level of configurability
▶ Supports many architectures, include some not supported by glibc
▶ Supports only Linux as operating system
▶ No ABI backward compatibility
▶ Supports numerous no-MMU architectures: ARM noMMU, Blackfin, etc.
▶ No longer related to uClinux
▶ Support for static linking
▶ Original uClibc project dead (last release in May 2012), but the uClibc-ng fork is
very active and is the de-facto replacement.
▶ LGPLv2.1
▶ http://uclibc-ng.org/
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 15/1
C library: musl
▶ Started in 2011
▶ MIT licensed
▶ Very active development
▶ Support for ARM, ARM64, i386, Microblaze, MIPS(64), OpenRisc, PowerPC(64),
SuperH, x86-4
▶ Recently, noMMU support was added for SuperH2, for the J-core Open
Processor
▶ No configurability
▶ Small, even smaller than uClibc, especially for static linking scenarios
▶ Strict conformance to standards (stricter than glibc, uClibc), causes a few build
issues with a number of packages
▶ Nice comparison of the three main C libraries:
http://www.etalabs.net/compare_libcs.html
▶ http://www.musl-libc.org/
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 16/1
C library: size comparison
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 17/1
gcc dependencies
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 18/1
Overall build process
▶ The build process for a regular Linux cross-compilation toolchain is in fact fairly
easy:
1. Build binutils
2. Build the dependencies of gcc: mpfr, gmp, mpc
3. Install the Linux kernel headers
4. Build a first stage gcc: no support for a C library, support only for static linking
5. Build the C library using the first stage gcc
6. Build the final gcc, with C library and support for dynamic linking
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 19/1
Overall build process: example in Buildroot
host-gawk
host-binutils
toolchain toolchain-buildroot host-gcc-final glibc host-gcc-initial
host-mpc host-mpfr host-gmp host-m4
linux-headers
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 20/1
Concept of sysroot
▶ The sysroot is the the logical root directory for headers and libraries
▶ Where gcc looks for headers, and ld looks for libraries
▶ Both gcc and binutils are built with --with-sysroot=<SYSROOT>
▶ The kernel headers and the C library are installed in <SYSROOT>
▶ If the toolchain has been moved to a different location, gcc will still find its
sysroot if it’s in a subdir of --prefix
▶ --prefix=/home/thomas/buildroot/arm-uclibc/host/usr
▶ --with-sysroot=/home/thomas/buildroot/arm-uclibc/host/usr/arm-
buildroot-linux-uclibcgnueabihf/sysroot
▶ Can be overridden at runtime using gcc’s --sysroot option.
▶ The current sysroot can be printed using the -print-sysroot option.
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 21/1
Multilib toolchains (1)
▶ Most toolchains provide a single sysroot with the C library and gcc runtime
libraries
▶ These libraries, built for the target, are optimized for a specific architecture
variant and ABI
▶ Need to have one toolchain for each architecture variant or ABI
▶ Multilib toolchains contain multiple sysroot, each having a version of the target
libraries for different architecture/ABI variants.
▶ Example of the Sourcery CodeBench ARM toolchain:
$ arm-none-linux-gnueabi-gcc -print-multi-lib
.;
armv4t;@march=armv4t
thumb2;@mthumb@march=armv7-a
▶ Three sysroots: ARMv5, ARMv4 and ARMv7 Thumb-2
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 22/1
Multilib toolchains (2)
▶ The compiler automatically selects the right sysroot depending on the gcc flags:
$ arm-none-linux-gnueabi-gcc -march=armv5te -print-sysroot
.../bin/../arm-none-linux-gnueabi/libc
$ arm-none-linux-gnueabi-gcc -march=armv4t -print-sysroot
.../bin/../arm-none-linux-gnueabi/libc/armv4t
$ arm-none-linux-gnueabi-gcc -march=armv7-a -mthumb -print-sysroot
.../bin/../arm-none-linux-gnueabi/libc/thumb2
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 23/1
Toolchain contents
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 24/1
Toolchain contents
▶ arm-buildroot-linux-uclibcgnueabihf/
▶ bin/
▶ Limited set of binutils programs, without their cross-compilation prefix. Hard links to
their counterparts with the prefix. This is where gcc finds them.
▶ include/c++/4.9.4/
▶ Headers for the C++ standard library, installed by gcc
▶ Interestingly, they are not part of the sysroot per-se.
▶ lib/
▶ The gcc runtime libraries, built for the target
▶ libatomic, provides a software implementation of atomic built-ins, when needed
▶ libgcc, the main gcc runtime (optimized functions, 64-bit division, floating point
emulation)
▶ libitm, transactional memory library
▶ libstdc++, standard C++ library
▶ libsupc++, subset of libstdc++ with only the language support functions
▶ sysroot/
▶ lib/, usr/lib/: C library and gcc runtime libraries (shared and static)
▶ usr/include/, Linux kernel and C library headers
▶ bin/
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 24/1
Toolchain contents
▶ arm-buildroot-linux-uclibcgnueabihf/
▶ bin/
▶ arm-buildroot-linux-uclibcgnueabihf- prefixed tools
▶ From binutils: addr2line, ar, as, elfedit, gcov, gprof, ld, nm, objcopy, objdump,
ranlib, readelf, size, strings, strip
▶ From gcc: c++ (same as g++), cc (same as gcc), cpp, g++, gcc, gcc-ar, gcc-nm,
gcc-ranlib
▶ The gcc-{ar,nm,ranlib} are wrappers for the corresponding binutils program, to
support Link Time Optimization (LTO)
▶ include/
▶ lib/
▶ libexec/
▶ share/
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 24/1
Toolchain contents
▶ arm-buildroot-linux-uclibcgnueabihf/
▶ bin/
▶ include/
▶ Headers of the host libraries (gmp, mpfr, mpc)
▶ lib/
▶ libexec/
▶ share/
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 24/1
Toolchain contents
▶ arm-buildroot-linux-uclibcgnueabihf/
▶ bin/
▶ include/
▶ lib/
▶ gcc/arm-buildroot-linux-uclibcgnueabihf/4.9.4/
▶ crtbegin*.o, crtend*.o, object files handling constructors/destructors, linked into
executables
▶ include/, headers provided by the compiler (stdarg.h, stdint.h, stdatomic.h, etc.)
▶ include-fixed/, system headers that gcc fixed up using fixincludes
▶ install-tools/, also related to the fixincludes process
▶ libgcc.a, libgcc_eh.a, libgcov.a, static variants of the gcc runtime libraries
▶ ldscripts/, linker scripts provided by gcc to link programs and libraries
▶ Host version of gmp, mpfr, mpc, needed for gcc
▶ libexec/
▶ share/
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 24/1
Toolchain contents
▶ arm-buildroot-linux-uclibcgnueabihf/
▶ bin/
▶ include/
▶ lib/
▶ libexec/
▶ gcc/arm-buildroot-linux-uclibcgnueabihf/4.9.4/
▶ cc1, the actual C compiler
▶ cc1plus, the actual C++ compiler
▶ collect2, program from gcc collecting initialization functions, wrapping the linker
▶ install-tools/, misc gcc related tools, not needed for the compilation process
▶ liblto_plugin.so.0.0.0, lto-wrapper, lto1, related to LTO support (outside of the
scope of this talk)
▶ share/
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 24/1
Toolchain contents
▶ arm-buildroot-linux-uclibcgnueabihf/
▶ bin/
▶ include/
▶ lib/
▶ libexec/
▶ share/
▶ documentation (man pages and info pages)
▶ translation files for gcc and binutils
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 24/1
Architecture tuning
▶ gcc provides several configure-time options to tune for a specific
architecture/CPU variant: --with-arch, --with-cpu, --with-abi, --with-fpu
▶ These define the default architecture/CPU variant for which gcc will generate
code.
▶ They can be overridden at runtime using the -march, -mcpu, -mabi, -mfpu
options.
▶ However, be careful: parts of the toolchain are built for the target!
▶ The gcc runtime libraries
▶ The C library, dynamic linker, and startup code
▶ They are built together with the rest of the toolchain, so it’s important to know
with what optimization level they were built!
▶ Passing -march=armv5te is not sufficient to make your binary work on ARMv5 if
your toolchain originally targets ARMv7.
▶ Read the gcc documentation, section Machine-dependent options to get the
complete list of possible values.
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 25/1
ABI: definition
▶ OABI: obsolete ABI. Forced the use of hard-float instructions, which required
emulation of floating-point operations in the kernel. No longer supported
anywhere.
▶ EABI, standardized by ARM. Allows mixing hard-float code with soft-float code.
Floating point arguments passed in integer registers.
▶ Hard-float code: uses floating point instructions directly.
▶ Soft-float code: emulates floating point instructions using a userspace library
provided by gcc
▶ EABIhf, also standardized by ARM. Requires a floating point unit: only
hard-float code. Floating point arguments passed in floating point registers.
▶ gcc options
▶ EABI soft-float: -mabi=aapcs-linux -mfloat-abi=soft
▶ EABI hard-float: -mabi=aapcs-linux -mfloat-abi=softfp
▶ EABIhf: -mabi=aapcs-linux -mfloat-abi=hard
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 27/1
Difference between toolchain and SDK
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 28/1
How to get a cross-compilation toolchain
▶ Pre-built
▶ From your distribution. Ubuntu and Debian have numerous cross-compilers readily
available.
▶ From various organization: Linaro provides ARM and AArch64 toolchains, Mentor
provides a few free Sourcery CodeBench toolchains, Imagination provides MIPS
toolchains, etc.
▶ Built it yourself
▶ Crosstool-NG, tool specialized in building cross-compilation toolchain. By far the
most configurable/versatile.
▶ Embedded Linux build systems generally all know how to build a cross-compilation
toolchain: Yocto/OpenEmbedded, Buildroot, OpenWRT, etc.
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 29/1
References
▶ Crosstool-NG documentation,
https://github.com/crosstool-ng/crosstool-ng/blob/master/docs/
▶ GCC documentation, https://gcc.gnu.org/onlinedocs/
▶ Binutils documentation, https://sourceware.org/binutils/docs/
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 30/1
Thanks for your attention!
Questions?
Thomas Petazzoni
[email protected]
Slides under CC-BY-SA 3.0
http://free-electrons.com/pub/conferences/2016/elce/petazzoni-toolchain-anatomy/
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com 31/1