Cross compile Linux for ARM using LLVM clang on Arch Linux: Benefits and Challenges

The LLVMLinux team made quite some progress in enabling LLVM/clang to build the Linux kernel. There has also been a talk at the Collaboration Summit 2015 in Santa Rosa (slides). So I thought, give it a try and compile a kernel for my ARM based board of choice. The upstream kernel.org Linux kernel is not yet ready to be built using LLVM/clang, some patches are still required. Therefor I used the kernel from the LLVMLinux git server.

To compile the kernel, clang does not use its internal assembler but uses the GNU assembler (by using the -no-integrated-as option). In this case the build system picked your hosts assembler (x86_64) instead of the one provided by the cross compiler toolchain. The simplest way to work around this is creating a symlink to clang in the cross compiler toolchains directory:

Cross compile Linux for ARM using LLVM clang on Arch Linux

Download Zip

This is actually not related to the LLVM/clang build itself. It happened to me when using the default cross compiler toolchain from my Ångström distribution build. This toolchain uses the gold linker by default, which does not work with the Linux kernel, the standard BFD linker is required. You can either append the Linker explicitly or symlink the correct assembler. arm-linux-gnueabihf-ld.gold

I found that clang is very different to gcc in respect to cross compiling, in GCC world you should have separate binary for everything, like PLATFORM_make or PLATFORM_ld (i*86-pc-cygwin i*86-*-linux-gnu etc. )

For those interested in CPUs to tune for, you can use --print-supported-cpus (added here). If your clang binary can compile for more than one architecture, you have to pass --target as well to select the proper architecture; for instance, to list the ARM CPUs available on the Android cross compiler, you could use --target=aarch64-unknown-linux-gui.

This document will guide you in choosing the right Clang optionsfor cross-compiling your code to a different architecture. It assumes youalready know how to compile the code in question for the host architecture,and that you know how to choose additional include and library paths.

On the other hand, Clang/LLVM is natively a cross-compiler, meaning thatone set of programs can compile to all targets by setting the -targetoption. That makes it a lot easier for programmers wishing to compile todifferent platforms and architectures, and for compiler developers thatonly have to maintain one build system, and for OS distributions, thatneed only one set of main packages.

The unofficial repository archlinuxcn has rust-nightly and Rust std library for i686, ARM, ARMv7, Windows 32 and 64 so you can just install the one you want then enjoy cross-compiling. However, you have to find an ARM toolchain by yourself. For Windows 32bit targets, you will need to get a libgcc_s_dw2-1.dll (provided by mingw-w64-gcc) to build and run.

The package name shall not be prefixed with the word cross- (it was previously proposed, but was not adopted in official packages, probably due to additional length of names), and shall consist of the package name, prefixed by GNU triplet without vendor field or with "unknown" in vendor field; example: arm-linux-gnueabihf-gcc. If shorter naming convention exists (e.g. mips-gcc), it may be used, but this is not recommended.

This weird thing allows easier cross-compiling. Sometimes, project Makefiles do not use CC & co. variables and instead use gcc directly. If you just want to try to cross-compile such project, editing the Makefile could be a very lengthy operation. However, changing the $PATH to use "our" executables first is a very quick solution. You would then run PATH=/usr/arch/bin/:$PATH make instead of make.

Once you have a GCC toolchain, configure your build of LLVM to use the newtoolchain for your host compiler and C++ standard library. Because the newversion of libstdc++ is not on the system library search path, you need to passextra linker flags so that it can be found at link time (-L) and at runtime(-rpath). If you are using CMake, this invocation should produce workingbinaries:

In this section, Cross Compiling will be done for the ARM arm64 architecutre, not for the host system. To compile the ARM arm64 architecture, you need to add two command line parameters: ARCH=arm64 and CROSS_COMPILE=aarch64-linux-gnu-. The rest of the kernel compilation process is the same as the host system.

To verify your current setup, run %LINUX_MULTIARCH_ROOT%x86_64-unknown-linux-gnu\bin\clang++ -v from the Command Prompt, which should display "clang version X.X.X ..." (see below):

The next step is to compile the same source for ARM. We simply do this by using a different compiler (arm-linux-gnueabi-gcc instead of gcc for 32 bit ARM or gcc-aarch64-linux-gnu for 64 bit ARM or aarch64).

After downloading and extracting the source code, we need to run the configure-script for libpcap first. Only this time, we need to set our compiler to the musl-compiler for our cross compilation by setting CC to: aarch64-linux-musl-gcc:

The aarch64 machine will need those dependencies. Either you could install them through the package manager (if there is any over there) or cross compile these separately. The versions do not need to be an exact match but too far away will probably give you troubles.

$(CROSS_COMPILE) is the prefix used to invoke the (32-bit) cross-compilertoolchain. The default value is arm-linux-gnueabihf-. This is the variableyou want to change in case you want to use ccache to speed you recompilations:

If the build includes a mix of 32-bit and 64-bit code, for instance if you setCFG_ARM64_core=y to build a 64-bit secure kernel, then two differenttoolchains are used, that are controlled by $(CROSS_COMPILE32) and$(CROSS_COMPILE64). The default value of $(CROSS_COMPILE32) is the valueof CROSS_COMPILE, which defaults to arm-linux-gnueabihf- as mentionedabove. The default value of $(CROSS_COMPILE64) is aarch64-linux-gnu-.Examples:

The good thing about clang is that clang is inherently a cross-compiler. It lowers C/C++ source code into LLVM IR, which is platform independent. And LLVM then lower the IR further into platform-dependent machine code. One less tool to install, one less tool to maintain.

We need to tell cmake that we are cross-compiling with clang by specifying a cmake toolchian file on the commandline.The toolchain contains the information about target platform, the compiler to use and where to find libraries.

LLVM-MinGW is a toolchain built with Clang, LLD, libc++, targetingi686, x86_64, arm and aarch64 (ARM64), with releases both for runningas a cross compiler from Linux and for running on Windows. It supportsAddress Sanitizer, Undefined Behaviour Sanitizer, and generating debuginfo in PDB format.

Downloads are archive files (.zip or .7z). No installation is required,just extract the archive and start using the programs in mingw32\bin or mingw64\bin.This allows for a relocatable compiler suite and allows having multiple versions on the same system.

In this article, I will show you how to cross compile C and C++ programs on a x86-64 machine for Raspberry Pi using Clang 9. The advantage of cross compiling on a x86-64 system for armhf is that, usually one has a beefy laptop or desktop computer that can speed up, by an order of magnitude or more, the compilation of C and C++ programs.

You need to use a cross-compiler unless you are developing on your own operating system. The compiler must know the correct target platform (CPU, operating system), otherwise you will run into trouble. If you use the compiler that comes with your system, then the compiler won't know it is compiling something else entirely. Some tutorials suggest using your system compiler and passing a lot of problematic options to the compiler. This will certainly give you a lot of problems in the future and the solution is build a cross-compiler. If you have already attempted to make an operating system without using a cross-compiler, please read the article Why do I need a Cross Compiler?.

The newest GCC is recommended as it is the latest and greatest release. For instance, you may run into trouble if you use GCC 4.6.3 to build a GCC 4.8.0 cross-compiler. If you are not using the latest major GCC release for your system compiler, we recommend that you build the newest GCC as your system compiler.

You may be able to use an older major GCC release to build a cross-compiler of a newer major GCC release. For instance, GCC 4.7.3 may be able to build a GCC 4.8.0 cross-compiler. However, if you want to use the latest and greatest GCC version for your cross-compiler, we recommend that you bootstrap the newest GCC as your system compiler first. Individuals using OS X 10.7 or earlier might want to invest in either building a system GCC (that outputs native Mach-O), or upgrading the local LLVM/Clang installation. Users with 10.8 and above should install the Command Line Tools from Apple's developer website and use Clang to cross-compile GCC.

Your distribution may ship its own patched GCC and Binutils that is customized to work on your particular Linux distribution. You should be able to build a working cross-compiler using the above source code, but you might not be able to build a new system compiler for your current Linux distribution. In that case, try a newer GCC release or get the patched source code.

The "Windows Subsystem for Linux (Beta)", released with the Windows 10 Anniversary update is also an option for using a cross compiler. (Tested 08/08/2016 with GCC 6.1.0 and Binutils 2.27) This cross-compiler works reasonably fast, although being in beta state, it may not be ideal permanent development platform.

The prefix will configure the build process so that all the files of your cross-compiler environment end up in $HOME/opt/cross. You can change that prefix to whatever you like (e.g., /opt/cross or $HOME/cross would be options). If you have administrator access and wish to make the cross-compiler toolchain available to all users, you can install it into the /usr/local prefix - or perhaps a /usr/local/cross prefix if you are willing to change the system configuration such that this directory is in the search paths for all users. Technically, you could even install directly to /usr, so that your cross-compiler would reside alongside your system compiler, but that is not recommended for several reasons (like risking to overwrite your system compiler if you get TARGET wrong, or getting into conflict with your system's package management). 2ff7e9595c