64bit kernel on Raspberry Pi 4 (quirks)
by Andrei Gherzan 2019-07-04
- this braindump is based on a more general one available here
- it adds specific quirks currently needed for Raspberry Pi 4
There are a couple of things we discovered while adding support in Yocto/meta-raspberrypi for the new Raspberry Pi 4 and I realized that they can be a good guide for anyone trying to boot a 64bit kernel for this board with or without a 64bit user-space. I will try to keep this post updated as further development will invalidate items discussed below.
A. Raspberry Pi firmware
If you are working on top of a 32bit Raspbian distribution (at least
2019-06-20), you will have all the needed blobs in the boot partition. If you build a distro from scratch or working on a different distro, you will need to include the boot files from a raspberrypi-firmware release version greater or equal to 1.20190620.
B. Compile aarch64 toolchain
This step is only needed if you need to build a custom
armstub (see below). Initially, this guide included the instructions for building a toolchain and the kernel. That was split out of it because it was useful for other boards as well (see Raspberry Pi 3). Also, all these quirks will be eventually obsolete. That being said, for now, follow the guide here to build a 64bit kernel for Raspberry Pi 4 (be aware to pick up the right
C. arm8 stub with
A new Raspberry Pi firmware was released which includes the needed armstubs support for bcm2838 (bcm2711). It means that you can skip this entire step by just using a newer firmware version. The minimum version needed to avoid having custom stubs is 2019-07-09. This version is included in Raspbian 2019-07-10.
If you followed the braindump above, you have a toolchain and a kernel compiled. I assume that the exported variables (I define some paths to be able to reference them later) are in place. Otherwise feel free to tweak the instructions below as needed.
armstub is the code that the GPU loads and sets the ARM CPU before running the kernel. This support for
BCM2711 was not included in the current latest firmware (see above) so you will need to compile and configure the board to use the appropriate one.
git clone https://github.com/raspberrypi/tools.git rpi-tools cd rpi-tools/armstubs git checkout 7f4a937e1bacbc111a22552169bc890b4bb26a94 PATH=$PATH:$TOOLCHAIN/bin make armstub8.bin # See B above for the TOOLCHAIN path export ARMSTUB=`realpath armstub8.bin` # used if you want to deploy it to raspbian, ignore otherwise
D. Tweak NVRAM configuration for
2019-06-20 includes this change by default so skip if that is what you are working on. Otherwise, the current firmware-nonfree uses a NVRAM configuration which renders the WiFi interface unusable. You will need to change
bootflags3 as it follows:
E. Switch Raspbian
2019.06.20 to 64bit kernel
Let’s deploy now all the needed quirks for a Raspberry Pi 4 running a 64bit kernel on Raspbian.
The following commands assume that the boot partition mount point is
/run/media/me/boot. Also, the paths rely on the exports set throughout this braindump and the one described in B.
armstub is now included from Raspbian 2019-07-10 (the packaged firmware includes the armstubs for
bcm2711). Deploying and using a custom one is only needed if you are working (for whatever reason) on an older version:
cp $ARMSTUB /run/media/me/boot echo "armstub=armstub8-gic.bin" >> /run/media/me/boot/config.txt echo "enable_gic=1" >> /run/media/me/boot/config.txt
Kernel on 64bit had a memory size limitation due to the fact that only the first 1Gb can be used for DMA. This was mitigated in the upstream Raspberry Pi linux fork. If you are using an older version of the kernel, you will need to limit the memory which in turn will limit the memory for DMA:
echo "total_mem=1024" >> /run/media/me/boot/config.txt
Even in the current kernel there seems to be a bug that the USB ports are not working with 4 GB of RAM but the 64 bit kernel works fine with just 3 GB memory, see this issue. So we should limit the ram to 3 GB by
echo "total_mem=3072" >> /run/media/me/boot/config.txt
The last piece in the jigsaw is forcing the firmware to set the arm in 64bit mode. This should happen automatically based on the kernel image filename. Currently that doesn’t seem to be stable so we will force it in the
echo "arm_64bit=1" >> /run/media/me/boot/config.txt
And that is it.
sync umount /run/media/me/boot
In conclusion, if you are working on the latest Raspbian and the latest revision
4.19.y kernel, all you need to do is force the arm in 64bit mode and an 3 GB memory limit as describe above.