qemu-system-aarch64 使用记录

• • • 下载安装qemu
    • 查看是否支持KVM
    • 运行qemu
    • • -M
      • 内核启动
      • 问题
    • 内核编译

下载安装qemu

#!/bin/bashsudo apt update  > /dev/null
sudo apt upgrade  > /dev/null
sudo apt-get install -y make gcc g++ git > /dev/nullcd
sudo apt-get install -y re2c > /dev/nullninja --versionif [[ $? -ne 0 ]]; thenif [[ ! -d ${HOME}/ninja ]]; thengit clone https://gitee.com/gitmirror/ninja.gitif [[ $? -ne 0 ]]; thenecho " git clone ninja fail"exit 1fificd ${HOME}/ninjapython3 ./configure.py --bootstrap  sudo cp ./ninja  /usr/bin# 测试ninja --versionif [[ $? -ne 0 ]];thenecho " ninja install fail"exit 1fi 
fi# 安装依赖包
sudo apt-get install -y build-essential zlib1g-dev pkg-config libglib2.0-dev > /dev/null
sudo apt-get install -y binutils-dev libboost-all-dev autoconf libtool libssl-dev libpixman-1-dev > /dev/null
# 支持enable-virtfs 共享文件
sudo apt-get install -y libcap-ng-dev libattr1-dev > /dev/nullcd
if [[ ! -d ${HOME}/qemu7 ]]; thenwget https://download.qemu.org/qemu-7.0.0.tar.xzif [[ $? -ne 0 ]];thenecho " qemu download fail"exit 1fi tar xJf qemu-7.0.0.tar.xzmv qemu-7.0.0 qemu7
fi${HOME}/qemu7/build/qemu-system-aarch64 -versionif [[ $? -ne 0 ]]; thenecho "qemu not compile"cd ${HOME}/qemu7# arm64 编译./configure --target-list=aarch64-softmmu --enable-debug --enable-debug-info  --enable-kvm  \--enable-trace-backends=simple --enable-virtfs if [[ $? -ne 0 ]];thenecho " qemu configure fail"exit 1fi# 开始编译make -j$(nproc)if [[ $? -ne 0 ]];thenecho " qemu make fail"exit 1fi${HOME}/qemu7/build/qemu-system-aarch64 -version
ficd

查看是否支持KVM

参考：

• ARM平台检测是否支持虚拟化的几种常见方法
• 如何验证内核是否支持KVM虚拟化？
• ARM对KVM支持概述

对于aarch64（arm64）架构,若支持KVM虚拟化，那么KVM代码会直接编译进内核

2020年3月，Linux 5.7 Kernel宣布将放弃支持 32位架构的 KVM虚拟化支持，所以目前来看，要想较好的在ARM架构上运行KVM虚拟化，需要使用现代化的64位ARM架构

# 如果/dev/kvm和/sys/module/kvm二者之一不存在说明KVM虚拟化是不支持的
ls -l /dev/kvm
ls -l /sys/module/kvmcat /boot/config-`uname -r` | grep VIRTUAL
CONFIG_VIRTUALIZATION=y
# CONFIG_GKI_HIDDEN_VIRTUAL_CONFIGS is not set
# CONFIG_REGULATOR_VIRTUAL_CONSUMER is not set
CONFIG_FB_VIRTUAL=y
CONFIG_ARCH_HAS_DEBUG_VIRTUAL=y
# CONFIG_DEBUG_VIRTUAL is not set

运行qemu

参考

• linux把用户添加到组
• 使用 QEMU 启动 ARM aarch64 架构 Ubuntu 虚拟机
• failed to find romfile “efi-virtio.rom“
• qemu 文档

ubuntu-22.04.1-live-server-arm64.iso 下载

# 用户添加到多个次要组中
usermod -a -G kvm ostest# 创建虚拟机硬盘
${HOME}/qemu7/build/qemu-img create -f qcow2 ${HOME}/disk/ubuntu.img 100Gsudo ${HOME}/qemu7/build/qemu-system-aarch64 \-m 2048M -smp 4 -M virt -enable-kvm  \-boot order=dc \-drive file=${HOME}/disk/ubuntu.img,index=0,media=disk,format=qcow2  \-cdrom ${HOME}ubuntu-22.04.1-live-server-arm64.iso  \-vnc :1
qemu-system-aarch64: KVM is not supported for this guest CPU type
qemu-system-aarch64: kvm_init_vcpu: kvm_arch_init_vcpu failed (0): Invalid argumentqemu-system-aarch64: The 'host' CPU type can only be used with KVM or HVF# qemu 文档
-cpu modelSelect CPU model (-cpu help for list and additional feature selection)sudo ${HOME}/qemu7/build/qemu-system-aarch64 \-M virt \-m 2048M -smp 4 \-cpu host \-enable-kvm  \-boot order=dc \-drive file=${HOME}/disk/ubuntu.img,index=0,media=disk,format=qcow2  \-cdrom ${HOME}/ubuntu-22.04.1-live-server-arm64.iso  \-vnc :1
qemu-system-aarch64: failed to find romfile "efi-virtio.rom"# 查看qemu 文档
-L  pathSet the directory for the BIOS, VGA BIOS and keymaps.To list all the data directories, use -L help.sudo ${HOME}/qemu7/build/qemu-system-aarch64 \-M virt \-m 2048M -smp 4 \-cpu host \-enable-kvm  \-L /home/ostest/qemu7/pc-bios/ \-boot order=dc \-drive file=${HOME}/disk/ubuntu.img,index=0,media=disk,format=qcow2  \-cdrom ${HOME}/ubuntu-22.04.1-live-server-arm64.iso  \-vnc :1

-M

ostest@firefly:~$ ${HOME}/qemu7/build/qemu-system-aarch64  -M help | grep a7
ostest@firefly:~$ ${HOME}/qemu7/build/qemu-system-aarch64  -M help | grep A7
ast2600-evb          Aspeed AST2600 EVB (Cortex-A7)
bletchley-bmc        Facebook Bletchley BMC (Cortex-A7)
fuji-bmc             Facebook Fuji BMC (Cortex-A7)
mcimx6ul-evk         Freescale i.MX6UL Evaluation Kit (Cortex-A7)
mcimx7d-sabre        Freescale i.MX7 DUAL SABRE (Cortex-A7)
orangepi-pc          Orange Pi PC (Cortex-A7)
rainier-bmc          IBM Rainier BMC (Cortex-A7)
tacoma-bmc           OpenPOWER Tacoma BMC (Cortex-A7)
ostest@firefly:~$ ${HOME}/qemu7/build/qemu-system-aarch64  -M help | grep A5
xlnx-zcu102          Xilinx ZynqMP ZCU102 board with 4xA53s and 2xR5Fs based on the value of smp
ostest@firefly:~$ ${HOME}/qemu7/build/qemu-system-aarch64  -M help | grep a5
ostest@firefly:~$                                                                                                  

测试记录

ostest@firefly:~$ sudo ${HOME}/qemu7/build/qemu-system-aarch64 \
>     -M ast2600-evb \
>     -m 1G -smp 2 \
>     -cpu host \
>     --enable-kvm  \
>     -bios ${HOME}/QEMU_EFI.fd \
>     -drive if=none,file=${HOME}/ubuntu-22.04.1-live-server-arm64.iso,id=cdrom,media=cdrom\
>     -drive file=${HOME}/disk/ubuntu.img,index=1,media=disk,format=qcow2,id=ubuntuhd \
>     -vnc :1 \
>     -monitor none
Unexpected error in arm_cpu_realizefn() at ../target/arm/cpu.c:1471:
qemu-system-aarch64: Cannot enable KVM when guest CPU has EL3 enabled
Abortedostest@firefly:~$ sudo ${HOME}/qemu7/build/qemu-system-aarch64 \
>     -M orangepi-pc   \
>     -m 1G -smp 4 \
>     -cpu host \
>     --enable-kvm  \
>     -bios ${HOME}/QEMU_EFI.fd \
>     -drive if=none,file=${HOME}/ubuntu-22.04.1-live-server-arm64.iso,id=cdrom,media=cdrom\
>     -drive file=${HOME}/disk/ubuntu.img,index=1,media=disk,format=qcow2,id=ubuntuhd \
>     -vnc :1 \
>     -monitor none
qemu-system-aarch64: BIOS not supported for this machineostest@firefly:~$ sudo ${HOME}/qemu7/build/qemu-system-aarch64 \
>     -M orangepi-pc   \
>     -m 1G -smp 4 \
>     -cpu cortex-a7 \
>     --enable-kvm  \
>     -drive if=none,file=${HOME}/ubuntu-22.04.1-live-server-arm64.iso,id=cdrom,media=cdrom\
>     -drive file=${HOME}/disk/ubuntu.img,index=1,media=disk,format=qcow2,id=ubuntuhd \
>     -vnc :1 \
>     -monitor none
qemu-system-aarch64: Cannot enable KVM when guest CPU has EL3 enabled

内核启动

参考

• 编译qemu/arm64的uefi
• 为QEMU创建基于UEFI的AARCH64虚拟机

在arm64上启动qemu虚拟机有两种方式，一种是通过-kernel 的方式boot kernel，另一种是先启动uefi再boot kernel。

QEMU默认将会采用seabios的启动方式

# 下载 QEMU_EFI.fd
wget http://releases.linaro.org/components/kernel/uefi-linaro/16.02/release/qemu64/QEMU_EFI.fd

参考通过qemu-system-aarch64在x86上安装aarch64虚拟机

sudo ${HOME}/qemu7/build/qemu-system-aarch64 \-M virt \-m 2G -smp 6 \-cpu cortex-a57 \-bios ${HOME}/QEMU_EFI.fd \-drive if=none,file=${HOME}/ubuntu-22.04.1-live-server-arm64.iso,id=cdrom,media=cdrom \-device virtio-scsi-device -device scsi-cd,drive=cdrom \-drive if=none,file=${HOME}/disk/ubuntu.img,id=hd0 \-vnc :1 \-monitor stdio# 不知道为啥这个monitor 重定向到 vnc里去了# qemu 文档
-monitor devRedirect the monitor to host device dev (same devices as the serial port). The default device is vc in graphical mode and stdio in non graphical mode. Use -monitor none to disable the default monitor.

带上kvm 选项

sudo ${HOME}/qemu7/build/qemu-system-aarch64 \-M virt \-m 2G -smp 6 \-cpu host \-enable-kvm  \-bios ${HOME}/QEMU_EFI.fd \-drive if=none,file=${HOME}/ubuntu-22.04.1-live-server-arm64.iso,id=cdrom,media=cdrom \-device virtio-scsi-device -device scsi-cd,drive=cdrom \-drive if=none,file=${HOME}/disk/ubuntu.img,id=hd0 \-vnc :1 \-monitor stdio

问题

• 虚拟机开机时出现Press ESC in 5 seconds to skip startup.nsh, any other key to continue问题的解决办法

内核编译

参考

• ARMv8架构下修改Linux内核并打开kvm硬件虚拟化支持