PVE 设置硬件直通

PVE 设置硬件直通

参考文档:

建议参考官方文档。

写在前面

BIOS 开启以下设置

  • Launch CSM: Enabled
  • Boot option filter: UEFI and legacy
  • Launch PXE OpROM policy: UEFI Only
  • Launch Storage OpROM policy: UEFI Only
  • Launch Video OpROM policy: Legacy Only
  • x2apic:Enabled
  • VT-x: Enabled
  • VT-d: Enabled

开启 IOMMU

修改 /etc/default/grub 文件

1
root@HomeLab ~$ vim /etc/default/grub
1
2
3
# GRUB_CMDLINE_LINUX_DEFAULT 修改为下面这样
# Intel CPU
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

更新 grub

1
2
3
4
5
6
7
8
9
10
11
12
root@HomeLab:~# vim /etc/default/grub
root@HomeLab:~# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.85-1-pve
Found initrd image: /boot/initrd.img-5.15.85-1-pve
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done

更新 systemd-boot

1
2
3
4
5
root@HomeLab:~# pve-efiboot-tool refresh
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

重启 PVE

1
root@HomeLab ~$ reboot

验证是否开启 IOMMU

出现 DMAR: IOMMU enabled 说明开启成功。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
root@HomeLab ~$ dmesg | grep -e DMAR -e IOMMU
[ 0.011885] ACPI: DMAR 0x0000000076CDF668 0000D4 (v01 ALASKA A M I 00000001 INTL 20091013)
[ 0.011914] ACPI: Reserving DMAR table memory at [mem 0x76cdf668-0x76cdf73b]
[ 0.135283] DMAR: IOMMU enabled
[ 0.372218] DMAR: Host address width 46
[ 0.372220] DMAR: DRHD base: 0x000000fbffd000 flags: 0x0
[ 0.372225] DMAR: dmar0: reg_base_addr fbffd000 ver 1:0 cap d2008c10ef0466 ecap f0205b
[ 0.372228] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[ 0.372232] DMAR: dmar1: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[ 0.372234] DMAR: RMRR base: 0x000000773e0000 end: 0x000000773f0fff
[ 0.372236] DMAR: ATSR flags: 0x0
[ 0.372237] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[ 0.372240] DMAR-IR: IOAPIC id 1 under DRHD base 0xfbffc000 IOMMU 1
[ 0.372242] DMAR-IR: IOAPIC id 2 under DRHD base 0xfbffc000 IOMMU 1
[ 0.372243] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000
[ 0.372244] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.372875] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.840583] DMAR: No SATC found
[ 0.840585] DMAR: IOMMU feature sc_support inconsistent
[ 0.840587] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 0.840588] DMAR: dmar0: Using Queued invalidation
[ 0.840592] DMAR: dmar1: Using Queued invalidation
[ 0.843805] DMAR: Intel(R) Virtualization Technology for Directed I/O

添加必要的模块

1
root@HomeLab ~$ vim /etc/modules

添加下面内容:

1
2
3
4
vfio
vfio_pci
vfio_virqfd
vfio_iommu_type1

IOMMU Interrupt Remapping

1
2
3
root@HomeLab ~$ dmesg | grep 'remapping'
[ 0.372244] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.372875] DMAR-IR: Enabled IRQ remapping in x2apic mode

出现下面其中一条内容没说明支持 IOMMU Interrupt Remapping

  • “AMD-Vi: Interrupt remapping enabled”
  • “DMAR-IR: Enabled IRQ remapping in x2apic mode” (‘x2apic’ can be different on old CPUs, but should still work)

如果没有,则需要执行:

1
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

验证 IOMMU 隔离

执行命令后出现下面的内容,说明支持 IOMMU 分组,可以直通:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
root@HomeLab:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:00:05.1
/sys/kernel/iommu_groups/7/devices/0000:ff:14.7
/sys/kernel/iommu_groups/7/devices/0000:ff:14.5
/sys/kernel/iommu_groups/7/devices/0000:ff:14.3
/sys/kernel/iommu_groups/7/devices/0000:ff:14.1
/sys/kernel/iommu_groups/7/devices/0000:ff:14.6
/sys/kernel/iommu_groups/7/devices/0000:ff:14.4
/sys/kernel/iommu_groups/7/devices/0000:ff:14.2
/sys/kernel/iommu_groups/7/devices/0000:ff:14.0
/sys/kernel/iommu_groups/25/devices/0000:00:1d.0
/sys/kernel/iommu_groups/15/devices/0000:00:03.0
/sys/kernel/iommu_groups/33/devices/0000:06:00.0
/sys/kernel/iommu_groups/5/devices/0000:ff:12.4
/sys/kernel/iommu_groups/5/devices/0000:ff:12.0
/sys/kernel/iommu_groups/5/devices/0000:ff:12.5
/sys/kernel/iommu_groups/5/devices/0000:ff:12.1
/sys/kernel/iommu_groups/23/devices/0000:00:1c.0
/sys/kernel/iommu_groups/13/devices/0000:00:01.0
/sys/kernel/iommu_groups/31/devices/0000:03:00.0
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.6
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.4
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.2
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.0
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.5
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.3
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.1
/sys/kernel/iommu_groups/21/devices/0000:00:1a.0
/sys/kernel/iommu_groups/11/devices/0000:ff:1f.0
/sys/kernel/iommu_groups/11/devices/0000:ff:1f.2
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.6
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.4
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.2
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.0
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.7
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.5
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.3
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.1
/sys/kernel/iommu_groups/28/devices/0000:01:00.1
/sys/kernel/iommu_groups/18/devices/0000:00:05.2
/sys/kernel/iommu_groups/8/devices/0000:ff:16.2
/sys/kernel/iommu_groups/8/devices/0000:ff:16.0
/sys/kernel/iommu_groups/8/devices/0000:ff:16.7
/sys/kernel/iommu_groups/8/devices/0000:ff:16.3
/sys/kernel/iommu_groups/8/devices/0000:ff:16.1
/sys/kernel/iommu_groups/8/devices/0000:ff:16.6
/sys/kernel/iommu_groups/26/devices/0000:00:1f.0
/sys/kernel/iommu_groups/26/devices/0000:00:1f.3
/sys/kernel/iommu_groups/16/devices/0000:00:05.0
/sys/kernel/iommu_groups/6/devices/0000:ff:13.2
/sys/kernel/iommu_groups/6/devices/0000:ff:13.0
/sys/kernel/iommu_groups/6/devices/0000:ff:13.7
/sys/kernel/iommu_groups/6/devices/0000:ff:13.3
/sys/kernel/iommu_groups/6/devices/0000:ff:13.1
/sys/kernel/iommu_groups/6/devices/0000:ff:13.6
/sys/kernel/iommu_groups/24/devices/0000:00:1c.3
/sys/kernel/iommu_groups/14/devices/0000:00:01.1
/sys/kernel/iommu_groups/32/devices/0000:04:00.1
/sys/kernel/iommu_groups/32/devices/0000:04:00.0
/sys/kernel/iommu_groups/4/devices/0000:ff:10.0
/sys/kernel/iommu_groups/4/devices/0000:ff:10.7
/sys/kernel/iommu_groups/4/devices/0000:ff:10.5
/sys/kernel/iommu_groups/4/devices/0000:ff:10.1
/sys/kernel/iommu_groups/4/devices/0000:ff:10.6
/sys/kernel/iommu_groups/22/devices/0000:00:1b.0
/sys/kernel/iommu_groups/12/devices/0000:00:00.0
/sys/kernel/iommu_groups/30/devices/0000:01:00.3
/sys/kernel/iommu_groups/2/devices/0000:ff:0d.7
/sys/kernel/iommu_groups/2/devices/0000:ff:0d.5
/sys/kernel/iommu_groups/2/devices/0000:ff:0d.3
/sys/kernel/iommu_groups/2/devices/0000:ff:0d.1
/sys/kernel/iommu_groups/2/devices/0000:ff:0d.6
/sys/kernel/iommu_groups/2/devices/0000:ff:0d.4
/sys/kernel/iommu_groups/2/devices/0000:ff:0d.2
/sys/kernel/iommu_groups/2/devices/0000:ff:0d.0
/sys/kernel/iommu_groups/20/devices/0000:00:14.0
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.4
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.2
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.0
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.3
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.1
/sys/kernel/iommu_groups/29/devices/0000:01:00.2
/sys/kernel/iommu_groups/0/devices/0000:ff:0b.1
/sys/kernel/iommu_groups/0/devices/0000:ff:0b.2
/sys/kernel/iommu_groups/0/devices/0000:ff:0b.0
/sys/kernel/iommu_groups/19/devices/0000:00:05.4
/sys/kernel/iommu_groups/9/devices/0000:ff:17.7
/sys/kernel/iommu_groups/9/devices/0000:ff:17.5
/sys/kernel/iommu_groups/9/devices/0000:ff:17.3
/sys/kernel/iommu_groups/9/devices/0000:ff:17.1
/sys/kernel/iommu_groups/9/devices/0000:ff:17.6
/sys/kernel/iommu_groups/9/devices/0000:ff:17.4
/sys/kernel/iommu_groups/9/devices/0000:ff:17.2
/sys/kernel/iommu_groups/9/devices/0000:ff:17.0
/sys/kernel/iommu_groups/27/devices/0000:01:00.0