注意距离 Jenkins CI 实际搭建时间有过了几周了，这篇博客完全依照搭建时的 txt 记录以及我的记忆，仅供参考。

安装

官方中文文档 https://www.jenkins.io/zh/doc/

在 Archlinux 中搭建非常简单

sudo pacman -S jenkins
sudo systemctl enable --now jenkins

此时可以看到首页了

curl localhost:8080

Libvirt Agents

为了使用 libvirt agents ，安装 "Libvirt Agents" 插件

pacman -S libvirt dnsmasq qemu-full
systemctl enable --now libvirtd

建立 QEMU cloud ： New cloud -> Hypervisor Type -> QEMU -> Hypervisor Transport -> unix

将 jenkins 用户加入 libvirt 用户组，否则连接报 error: authentication unavailable: no polkit agent available to authenticate action 'org.libvirt.unix.manage'

sudo usermod -a -G libvirt jenkins

Test connection 成功后保存即可

此时可将 Built-In Node 数量设为 0 （其实在网页会有相关黄色框框提示，所有提示不要 Dismiss ，都要按文档修复）

虚拟机镜像制作

pacman -S guestfs-tools virt-install linux

安装 linux 包是因为 virt-builder 需要 /boot 目录下的内核和内核模块

打开默认的网桥设备

sudo virsh net-autostart default
sudo virsh net-start default
virsh net-edit default

配置以解决 libvirtd 在 LXC 非特权容器会遇到的问题

vim /etc/libvirt/qemu.conf

remember_owner = 0
cgroup_device_acl = []

重启之

systemctl restart libvirtd

Debian 镜像

这个镜像是用于 mugen 测试的，所以以 root 用户登陆

virt-builder debian-11 \
    --size 32G \
    --output /var/lib/libvirt/images/bullseye-base.qcow2 \
    --format qcow2 \
    --hostname bullseye-test \
    --network \
    --run-command 'sed -i "s/http:\/\/deb.debian.org/https:\/\/mirrors.bfsu.edu.cn/" /etc/apt/sources.list' \
    --run-command 'sed -i "s/http:\/\/security.debian.org/https:\/\/mirrors.bfsu.edu.cn/" /etc/apt/sources.list' \
    --run-command 'apt-get update' \
    --ssh-inject root:file:/root/.ssh/id_ed25519.pub \
    --run-command "sed -E 's/GRUB_CMDLINE_LINUX=\"\"/GRUB_CMDLINE_LINUX=\"net.ifnames=0 biosdevname=0\"/' -i /etc/default/grub" \
    --run-command "grub-mkconfig -o /boot/grub/grub.cfg" \
    --run-command "echo 'auto eth0' >> /etc/network/interfaces" \
    --run-command "echo 'allow-hotplug eth0' >> /etc/network/interfaces" \
    --run-command "echo 'iface eth0 inet dhcp' >> /etc/network/interfaces"

会生成一个随机 root 密码，保存好， ssh 用： virt-builder: Setting random password of root to MWlK7j07t0BBtQMb

这里建立一个 debian11 镜像

qemu-img create -f qcow2 -b /var/lib/libvirt/images/bullseye-base.qcow2 /var/lib/libvirt/images/bullseye-ready.qcow2 -F qcow2

virt-install \
    --name bullseye-virt \
    --os-variant debian11 \
    --disk /var/lib/libvirt/images/bullseye-ready.qcow2 \
    --import \
    --vcpus=2 \
    --ram=2048 \
    --network default \
    --graphics none \
    --noautoconsole \
    --noreboot

--noreboot 禁用自动启动，此时手动启动

virsh start bullseye-virt

看看 ip （可能没有用，因为 sshd 不知道为啥炸了，看起来 reinstall 一下就会好）

virsh -q domifaddr bullseye-virt

更新一下软件包啥的，其实是进去看一下具体情况

virsh console bullseye-virt

看一下 mac

virsh dumpxml bullseye-virt | grep "mac address"

弄静态 ip

virsh net-edit default
<host mac='52:54:00:4a:54:26' name='bullseye-virt' ip='192.168.123.2'/>

virsh shutdown bullseye-virt
virsh net-destroy default
virsh net-start default
virsh start bullseye-virt

别忘了需要 openJDK 环境

apt-get install openjdk-11-jdk

建立快照

virsh snapshot-create bullseye-virt
virsh snapshot-revert bullseye-virt <snapshot>

如果要删除虚拟机

virsh shutdown bullseye-virt
virsh snapshot-delete --current bullseye-virt
virsh undefine bullseye-virt

另外下面给出一个通常用于构建的镜像，以 bookworm 为例

virt-builder debian-12 \
    --size 32G \
    --output /var/lib/libvirt/images/bookworm-base.qcow2 \
    --format qcow2 \
    --hostname bookworm-runner \
    --network \
    --run-command 'sed -i "s/http:\/\/deb.debian.org/https:\/\/mirrors.bfsu.edu.cn/" /etc/apt/sources.list' \
    --run-command 'sed -i "s/http:\/\/security.debian.org/https:\/\/mirrors.bfsu.edu.cn/" /etc/apt/sources.list' \
    --install sudo,default-jdk,openssh-server \
    --run-command 'useradd -m -p "" jenkins -s /bin/bash' \
    --ssh-inject jenkins:file:/root/.ssh/id_ed25519.pub \
    --run-command "echo 'jenkins ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers" \
    --run-command "sed -E 's/GRUB_CMDLINE_LINUX=\"\"/GRUB_CMDLINE_LINUX=\"net.ifnames=0 biosdevname=0\"/' -i /etc/default/grub" \
    --run-command "grub-mkconfig -o /boot/grub/grub.cfg" \
    --run-command "echo 'auto eth0' >> /etc/network/interfaces" \
    --run-command "echo 'allow-hotplug eth0' >> /etc/network/interfaces" \
    --run-command "echo 'iface eth0 inet dhcp' >> /etc/network/interfaces"

给出一个带 docker 支持的 Agent ，注意 qemu-user-static 建议安装当前最新版本，具体版本号自行检查

virt-builder debian-12 \
    --size 32G \
    --output /var/lib/libvirt/images/bookworm-base.qcow2 \
    --format qcow2 \
    --hostname bookworm-runner \
    --network \
    --run-command 'sed -i "s/http:\/\/deb.debian.org/https:\/\/mirrors.bfsu.edu.cn/" /etc/apt/sources.list' \
    --run-command 'sed -i "s/http:\/\/security.debian.org/https:\/\/mirrors.bfsu.edu.cn/" /etc/apt/sources.list' \
    --run-command "echo deb https://mirrors.bfsu.edu.cn/debian/ bookworm-backports main contrib non-free non-free-firmware >> /etc/apt/sources.list" \
    --install sudo,default-jdk,openssh-server,docker.io,qemu-user-static=1:8.1.2+ds-1~bpo12+1 \
    --run-command 'useradd -m -p "" jenkins -s /bin/bash' \
    --ssh-inject jenkins:file:/root/.ssh/id_rsa.pub \
    --run-command "usermod -a -G docker jenkins" \
    --run-command "echo 'jenkins ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers" \
    --run-command "sed -E 's/GRUB_CMDLINE_LINUX=\"\"/GRUB_CMDLINE_LINUX=\"net.ifnames=0 biosdevname=0\"/' -i /etc/default/grub" \
    --run-command "grub-mkconfig -o /boot/grub/grub.cfg" \
    --run-command "echo 'auto eth0' >> /etc/network/interfaces" \
    --run-command "echo 'allow-hotplug eth0' >> /etc/network/interfaces" \
    --run-command "echo 'iface eth0 inet dhcp' >> /etc/network/interfaces"

Ubuntu 镜像

其实也是可以用 virt-builder 来的，可以 list 到镜像，不过有点老？

virt-builder --list | grep ubuntu

但是这里以 Ubuntu 发布的镜像 https://cloud-images.ubuntu.com/releases/jammy/release/ubuntu-22.04-server-cloudimg-amd64-disk-kvm.img 作为另一种方法的演示

其实这个 img 是个 qcow2 镜像，而不是 raw

file ubuntu-22.04-server-cloudimg-amd64-disk-kvm.img

ubuntu-22.04-server-cloudimg-amd64-disk-kvm.img: QEMU QCOW Image (v2), 2361393152 bytes (v2), 2361393152 bytes

看一下分区

virt-df -h ubuntu-22.04-server-cloudimg-amd64-disk-kvm.img

Filesystem                                Size       Used  Available  Use%
ubuntu-22.04-server-cloudimg-amd64-disk-kvm.img:/dev/sda1
                                          2.0G       1.3G       634M   68%
ubuntu-22.04-server-cloudimg-amd64-disk-kvm.img:/dev/sda15
                                          104M       6.0M        98M    6%

分区是很小的，需要扩大分区

有的博客会说用下面的方法来扩大分区，实际上这样会使 grub 配置失效，无法启动

# 不要这样做
# qemu-img create -f qcow2 ubuntu-22.04-server-cloudimg-amd64-disk-kvm-expand.img 32G
# virt-resize --expand /dev/sda1 ubuntu-22.04-server-cloudimg-amd64-disk-kvm.img ubuntu-22.04-server-cloudimg-amd64-disk-kvm-expand.img

下面做个正确示范，首先备份，然后扩大镜像

cp ubuntu-22.04-server-cloudimg-amd64-disk-kvm.img ubuntu-22.04-server-cloudimg-amd64-disk-kvm-expand.img
qemu-img resize ubuntu-22.04-server-cloudimg-amd64-disk-kvm-expand.img 32G

我们需要 nbd 来挂载并用 fdisk 来操作分区

sudo modprobe nbd max_part=6
sudo qemu-nbd -c /dev/nbd0 ubuntu-22.04-server-cloudimg-amd64-disk-kvm-expand.img
sudo fdisk /dev/nbd0

fdisk 会报了下面这个，这个在直接后面编辑后 w 写入分区表就能解决

GPT PMBR size mismatch (4612095 != 67108863) will be corrected by write.
The backup GPT table is not on the end of the device. This problem will be corrected by write.

一个重点是分区 UUID 也不能变，因为 grub 需要它，变化了 UUID 可以预料到这样的报错 PARTUUID=df2ceeda-81fa-4699-b494-63b0e7cb31f3 does not exist

现在来编辑分区，其实就是删除旧分区，然后在原位建立一个新分区，这样只是参数变了其实数据都在，同时也要记录旧的 UUID ，将新的 UUID 改回去

$ sudo fdisk /dev/nbd0

Welcome to fdisk (util-linux 2.39.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

GPT PMBR size mismatch (4612095 != 67108863) will be corrected by write.
The backup GPT table is not on the end of the device. This problem will be corrected by write.

Command (m for help): x

Expert command (m for help): p

Disk /dev/nbd0: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C955BBCE-6F5E-4066-A8A1-80861F993C50
First usable LBA: 34
Last usable LBA: 67108830
Alternative LBA: 67108863
Partition entries starting LBA: 2
Allocated partition entries: 128
Partition entries ending LBA: 33

Device        Start     End Sectors Type-UUID                            UUID                                 Name Attrs
/dev/nbd0p1  227328 4612062 4384735 0FC63DAF-8483-4772-8E79-3D69D8477DE4 DF2CEEDA-81FA-4699-B494-63B0E7CB31F3
/dev/nbd0p14   2048   10239    8192 21686148-6449-6E6F-744E-656564454649 63C6D1D7-D820-4163-8D7C-BF78F7ECA991
/dev/nbd0p15  10240  227327  217088 C12A7328-F81F-11D2-BA4B-00A0C93EC93B 653F9585-90C4-48BA-B947-DBE802BFEFF3

Partition table entries are not in disk order.

Expert command (m for help): r

Command (m for help): d
Partition number (1,14,15, default 15): 1

Partition 1 has been deleted.

Command (m for help): n
Partition number (1-13,16-128, default 1):
First sector (34-67108830, default 227328): 227328
Last sector, +/-sectors or +/-size{K,M,G,T,P} (227328-67108830, default 67106815):

Created a new partition 1 of type 'Linux filesystem' and of size 31.9 GiB.
Partition #1 contains a ext4 signature.

Do you want to remove the signature? [Y]es/[N]o: N

Command (m for help): x

Expert command (m for help): p

Disk /dev/nbd0: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C955BBCE-6F5E-4066-A8A1-80861F993C50
First usable LBA: 34
Last usable LBA: 67108830
Alternative LBA: 67108863
Partition entries starting LBA: 2
Allocated partition entries: 128
Partition entries ending LBA: 33

Device        Start      End  Sectors Type-UUID                            UUID                                 Name Attrs
/dev/nbd0p1  227328 67106815 66879488 0FC63DAF-8483-4772-8E79-3D69D8477DE4 4FEE75C4-3FDF-466B-9B47-5823C441E3D1
/dev/nbd0p14   2048    10239     8192 21686148-6449-6E6F-744E-656564454649 63C6D1D7-D820-4163-8D7C-BF78F7ECA991
/dev/nbd0p15  10240   227327   217088 C12A7328-F81F-11D2-BA4B-00A0C93EC93B 653F9585-90C4-48BA-B947-DBE802BFEFF3

Partition table entries are not in disk order.

Expert command (m for help): u
Partition number (1,14,15, default 15): 1

New UUID (in 8-4-4-4-12 format): DF2CEEDA-81FA-4699-B494-63B0E7CB31F3

Partition UUID changed from 4FEE75C4-3FDF-466B-9B47-5823C441E3D1 to DF2CEEDA-81FA-4699-B494-63B0E7CB31F3.

Expert command (m for help): r

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

下面扩展一下分区

sudo e2fsck -f /dev/nbd0p1
sudo resize2fs /dev/nbd0p1

好了从 nbd 解除绑定

sudo qemu-nbd -d /dev/nbd0

建立一个 ready 镜像来测试

qemu-img create -f qcow2 -F qcow2 -b ubuntu-22.04-server-cloudimg-amd64-disk-kvm-expand.img ubuntu-22.04-server-cloudimg-amd64-disk-kvm-ready.qcow2

测试启动成功，下面来配置镜像——添加 key ，更改软件源，更改 root 密码

qemu-img create -f qcow2 -F qcow2 -b ubuntu-22.04-server-cloudimg-amd64-disk-kvm-expand.img ubuntu-22.04-server-cloudimg-amd64-disk-kvm-ready.qcow2
sudo qemu-nbd -c /dev/nbd0 ubuntu-22.04-server-cloudimg-amd64-disk-kvm-ready.qcow2
sudo mount /dev/nbd0p1 /mnt/img
sudo chroot /mnt/img

......

sudo umount /mnt/img

安装配置好的镜像

virt-install \
    --name jammy-virt \
    --os-variant ubuntujammy \
    --disk /var/lib/libvirt/images/ubuntu-22.04-server-cloudimg-amd64-disk-kvm-ready.qcow2 \
    --import \
    --vcpus=2 \
    --ram=2048 \
    --network default \
    --graphics none \
    --noautoconsole \
    --noreboot

同样配置静态 ip

virsh dumpxml jammy-virt | grep "mac address"
virsh net-edit default

<host mac='52:54:00:fd:51:fc' name='jammy-virt' ip='192.168.123.3'/>

这里要重启 net ，记得先确认没有虚拟机在运作

virsh net-destroy default
virsh net-start default

启动虚拟机看是否生效

virsh start jammy-virt

看看 ip ？

virsh -q domifaddr jammy-virt

两分钟后启动完成，但是没有获取到 ip ，乐

检查镜像使用的是 systemd-networkd ，然后并没有配置，手动配置一下

cat /etc/systemd/network/enp1s0.network

[Match]
Name=enp1s0
[Network]
DHCP=true
[DHCPv4]
UseDomains=true

试一下

systemctl restart systemd-networkd

可以看到了，现在可以配置软件包，注意 ssh 需要重装才能使用

apt-get reinstall openssh-server
apt-get install openjdk-11-jdk

还需要禁用 apt 的自动更新，为什么服务器镜像会有这种东西

nano /etc/apt/apt.conf.d/20auto-upgrades

APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Unattended-Upgrade "0";

关机

poweroff

建立快照

virsh snapshot-create jammy-virt

openEuler 镜像

openEuler 镜像可以无需配置直接使用，但是要求较高，一个是 UEFI 启动，二个是 --cpu 默认 qemu64 会 kp ，改成 Haswell 可行

virt-install \
    --name openEuler-2309-x86_64 \
    --os-variant fedora-unknown \
    --cpu Haswell \
    --disk /var/lib/libvirt/images/openEuler-23.09-x86_64-ready.qcow2 \
    --import \
    --vcpus=2 \
    --ram=2048 \
    --network default \
    --graphics none \
    --noautoconsole \
    --boot uefi

安装之

virt-install \
    --name openEuler-2203-x86_64 \
    --os-variant fedora-unknown \
    --disk /var/lib/libvirt/images/openEuler-22.03-LTS-SP2-x86_64-ready.qcow2 \
    --import \
    --vcpus=2 \
    --ram=2048 \
    --network default \
    --graphics none \
    --noautoconsole

启动后安装需要的软件包

dnf install java-11-openjdk -y

openEuler RISC-V

openEuler 2309 RISC-V 可以在这里下载 https://mirrors.tuna.tsinghua.edu.cn/openeuler/openEuler-23.09/virtual_machine_img/riscv64/

直接安装之

virt-install \
    --name openEuler-2309-riscv64 \
    --arch riscv64 \
    --boot loader=/var/lib/libvirt/images/fw_payload_oe_uboot_2304.bin \
    --os-variant fedora-unknown \
    --disk /var/lib/libvirt/images/openEuler-23.09-RISC-V-qemu-riscv64-ready.qcow2 \
    --import \
    --vcpus=2 \
    --ram=2048 \
    --network default \
    --graphics none \
    --noautoconsole

注意软件源中，除了 OS 以外的软件源全部 enabled=0 ，不知道不存在的软件源默认 enable 是想干啥

另外软件源中的 java-1.8.0-openjdk 版本过低，使用这里的 java-11 https://repo.tarsier-infra.com/openEuler-RISC-V/obs/hwobs_2309/riscv64/

Fedora 镜像

看一下是有的，所以用 virt-builder

virt-install --osinfo list | grep fedora
virt-builder fedora-38 \
    --size 32G \
    --output /var/lib/libvirt/images/fedora38-base.qcow2 \
    --format qcow2 \
    --hostname fedora38-test \
    --network \
    --run-command 'sed -e "s|^metalink=|#metalink=|g" -e "s|^#baseurl=http://download.example/pub/fedora/linux|baseurl=https://mirrors.bfsu.edu.cn/fedora|g" -i.bak /etc/yum.repos.d/fedora.repo /etc/yum.repos.d/fedora-modular.repo /etc/yum.repos.d/fedora-updates.repo /etc/yum.repos.d/fedora-updates-modular.repo' \
    --ssh-inject root:file:/root/.ssh/id_ed25519.pub

记得复制终端的随机密码。Fedora 使用 NetworkManager ，所以不用担心网络问题了，也可以后续直接 ssh 进入

本来添加了 --run-command 'dnf update -y' 和 --install java-11-openjdk ，但是出现了下面所示有点意外的错误，所以最后还是手动安装了

Errors during downloading metadata for repository 'fedora':
  - Curl error (6): Couldn't resolve host name for https://mirrors.bfsu.edu.cn/fedora/releases/38/Everything/x86_64/os/repodata/repomd.xml [Could not resolve host: mirrors.bfsu.edu.cn]
Error: Failed to download metadata for repo 'fedora': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

安装之

qemu-img create -f qcow2 -F qcow2 -b /var/lib/libvirt/images/fedora38-base.qcow2 /var/lib/libvirt/images/fedora38-ready.qcow2

virt-install \
    --name fedora38-x86_64 \
    --os-variant fedora38 \
    --disk /var/lib/libvirt/images/fedora38-ready.qcow2 \
    --import \
    --vcpus=2 \
    --ram=2048 \
    --network default \
    --graphics none \
    --noautoconsole

看一下 ip 然后直接 ssh 进入

virsh -q domifaddr fedora38-x86_64

当然也可以 virsh console fedora38-x86_64 用密码登陆

dnf update -y && dnf install java-11-openjdk -y && dnf clean packages && poweroff

打快照

virsh snapshot-create fedora38-x86_64

静态 ip

virsh dumpxml fedora38-x86_64 | grep "mac address"
virsh net-edit default
<host mac='52:54:00:e7:fb:8b' name='fedora38-x86_64' ip='192.168.123.8'/>

重启 net

virsh net-destroy default
virsh net-start default

另外要注意快照也会保存虚拟机配置的，如果更改虚拟机 vcpu 和 memory 配置又不想重新打快照，注意修改 /var/lib/libvirt/qemu/snapshot/ 下快照的配置，否则 virsh edit 修改后回复快照会被覆盖

进入 Jenkins CI

直接用 webui 连接

Jenkins Pipeline

编写 jenkins pipeline 时，注意每个单独的命令都会回到 ${env.WORKSPACE}

by ISCAS weilinfox