在 LXC 容器中搭建 GitLab 简单记录

注意距离 GitLab 实际搭建时间有过了几周了，这篇博客完全依照搭建时的 txt 记录以及我的记忆，仅供参考。

环境： Debian Bookworm LXC

安装

安装过程非常简单，因为 gitlab 基本做到了开箱即用，注意这是 gitlab-ee 版本，但是默认只有 gitlab-ce 的功能

注意虽然 gitlab 依赖于 postgresql 和 redis ，但是不需要使用 apt 安装，因为 gitlab 自带有

首先按照 gitlab 官方的安装指引，运行这个脚本，它的目的是配置上 gitlab 的软件源并刷新缓存

curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh -o gitlab.sh
sudo bash gitlab.sh

直接安装

sudo apt-get install gitlab-ee

......

It looks like GitLab has not been configured yet; skipping the upgrade script.

       *.                  *.
      ***                 ***
     *****               *****
    .******             *******
    ********            ********
   ,,,,,,,,,***********,,,,,,,,,
  ,,,,,,,,,,,*********,,,,,,,,,,,
  .,,,,,,,,,,,*******,,,,,,,,,,,,
      ,,,,,,,,,*****,,,,,,,,,.
         ,,,,,,,****,,,,,,
            .,,,***,,,,
                ,*,.



     _______ __  __          __
    / ____(_) /_/ /   ____ _/ /_
   / / __/ / __/ /   / __ `/ __ \
  / /_/ / / /_/ /___/ /_/ / /_/ /
  \____/_/\__/_____/\__,_/_.___/


Thank you for installing GitLab!
GitLab was unable to detect a valid hostname for your instance.
Please configure a URL for your GitLab instance by setting `external_url`
configuration in /etc/gitlab/gitlab.rb file.
Then, you can start your GitLab instance by running the following command:
  sudo gitlab-ctl reconfigure

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

Help us improve the installation experience, let us know how we did with a 1 minute survey:
https://gitlab.fra1.qualtrics.com/jfe/form/SV_6kVqZANThUQ1bZb?installation=omnibus&release=16-5

Processing triggers for libc-bin (2.36-9+deb12u3) ...

在 LXC 非特权容器中，需要关闭内核相关的特性

sudo vim /etc/gitlab/gitlab.rb

package['modify_kernel_parameters'] = false

由于我的环境中 nginx 和 gitlab 不在同一个机器上，且当 gitlab 被配置为 https 时，会自动获取证书，所以还需要一些额外配置

sudo vim /etc/gitlab/gitlab.rb

letsencrypt['enable'] = false
nginx['enable'] = false

日志里经常能看到这样的东西，但是不太清楚啥意思

2023-10-25_04:03:07.07624 E, [2023-10-25T04:03:07.076078 #106306] ERROR -- : Error connecting to the database: PQsocket() can't get socket descriptor
2023-10-25_04:03:07.07655 E, [2023-10-25T04:03:07.076481 #106306] ERROR -- : Error connecting to the database: PQsocket() can't get socket descriptor
2023-10-25_04:03:07.07691 E, [2023-10-25T04:03:07.076801 #106306] ERROR -- : Error connecting to the database: PQsocket() can't get socket descriptor
2023-10-25_04:03:07.11613 ::1 - - [25/Oct/2023:04:03:07 UTC] "GET /database HTTP/1.1" 200 0
2023-10-25_04:03:07.11616 - -> /database

这样其实就安装结束了，但是，它监听在哪里呢？

一开始我以为要用 puma ，但是需要一系列 nginx 操作，实际上 gitlab 服务器为 workhorse ，默认监听在 unix socket ，需要改为 tcp

sudo vim /etc/gitlab/gitlab.rb

gitlab_workhorse['listen_network'] = "tcp"
gitlab_workhorse['listen_addr'] = "0.0.0.0:8891"

然后再配置 nginx

安装 gitlab runner

参考 https://docs.gitlab.com/ee/tutorials/create_register_first_runner/#create-and-register-a-project-runner和https://docs.gitlab.com/runner/executors/custom_examples/libvirt.html 安装

curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
sudo chmod +x /usr/local/bin/gitlab-runner
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
gitlab-runner register  --url https://gitlab.inuyasha.love  --token glrt-Z6TxqBcDixmTJVT9WKYo
vim /etc/gitlab-runner/config.toml
gitlab-runner run

如果出现这样的东西，那是大结界导致的

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
curl: (56) Recv failure: Connection reset by peer
virt-builder: error: external command ‘'curl' -q --config
'/tmp/virt-builder.njQovh/guestfscurlc08643.conf'’ exited with error 56

If reporting bugs, run virt-builder with debugging enabled and include the
complete output:

  virt-builder -v -x [...]

The server behind archive.libguestfs.org is down - an engineer is coming today
to look at it.

In the meantime you can edit /etc/virt-builder/repos.d/libguestfs.conf and
comment out the [archive.libguestfs.org] section.

libvirt runner 配置

用 virt-builder 建立虚拟机

virt-builder debian-11 \
    --size 32G \
    --output /srv/runner/libvirt/images/gitlab-runner-base.qcow2 \
    --format qcow2 \
    --hostname gitlab-runner-bullseye \
    --network \
    --run-command 'sed -i "s/http:\/\/deb.debian.org/https:\/\/mirrors.bfsu.edu.cn/" /etc/apt/sources.list' \
    --run-command 'sed -i "s/http:\/\/security.debian.org/https:\/\/mirrors.bfsu.edu.cn/" /etc/apt/sources.list' \
    --install curl \
    --run-command 'curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | bash' \
    --run-command 'curl -s "https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh" | bash' \
    --run-command 'useradd -m -p "" gitlab-runner -s /bin/bash' \
    --install gitlab-runner,git,git-lfs,openssh-server \
    --run-command "git lfs install --skip-repo" \
    --ssh-inject gitlab-runner:file:/root/.ssh/id_rsa.pub \
    --run-command "echo 'gitlab-runner ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers" \
    --run-command "sed -E 's/GRUB_CMDLINE_LINUX=\"\"/GRUB_CMDLINE_LINUX=\"net.ifnames=0 biosdevname=0\"/' -i /etc/default/grub" \
    --run-command "grub-mkconfig -o /boot/grub/grub.cfg" \
    --run-command "echo 'auto eth0' >> /etc/network/interfaces" \
    --run-command "echo 'allow-hotplug eth0' >> /etc/network/interfaces" \
    --run-command "echo 'iface eth0 inet dhcp' >> /etc/network/interfaces"

如果遇到 curl: (56) Recv failure: Connection reset by peer ，同样是结界的问题，可以设置 http_proxy 环境变量

另外网桥默认没开，这个报错可以在 libvirtd 看到

ERROR    Requested operation is not valid: network 'default' is not active

打开它

sudo virsh net-autostart default

 Name      State      Autostart   Persistent
----------------------------------------------
 default   inactive   no          yes

sudo virsh net-autostart default
sudo virsh net-start default

嵌套虚拟化

libvirtd 会出现一些额外的问题

首先解决 /dev/kvm

WARNING  KVM acceleration not available, using 'qemu'

在 LXC 容器配置中挂载 kvm

lxc.cgroup2.devices.allow = c 10:232 rwm
lxc.mount.entry = /dev/kvm dev/kvm none bind,create=file 0 0

禁用 remember owner

ERROR    Unable to set XATTR trusted.libvirt.security.dac on /var/lib/libvirt/qemu/domain-3-runner-1-project-1-c/master-key.aes: Operation not permitted

配置

vim /etc/libvirt/qemu.conf

remember_owner = 0

如果遇到 tun 节点没有

ERROR    Unable to open /dev/net/tun, is tun module loaded?: No such file or directory

net/tun 设备也需要像 kvm 那样设置，另外注意注释掉 userns

#lxc.include = /usr/share/lxc/config/userns.conf
lxc.cgroup2.devices.allow = c 10:200 rwm
lxc.mount.entry = /dev/net dev/net none bind,create=dir

另外由于 namespace 特性， libvirtd 会 chown 一些设备导致失败

libvirtd[6711]: Failed to chown device /dev/urandom: Operation not permitted

有一些办法禁用

vim /etc/libvirt/qemu.conf
cgroup_device_acl = []

并 virt-install 添加参数 --rng builtin

另外可以参考这里 https://gitlab.com/libvirt/libvirt/-/issues/556

by ISCAS weilinfox