About This Document

This document describes how to use kubeadm to deploy a Kubernetes cluster, set up a Kubernetes+iSulad environment, deploy GitLab Runner in the environment, and perform verification tests by executing pipeline cases. This document consists of the following parts:

1. Kubernetes+iSulad environment deployment
2. GitLab Runner deployment and tesing

Note: All operations in this document are executed with root privileges.

Kubernetes+iSulad Environment Deployment

Preparing Cluster Hosts

Prepare at least three openEuler hosts. openEuler 22.03 LTS or later is recommended. The following table lists the host information.

Images and Software

The following table lists the software and images required during installation. The version numbers are for reference only. If the Internet is unavailable in the environment, download the required software packages, dependencies, and images in advance.

• Download the software packages from https://repo.openeuler.org/openEuler-22.03-LTS-SP1/.
• Download the images from the Docker Hub official website https://hub.docker.com.

Modifying the Hosts Files

1. Change the host name. The following uses one host as an example.

   # hostnamectl set-hostname lab1
   # sudo -i
   

2. Open the /etc/hosts files of the three hosts.

   # vim /etc/hosts
   

3. Add the following content (IP address + host name) to the hosts file:

   197.xxx.xxx.xxx lab1
   197.xxx.xxx.xxx lab2
   197.xxx.xxx.xxx lab3
   

Setting Up the Environment

1. Disable the firewall.

   # systemctl stop firewalld
   # systemctl disable firewalld
   

2. Disable SELinux.

   # setenforce 0
   

3. Disable the system swap.

   # swapoff -a
   # sed -ri 's/.*swap.*/#&/' /etc/fstab
   

4. Enable the forwarding mechanisms.

   # cat > /etc/sysctl.d/kubernetes.conf <<EOF
   net.bridge.bridge-nf-call-iptables = 1
   net.ipv4.ip_forward = 1
   net.bridge.bridge-nf-call-ip6tables = 1
   vm.swappiness=0
   EOF
   

5. Make the rules take effect.

   # modprobe overlay
   # modprobe br_netfilter
   # sysctl -p /etc/sysctl.d/kubernetes.conf
   

6. Configure the startup script.

   # vim /etc/init.d/k8s.sh
   

   • Add the following content to the k8s.sh script:

     #!/bin/sh
     modprobe br_netfilter
     sysctl -w net.bridge.bridge-nf-call-ip6tables = 1
     sysctl -w net.bridge.bridge-nf-call-iptables = 1
     

   • Change the script permissions.

     # chmod +x /etc/init.d/k8s.sh
     

7. Add the configuration file.

   # vim br_netfilte.service
   
   [Unit]
   Description=To enable the core module br_netfilter when reboot
   After=default.target
   [Service]
   ExecStart=/etc/init.d/k8s.sh //The script can be customized.
   [Install]
   WantedBy=default.target
   

   • Start the service.

     # systemctl daemon-reload
     # systemctl enable br_netfilter.service
     

8. Configure sysctl.

   # sed -i "s/net.ipv4.ip_forward=0/net.ipv4.ip_forward=1/g" /etc/sysctl.conf
   # sed -i 12a\vm.swappiness=0 /etc/sysctl.conf
   

Installing kubeadm, kubectl, kubelet, and iSulad

1. Run yum install to install the software packages.

# yum install -y kubernetes-kubeadm
# yum install -y kubernetes-client
# yum install -y kubernetes-kubelet
# yum install -y iSulad

2. Set kubelet to start upon system boot.

# systemctl enable kubelet

Modifying the iSulad Configurations

1. Go to /etc/isulad and open the daemon.json file.

# vi /etc/isulad/daemon.json

2. Modify and add the following content to the configuration file:

{
  "group": "isula",
  "default-runtime": "runc",
  "graph": "/var/lib/isulad",
  "state": "/var/run/isulad",
  "engine": "lcr",
  "log-level": "ERROR",
  "pidfile": "/var/run/isulad.pid",
  "log-opts": {
      "log-file-mode": "0600",
      "log-path": "/var/lib/isulad",
      "max-file": "1",
      "max-size": "30KB"
  },
  "log-driver": "stdout",
  "container-log": {
      "driver": "json-file"
  },
  "hook-spec": "/etc/default/isulad/hooks/default.json",
  "start-timeout": "2m",
  "storage-driver": "overlay2",
  "storage-opts": [
      "overlay2.override_kernel_check=true"
  ],
  "registry-mirrors": [
              "docker.io"
  ],
  "insecure-registries": [
               "k8s.gcr.io",
               "quay.io",
               "oci.inhuawei.com",
               "rnd-dockerhub.huawei.com",
               "registry.aliyuncs.com",
               "[IP address of the local private image repository]"
  ],
  "pod-sandbox-image": "k8s.gcr.io/pause:3.2",
  "native.umask": "normal",
  "network-plugin": "cni",
  "cni-bin-dir": "/opt/cni/bin",
  "cni-conf-dir": "/etc/cni/net.d",
  "image-layer-check": false,
  "use-decrypted-key": true,
  "insecure-skip-verify-enforce": false,
  "cri-runtimes": {
      "kata": "io.containerd.kata.v2"
  }
}

3. Restart the iSulad service.

# systemctl restart isulad

Loading the iSulad Images

1. View the required images.

   # kubeadm config images list
   

Pay attention to the version numbers. The following figure shows the command output.

2. Run isula to pull the required images.

   Note: The following downloaded image versions are examples. Use the actual versions in the command output.

   # isula pull k8simage/kube-apiserver:v1.20.15
   # isula pull k8smx/kube-controller-manager:v1.20.15
   # isula pull k8smx/kube-scheduler:v1.20.15
   # isula pull k8smx/kube-proxy:v1.20.15
   # isula pull k8smx/pause:3.2
   # isula pull k8smx/coredns:1.7.0
   # isula pull k8smx/etcd:3.4.13-0
   

3. Modify the tags of the downloaded images.

   # isula tag k8simage/kube-apiserver:v1.20.15 k8s.gcr.io/kube-apiserver:v1.20.15
   # isula tag k8smx/kube-controller-manager:v1.20.15 k8s.gcr.io/kube-controller-manager:v1.20.15
   # isula tag k8smx/kube-scheduler:v1.20.15 k8s.gcr.io/kube-scheduler:v1.20.15
   # isula tag k8smx/kube-proxy:v1.20.15 k8s.gcr.io/kube-proxy:v1.20.15
   # isula tag k8smx/pause:3.2 k8s.gcr.io/pause:3.2
   # isula tag k8smx/coredns:1.7.0 k8s.gcr.io/coredns:1.7.0
   # isula tag k8smx/etcd:3.4.13-0 k8s.gcr.io/etcd:3.4.13-0
   

4. Delete old images.

   # isula rmi k8simage/kube-apiserver:v1.20.15
   # isula rmi k8smx/kube-controller-manager:v1.20.15
   # isula rmi k8smx/kube-scheduler:v1.20.15
   # isula rmi k8smx/kube-proxy:v1.20.15
   # isula rmi k8smx/pause:3.2
   # isula rmi k8smx/coredns:1.7.0
   # isula rmi k8smx/etcd:3.4.13-0
   

5. View the pulled images.

   # isula images
   

Installing crictl

# yum install -y cri-tools

Initializing the Master Node

Run the following command to initialize the master node:

# kubeadm init --kubernetes-version v1.20.2 --cri-socket=/var/run/isulad.sock --pod-network-cidr=[IP address range allocated to the pod network]

//Description of the preceding parameters
kubernetes-version indicates the version to be installed.
cri-socket specifies the iSulad engine.
pod-network-cidr specifies the IP address range allocated to the pod network.

Enter the following commands as prompted:

# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config

After the initialization is successful, copy the last two lines and run the copied commands on the node to add the node to the master cluster. If the preceding commands are not recorded, run the following command:

# kubeadm token create --print-join-command

Adding a Node to a Cluster

Paste the kubeadm join ... command generated during initialization on the master node, add --cri-socket=/var/run/isulad.sock in front of discovery, and run the following command:

# kubeadm join [IP address] --token bgyis4.euwkjqb7jwuenwvs --cri-socket=/var/run/isulad.sock --discovery-token-ca-cert-hash sha256:3792f02e136042e2091b245ac71c1b9cdcb97990311f9300e91e1c339e1dfcf6

Installing the Calico CNI

1. Pull the Calico images.

   Configure the Calico CNI on the master node and pull the images of the required version on each node in advance.

   isula pull calico/node:v3.14.2
   isula pull calico/cni:v3.14.2
   isula pull calico/kube-controllers:v3.14.2
   isula pull calico/pod2daemon-flexvol:v3.14.2
   

2. Obtain the configuration file on the master node.

   wget https://docs.projectcalico.org/v3.14/manifests/calico.yaml
   

3. Modify the calico.yaml file.

   # vim calico.yaml
   
   //Modify the following parameters:
   
   - name: IP_AUTODERECTION_METHOD
   Value: "can-reach=197.3.10.254"
   
   - name: CALICO_IPV4POOL_IPIP
   Value: "CrossSubnet"
   

   

   • If the default CNI of the pod is Flannel, add the following content to flannel.yaml:

     --iface=enp4s0
     

     

4. Create a pod after the modification.

   # kubectl apply -f calico.yaml
   

   • To delete the pod, run the following command:

     # kubectl delete -f calico.yaml
     

5. View the pod information.

   # kubectl get pod -A -o wide
   

Viewing the Information About the Master Node

Run the following command to view the detailed information about the node:

# kubectl get nodes -o wide

To reset the node, run the following command:

# kubeadm reset

GitLab Runner Deployment and Testing

Deploying GitLab

Copy the required YAML files to the /home directory and start the corresponding pod.

Note: GitLab YAML files of the required versions can be obtained from the official website.

# kubectl apply -f gitlab-redis.yaml
# kubectl apply -f gitlab-postgresql.yaml
# kubectl apply -f gitlab.yaml

Run the following command to check whether the GitLab pod has been set up:

# kubectl get pod -A -owide

Check whether you can log in to the GitLab web IDE. The URL is the IP address and port number.

The user name is root. To view the default password, log in to the container and view the password file.

# kubectl exec -it gitlab-lab -n default /bin/sh
# cat /etc/gitlab/initial_root_password

• The following figure shows the web IDE.

Using gitlab-runner.yaml to Start the Runner Container

Change the image name in the gitlab-runner.yaml file.

# vim gitlab-runner.yaml
# kubectl apply -f gitlab-runner.yaml
# kubectl get pod -A -o wide

! image

Creating a Container Project That Uses User Certificates for Authentication in GitLab

1. Click New project.

2. Select Create blank project.

3. Enter a name for the project.

4. Choose Settings > CI/CD > Runners > Expand.

5. Record the address and token for registering the Runner.

6. Import certificate files.

   Check and generate certificate files admin.crt, admin.key, and ca.crt on the master node.

   • View certificate information.

     # cat /etc/kubernetes/admin.conf
     

     

   • Generate the encrypted admin.crt.

     # echo "${client-certificate-data}" | base64 -d > admin.crt
     

   • Generate the encrypted admin.key.

     # echo "${client-key-data}" | base64 -d > admin.key
     

   • Obtain the CA certificate on the manager node.

     # cp /etc/kubernetes/pki/ca.crt ./
     

7. Import the three certificate files to the GitLab Runner container on the node where the Runner is running.

   Note: To import the certificate files, check the node where the GitLab Runner is running, copy the certificate files to the node, and run the isula cp command to import the certificate files.

   # isula cp admin.crt [Container ID]: Storage path
   # isula cp admin.key [Container ID]: Storage path
   # isula cp ca.crt [Container ID]: Storage path
   

   Note: The isula cp command can copy only one file at a time.

   

Registering the GitLab Runner

Perform registration in the GitLab Runner container. Currently, interactive registration is used. Obtain the registration information from GitLab. Choose GitLab > Group runners > Settings > CI/CD > Runners.

Upload the prepared gitlab-runner-helper image to the private image repository in advance, go to the GitLab Runner container, and modify the configuration file.

# cd /etc/gitlab-runner
# mkdir kubessl
# cp /home/admin.crt /etc/gitlab-runner/kubessl
# cp /home/ca.crt /etc/gitlab-runner/kubessl
# cp /home/admin.key /etc/gitlab-runner/kubessl
# vim /etc/gitlab-runner/config.toml

Adding the DNS Record of the GitLab Container to the Manager Node

1. View the IP address of the GitLab container.

   # kubectl get pods –Aowide
   

2. Add the IP address of the GitLab container to the Kubernetes DNS configuration file.

   # kubectl edit configmaps coredns -n kube-system
   

3. Restart the CoreDNS service.

   # kubectl scale deployment coredns -n kube-system --replicas=0
   # kubectl scale deployment coredns -n kube-system --replicas=2
   

GitLab Running Testing

Return to the GitLab web IDE and choose CI/CD > Editor > Create new CI/CD pipeline.

• Compile the YAML file as follows:

• Choose Pipelines and view the status.