Setting Up Kubernetes and iSulad

Unless otherwise specified, perform the following steps on both the master and node. This tutorial uses the master as an example.

Before You Start

Prepare NestOS-22.03-date.x86_64.iso and two hosts act as the master and node respectively.

Downloading the Components

Open the repo source file to add the Alibaba Cloud source of Kubernetes.

vi /etc/yum.repos.d/openEuler.repo

Add the following content:

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

Downloads the Kubernetes components and the components for synchronizing the system time.

rpm-ostree install kubelet kubeadm kubectl ntp ntpdate wget

Restart the system to use the components.

systemctl reboot

Select the latest version branch and enter the system.

Configuring the Environment

Change the Host Name of the Master

hostnamectl set-hostname k8s-master
sudo -i

Open the /etc/hosts file.

vi /etc/hosts

Add the IP addresses of the hosts.

192.168.237.133 k8s-master
192.168.237.135 k8s-node01

Synchronizing the System Time

ntpdate time.windows.com
systemctl enable ntpd

Disable the swap partition, firewall, and SELinux.

By default, the NestOS does not have the swap partition and the firewall is disabled. Run the following command to disable SELinux:

vi /etc/sysconfig/selinux
Change the value of SELINUX to disabled.

Enabling Forwarding Mechanisms

Create a configuration file.

vi /etc/sysctl.d/k8s.conf

Add the following content:

net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1

Make the configuration take effect.

modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf

Configuring iSula

Check the OS image required by Kubernetes. Pay attention to the version number of the pause container.

kubeadm config images list

Modify the daemon.json configuration file.

vi /etc/isulad/daemon.json

## Description of the added items ##
Set registry-mirrors to "docker.io".
Set insecure-registries to "rnd-dockerhub.huawei.com".
Set pod-sandbox-image to "registry.aliyuncs.com/google_containers/pause:3.5". (The Alibaba Cloud source is used. The pause version is obtained in the previous step.)
Set network-plugin to "cni".
Set cni-bin-dir to "/opt/cni/bin".
Set cni-conf-dir to "/etc/cni/net.d".

The modified file is as follows:

{"group": "isula",
"default-runtime": "lcr",
"graph": "/var/lib/isulad",
"state": "/var/run/isulad",
"engine": "lcr",
"log-level": "ERROR",
"pidfile": "/var/run/isulad.pid",
"log-opts": {
"log-file-mode": "0600",
"log-path": "/var/lib/isulad",
"max-file": "1",
"max-size": "30KB"
},
"log-driver": "stdout",
"container-log": {
"driver": "json-file"
},
"hook-spec": "/etc/default/isulad/hooks/default.json",
"start-timeout": "2m",
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
],
"registry-mirrors": [
"docker.io"
],
"insecure-registries": [
"rnd-dockerhub.huawei.com"
],
"pod-sandbox-image": "registry.aliyuncs.com/google_containers/pause:3.5",
"native.umask": "secure",
"network-plugin": "cni",
"cni-bin-dir": "/opt/cni/bin",
"cni-conf-dir": "/etc/cni/net.d",
"image-layer-check": false,
"use-decrypted-key": true,
"insecure-skip-verify-enforce": false
}

Start the services.

systemctl restart isulad
systemctl enable isulad
systemctl enable kubelet

Perform the preceding steps on both the master and node.

Initializing the Master

Perform this step only on the master. Run the following command and wait for the host to pull the image. You can also manually pull the image before performing this step.

kubeadm init --kubernetes-version=1.22.2 --apiserver-advertise-
address=192.168.237.133 --cri-socket=/var/run/isulad.sock --image-repository
registry.aliyuncs.com/google_containers --service-cidr=10.10.0.0/16 --pod-
network-cidr=10.122.0.0/16

## Description of initialization parameters ##
kubernetes-version indicates the version to be installed.
apiserver-advertise-address indicates the IP address of the master.
cri-socket specifies the iSulad engine.
image-repository specifies that the image source is Alibaba Cloud. You do not need to modify the tag.
service-cidr specifies the IP address range allocated to the service.
pod-network-cidr specifies the IP address range allocated to the Pod network.

After the initialization is successful, copy the kubeadm join command that is output by kubeadm init for subsequent node joining.

kubeadm join 192.168.237.133:6443 --token j7kufw.yl1gte0v9qgxjzjw --discovery-
token-ca-cert-hash
sha256:73d337f5edd79dd4db997d98d329bd98020b712f8d7833c33a85d8fe44d0a4f5 --cri-
socket=/var/run/isulad.sock

Note: --cri-socket=/var/run/isulad.sock specifies that iSulad is used as the container engine.
View the downloaded image.

isula images

Configure the cluster based on the output of the initialization command.

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf
source /etc/profile

Check the health status.

kubectl get cs

The status of controller-manager and scheduler may be unhealthy. To rectify the fault, perform the following steps:
Edit the configuration file.

vi /etc/kubernetes/manifests/kube-controller-manager.yaml

Comment out the following content:
--port=0
Modify hostpath:
Change all /usr/libexec/kubernetes/kubelet-plugins/volume/exec to /opt/libexec/...

vi /etc/kubernetes/manifests/kube-scheduler.yaml

Comment out the following content:
--port=0

After the modification is complete, check the health status again.

Configuring the Network Plugin

Configure the network plugin only on the master. However, you need to pull images on all hosts in advance. The commands for pulling images are as follows:

isula pull calico/node:v3.19.3
isula pull calico/cni:v3.19.3
isula pull calico/kube-controllers:v3.19.3
isula pull calico/pod2daemon-flexvol:v3.19.3

Perform the following steps only on the master.
Obtain the configuration file.

wget https://docs.projectcalico.org/v3.19/manifests/calico.yaml

Edit calico.yaml and change all /usr/libexec/… to /opt/libexec/…. Run the following command to install Calico:

kubectl apply -f calico.yaml

Run the kubectl get pod -n kube-system command to check whether Calico is successfully installed. Run the kubectl get pod -n kube-system command to check whether all Pods are in the** running** status.

Joining the Node to the Cluster

Run the following command on the node to join the node to the cluster:

kubeadm join 192.168.237.133:6443 --token j7kufw.yl1gte0v9qgxjzjw --discovery-
token-ca-cert-hash
sha256:73d337f5edd79dd4db997d98d329bd98020b712f8d7833c33a85d8fe44d0a4f5 --cri-
socket=/var/run/isulad.sock

Run the kubectl get node command to check whether the master and node statuses are ready.

If yes, Kubernetes is successfully deployed.

Using rpm-ostree

Installing Software Packages Using rpm-ostree

Install wget.

rpm-ostree install wget

Restart the system. During the startup, use the up and down arrow keys on the keyboard to enter system before or after the RPM package installation. ostree:0 indicates the version after the installation.

systemctl reboot

Check whether wget is successfully installed.

rpm -qa | grep wget

Manually Upgrading NestOS Using rpm-ostree

Run the following command in NestOS to view the current rpm-ostree status and version:

rpm-ostree status

Run the check command to check whether a new version is available.

rpm-ostree upgrade --check

Preview the differences between the versions.

rpm-ostree upgrade --preview

In the latest version, the nano package is imported. Run the following command to download the latest ostree and RPM data without performing the deployment.

rpm-ostree upgrade --download-only

Restart NestOS. After the restart, the old and new versions of the system are available. Enter the latest version.

rpm-ostree upgrade --reboot

Comparing NestOS Versions

Check the status. Ensure that two versions of ostree exist: LTS.20210927.dev.0 and LTS.20210928.dev.0.

rpm-ostree status

Compare the ostree versions based on commit IDs.

rpm-ostree db diff 55eed9bfc5ec fe2408e34148

Rolling Back the System

When a system upgrade is complete, the previous NestOS deployment is still stored on the disk. If the upgrade causes system problems, you can roll back to the previous deployment.

Temporary Rollback

To temporarily roll back to the previous OS deployment, hold down Shift during system startup. When the boot load menu is displayed, select the corresponding branch from the menu.

Permanent Rollback

To permanently roll back to the previous OS deployment, log in to the target node and run the rpm-ostree rollback command. This operation sets the previous OS deployment as the default deployment to boot into. Run the following command to roll back to the system before the upgrade:

rpm-ostree rollback

Switching Versions

NestOS is rolled back to an older version. You can run the following command to switch the rpm-ostree version used by NestOS to a newer version.

rpm-ostree deploy -r 22.03.20220325.dev.0

After the restart, check whether NestOS uses the latest ostree version.

Using Zincati for Automatic Update

Zincati automatically updates NestOS. Zincati uses the Cincinnati backend to check whether a new version is available. If a new version is available, Zincati downloads it using rpm-ostree.

Currently, the Zincati automatic update service is disabled by default. You can modify the configuration file to set the automatic startup upon system startup for Zincati.

vi /etc/zincati/config.d/95-disable-on-dev.toml

Set updates.enabled to true. Create a configuration file to specify the address of the Cincinnati backend.

vi /etc/zincati/config.d/update-cincinnati.toml

Add the following content:

[cincinnati]
base_url="http://nestos.org.cn:8080"

Restart the Zincati service.

systemctl restart zincati.service

When a new version is available, Zincati automatically detects the new version. Check the rpm-ostree status. If the status is busy, the system is being upgraded.

After a period of time, NestOS automatically restarts. Log in to NestOS again and check the rpm-ostree status. If the status changes to idle and the current version is 20220325, rpm-ostree has been upgraded.

View the zincati service logs to check the upgrade process and system restart logs. In addition, the information “auto-updates logic enabled” in the logs indicates that the update is automatic.

Customizing NestOS

You can use the nestos-installer tool to customize the original NestOS ISO file and package the Ignition file to generate a customized NestOS ISO file. The customized NestOS ISO file can be used to automatically install NestOS after the system is started for easy installation.

Before customizing NestOS, make the following preparations:

• Downloading the NestOS ISO.
• Preparing a config.ign File.

Generating a Customized NestOS ISO File

Setting Parameter Variables

$ export COREOS_ISO_ORIGIN_FILE=nestos-22.03.20220324.x86_64.iso
$ export COREOS_ISO_CUSTOMIZED_FILE=my-nestos.iso
$ export IGN_FILE=config.ign

Checking the ISO File

Ensure that the original NestOS ISO file does not contain the Ignition configuration.

$ nestos-installer iso ignition show $COREOS_ISO_ORIGIN_FILE 

Error: No embedded Ignition config.

Generating a Customized NestOS ISO File

Package the Ignition file into the original NestOS ISO file to generate a customized NestOS ISO file.

$ nestos-installer iso ignition embed $COREOS_ISO_ORIGIN_FILE --ignition-file $IGN_FILE $COREOS_ISO_ORIGIN_FILE --output $COREOS_ISO_CUSTOMIZED_FILE

Checking the ISO File

Ensure that the customized NestOS ISO file contains the Ignition configuration.

$ nestos-installer iso ignition show $COREOS_ISO_CUSTOMIZED_FILE

The previous command displays the Ignition configuration.

Installing the Customized NestOS ISO File

The customized NestOS ISO file can be used to directly boot the installation. NestOS is automatically installed based on the Ignition configuration. After the installation is complete, you can use nest/password to log in to NestOS on the VM console.