Setting Up Kubernetes and iSulad
Unless otherwise specified, perform the following steps on both the master and node. This tutorial uses the master as an example.
Before You Start
Prepare NestOS-22.03-date.x86_64.iso and two hosts act as the master and node respectively.
Downloading the Components
Open the repo source file to add the Alibaba Cloud source of Kubernetes.
vi /etc/yum.repos.d/openEuler.repo
Add the following content:
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
Downloads the Kubernetes components and the components for synchronizing the system time.
rpm-ostree install kubelet kubeadm kubectl ntp ntpdate wget
Restart the system to use the components.
systemctl reboot
Select the latest version branch and enter the system.
Configuring the Environment
Change the Host Name of the Master
hostnamectl set-hostname k8s-master
sudo -i
Open the /etc/hosts file.
vi /etc/hosts
Add the IP addresses of the hosts.
192.168.237.133 k8s-master
192.168.237.135 k8s-node01
Synchronizing the System Time
ntpdate time.windows.com
systemctl enable ntpd
Disabling the swap Partition, Firewall, and SELinux
By default, the NestOS does not have the swap partition and the firewall is disabled. Run the following command to disable SELinux:
vi /etc/sysconfig/selinux
# Change the value of SELINUX to disabled.
Enabling Forwarding Mechanisms
Create a configuration file.
vi /etc/sysctl.d/k8s.conf
Add the following content:
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
Make the configuration take effect.
modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf
Configuring iSula
Check the OS image required by Kubernetes. Pay attention to the version number of the pause container.
kubeadm config images list
Modify the daemon.json configuration file.
vi /etc/isulad/daemon.json
## Description of the added items ##
Set registry-mirrors to "docker.io".
Set insecure-registries to "rnd-dockerhub.huawei.com".
Set pod-sandbox-image to "registry.aliyuncs.com/google_containers/pause:3.5". (The Alibaba Cloud source is used. The pause version is obtained in the previous step.)
Set network-plugin to "cni".
Set cni-bin-dir to "/opt/cni/bin".
Set cni-conf-dir to "/etc/cni/net.d".
The modified file is as follows:
{"group": "isula",
"default-runtime": "lcr",
"graph": "/var/lib/isulad",
"state": "/var/run/isulad",
"engine": "lcr",
"log-level": "ERROR",
"pidfile": "/var/run/isulad.pid",
"log-opts": {
"log-file-mode": "0600",
"log-path": "/var/lib/isulad",
"max-file": "1",
"max-size": "30KB"
},
"log-driver": "stdout",
"container-log": {
"driver": "json-file"
},
"hook-spec": "/etc/default/isulad/hooks/default.json",
"start-timeout": "2m",
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
],
"registry-mirrors": [
"docker.io"
],
"insecure-registries": [
"rnd-dockerhub.huawei.com"
],
"pod-sandbox-image": "registry.aliyuncs.com/google_containers/pause:3.5",
"native.umask": "secure",
"network-plugin": "cni",
"cni-bin-dir": "/opt/cni/bin",
"cni-conf-dir": "/etc/cni/net.d",
"image-layer-check": false,
"use-decrypted-key": true,
"insecure-skip-verify-enforce": false
}
Start the services.
systemctl restart isulad
systemctl enable isulad
systemctl enable kubelet
Perform the preceding steps on both the master and node.
Initializing the Master
Perform this step only on the master. Run the following command and wait for the host to pull the image. You can also manually pull the image before performing this step.
kubeadm init --kubernetes-version=1.22.2 --apiserver-advertise-
address=192.168.237.133 --cri-socket=/var/run/isulad.sock --image-repository
registry.aliyuncs.com/google_containers --service-cidr=10.10.0.0/16 --pod-
network-cidr=10.122.0.0/16
## Description of initialization parameters ##
kubernetes-version indicates the version to be installed.
apiserver-advertise-address indicates the IP address of the master.
cri-socket specifies the iSulad engine.
image-repository specifies that the image source is Alibaba Cloud. You do not need to modify the tag.
service-cidr specifies the IP address range allocated to the service.
pod-network-cidr specifies the IP address range allocated to the Pod network.
After the initialization is successful, copy the kubeadm join command that is output by kubeadm init for subsequent node joining.
kubeadm join 192.168.237.133:6443 --token j7kufw.yl1gte0v9qgxjzjw --discovery-
token-ca-cert-hash
sha256:73d337f5edd79dd4db997d98d329bd98020b712f8d7833c33a85d8fe44d0a4f5 --cri-
socket=/var/run/isulad.sock
Note: --cri-socket=/var/run/isulad.sock specifies that iSulad is used as the container engine.
View the downloaded image.
isula images
Configure the cluster based on the output of the initialization command.
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf
source /etc/profile
Check the health status.
kubectl get cs
The status of controller-manager and scheduler may be unhealthy. To rectify the fault, perform the following steps:
Edit the configuration file.
vi /etc/kubernetes/manifests/kube-controller-manager.yaml
Comment out the following content: --port=0 Modify hostpath: Change all /usr/libexec/kubernetes/kubelet-plugins/volume/exec to /opt/libexec/...
vi /etc/kubernetes/manifests/kube-scheduler.yaml
Comment out the following content: --port=0
After the modification is complete, check the health status again.
Configuring the Network Plugin
Configure the network plugin only on the master. However, you need to pull images on all hosts in advance. The commands for pulling images are as follows:
isula pull calico/node:v3.19.3
isula pull calico/cni:v3.19.3
isula pull calico/kube-controllers:v3.19.3
isula pull calico/pod2daemon-flexvol:v3.19.3
Perform the following steps only on the master.
Obtain the configuration file.
wget https://docs.projectcalico.org/v3.19/manifests/calico.yaml
Edit calico.yaml and change all /usr/libexec/... to /opt/libexec/.... Run the following command to install Calico:
kubectl apply -f calico.yaml
Run the kubectl get pod -n kube-system command to check whether Calico is successfully installed.
Run the kubectl get pod -n kube-system command to check whether all Pods are in therunning status.
Joining the Node to the Cluster
Run the following command on the node to join the node to the cluster:
kubeadm join 192.168.237.133:6443 --token j7kufw.yl1gte0v9qgxjzjw --discovery-
token-ca-cert-hash
sha256:73d337f5edd79dd4db997d98d329bd98020b712f8d7833c33a85d8fe44d0a4f5 --cri-
socket=/var/run/isulad.sock
Run the kubectl get node command to check whether the master and node statuses are ready.
If yes, Kubernetes is successfully deployed.
Using rpm-ostree
Installing Software Packages Using rpm-ostree
Install wget.
rpm-ostree install wget
Restart the system. During the startup, use the up and down arrow keys on the keyboard to enter system before or after the RPM package installation. ostree:0 indicates the version after the installation.
systemctl reboot
Check whether wget is successfully installed.
rpm -qa | grep wget
Manually Upgrading NestOS Using rpm-ostree
Run the following command in NestOS to view the current rpm-ostree status and version:
rpm-ostree status
Run the check command to check whether a new version is available.
rpm-ostree upgrade --check
Preview the differences between the versions.
rpm-ostree upgrade --preview
In the latest version, the nano package is imported. Run the following command to download the latest ostree and RPM data without performing the deployment.
rpm-ostree upgrade --download-only
Restart NestOS. After the restart, the old and new versions of the system are available. Enter the latest version.
rpm-ostree upgrade --reboot
Comparing NestOS Versions
Check the status. Ensure that two versions of ostree exist: LTS.20210927.dev.0 and LTS.20210928.dev.0.
rpm-ostree status
Compare the ostree versions based on commit IDs.
rpm-ostree db diff 55eed9bfc5ec fe2408e34148
Rolling Back the System
When a system upgrade is complete, the previous NestOS deployment is still stored on the disk. If the upgrade causes system problems, you can roll back to the previous deployment.
Temporary Rollback
To temporarily roll back to the previous OS deployment, hold down Shift during system startup. When the boot load menu is displayed, select the corresponding branch from the menu.
Permanent Rollback
To permanently roll back to the previous OS deployment, log in to the target node and run the rpm-ostree rollback command. This operation sets the previous OS deployment as the default deployment to boot into.
Run the following command to roll back to the system before the upgrade:
rpm-ostree rollback
Switching Versions
NestOS is rolled back to an older version. You can run the following command to switch the rpm-ostree version used by NestOS to a newer version.
rpm-ostree deploy -r 22.03.20220325.dev.0
After the restart, check whether NestOS uses the latest ostree version.
Using Zincati for Automatic Update
Zincati automatically updates NestOS. Zincati uses the Cincinnati backend to check whether a new version is available. If a new version is available, Zincati downloads it using rpm-ostree.
Currently, the Zincati automatic update service is disabled by default. You can modify the configuration file to set the automatic startup upon system startup for Zincati.
vi /etc/zincati/config.d/95-disable-on-dev.toml
Set updates.enabled to true. Create a configuration file to specify the address of the Cincinnati backend.
vi /etc/zincati/config.d/update-cincinnati.toml
Add the following content:
[cincinnati]
base_url="http://nestos.org.cn:8080"
Restart the Zincati service.
systemctl restart zincati.service
When a new version is available, Zincati automatically detects the new version. Check the rpm-ostree status. If the status is busy, the system is being upgraded.
After a period of time, NestOS automatically restarts. Log in to NestOS again and check the rpm-ostree status. If the status changes to idle and the current version is 20220325, rpm-ostree has been upgraded.
View the zincati service logs to check the upgrade process and system restart logs. In addition, the information "auto-updates logic enabled" in the logs indicates that the update is automatic.
Customizing NestOS
You can use the nestos-installer tool to customize the original NestOS ISO file and package the Ignition file to generate a customized NestOS ISO file. The customized NestOS ISO file can be used to automatically install NestOS after the system is started for easy installation.
Before customizing NestOS, make the following preparations:
- Downloading the NestOS ISO.
- Preparing a config.ign File.
Generating a Customized NestOS ISO File
Setting Parameter Variables
export COREOS_ISO_ORIGIN_FILE=nestos-22.03.20220324.x86_64.iso
export COREOS_ISO_CUSTOMIZED_FILE=my-nestos.iso
export IGN_FILE=config.ign
Checking the ISO File
Ensure that the original NestOS ISO file does not contain the Ignition configuration.
$ nestos-installer iso ignition show $COREOS_ISO_ORIGIN_FILE
Error: No embedded Ignition config.
Generating a Customized NestOS ISO File
Package the Ignition file into the original NestOS ISO file to generate a customized NestOS ISO file.
nestos-installer iso ignition embed $COREOS_ISO_ORIGIN_FILE --ignition-file $IGN_FILE $COREOS_ISO_ORIGIN_FILE --output $COREOS_ISO_CUSTOMIZED_FILE
Checking the ISO File
Ensure that the customized NestOS ISO file contains the Ignition configuration.
nestos-installer iso ignition show $COREOS_ISO_CUSTOMIZED_FILE
The previous command displays the Ignition configuration.
Installing the Customized NestOS ISO File
The customized NestOS ISO file can be used to directly boot the installation. NestOS is automatically installed based on the Ignition configuration. After the installation is complete, you can use nest/password to log in to NestOS on the VM console.
