LTS

    Innovation Version

      Setting Up Kubernetes and iSulad

      Unless otherwise specified, perform the following steps on both the master and node. This tutorial uses the master as an example.

      Before You Start

      Prepare NestOS-22.03-date.x86_64.iso and two hosts act as the master and node respectively.

      Downloading the Components

      Open the repo source file to add the Alibaba Cloud source of Kubernetes.

      vi /etc/yum.repos.d/openEuler.repo
      

      Add the following content:

      [kubernetes]
      name=Kubernetes
      baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
      enabled=1
      gpgcheck=1
      repo_gpgcheck=1
      gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
      

      Downloads the Kubernetes components and the components for synchronizing the system time.

      rpm-ostree install kubelet kubeadm kubectl ntp ntpdate wget
      

      Restart the system to use the components.

      systemctl reboot
      

      Select the latest version branch and enter the system.

      Configuring the Environment

      Change the Host Name of the Master

      hostnamectl set-hostname k8s-master
      sudo -i
      

      Open the /etc/hosts file.

      vi /etc/hosts
      

      Add the IP addresses of the hosts.

      192.168.237.133 k8s-master
      192.168.237.135 k8s-node01
      

      Synchronizing the System Time

      ntpdate time.windows.com
      systemctl enable ntpd
      

      Disabling the swap Partition, Firewall, and SELinux

      By default, the NestOS does not have the swap partition and the firewall is disabled. Run the following command to disable SELinux:

      vi /etc/sysconfig/selinux
      # Change the value of SELINUX to disabled.
      

      Enabling Forwarding Mechanisms

      Create a configuration file.

      vi /etc/sysctl.d/k8s.conf
      

      Add the following content:

      net.bridge.bridge-nf-call-iptables=1
      net.bridge.bridge-nf-call-ip6tables=1
      net.ipv4.ip_forward=1
      

      Make the configuration take effect.

      modprobe br_netfilter
      sysctl -p /etc/sysctl.d/k8s.conf
      

      Configuring iSula

      Check the OS image required by Kubernetes. Pay attention to the version number of the pause container.

      kubeadm config images list
      

      Modify the daemon.json configuration file.

      vi /etc/isulad/daemon.json
      
      ## Description of the added items ##
      Set registry-mirrors to "docker.io".
      Set insecure-registries to "rnd-dockerhub.huawei.com".
      Set pod-sandbox-image to "registry.aliyuncs.com/google_containers/pause:3.5". (The Alibaba Cloud source is used. The pause version is obtained in the previous step.)
      Set network-plugin to "cni".
      Set cni-bin-dir to "/opt/cni/bin".
      Set cni-conf-dir to "/etc/cni/net.d".
      

      The modified file is as follows:

      {"group": "isula",
      "default-runtime": "runc",
      "graph": "/var/lib/isulad",
      "state": "/var/run/isulad",
      "engine": "lcr",
      "log-level": "ERROR",
      "pidfile": "/var/run/isulad.pid",
      "log-opts": {
      "log-file-mode": "0600",
      "log-path": "/var/lib/isulad",
      "max-file": "1",
      "max-size": "30KB"
      },
      "log-driver": "stdout",
      "container-log": {
      "driver": "json-file"
      },
      "hook-spec": "/etc/default/isulad/hooks/default.json",
      "start-timeout": "2m",
      "storage-driver": "overlay2",
      "storage-opts": [
      "overlay2.override_kernel_check=true"
      ],
      "registry-mirrors": [
      "docker.io"
      ],
      "insecure-registries": [
      "rnd-dockerhub.huawei.com"
      ],
      "pod-sandbox-image": "registry.aliyuncs.com/google_containers/pause:3.5",
      "native.umask": "secure",
      "network-plugin": "cni",
      "cni-bin-dir": "/opt/cni/bin",
      "cni-conf-dir": "/etc/cni/net.d",
      "image-layer-check": false,
      "use-decrypted-key": true,
      "insecure-skip-verify-enforce": false
      }
      

      Start the services.

      systemctl restart isulad
      systemctl enable isulad
      systemctl enable kubelet
      

      Perform the preceding steps on both the master and node.

      Initializing the Master

      Perform this step only on the master. Run the following command and wait for the host to pull the image. You can also manually pull the image before performing this step.

      kubeadm init --kubernetes-version=1.22.2 --apiserver-advertise-
      address=192.168.237.133 --cri-socket=/var/run/isulad.sock --image-repository
      registry.aliyuncs.com/google_containers --service-cidr=10.10.0.0/16 --pod-
      network-cidr=10.122.0.0/16
      
      ## Description of initialization parameters ##
      kubernetes-version indicates the version to be installed.
      apiserver-advertise-address indicates the IP address of the master.
      cri-socket specifies the iSulad engine.
      image-repository specifies that the image source is Alibaba Cloud. You do not need to modify the tag.
      service-cidr specifies the IP address range allocated to the service.
      pod-network-cidr specifies the IP address range allocated to the Pod network.
      

      After the initialization is successful, copy the kubeadm join command that is output by kubeadm init for subsequent node joining.

      kubeadm join 192.168.237.133:6443 --token j7kufw.yl1gte0v9qgxjzjw --discovery-
      token-ca-cert-hash
      sha256:73d337f5edd79dd4db997d98d329bd98020b712f8d7833c33a85d8fe44d0a4f5 --cri-
      socket=/var/run/isulad.sock
      

      Note: --cri-socket=/var/run/isulad.sock specifies that iSulad is used as the container engine.
      View the downloaded image.

      isula images
      

      Configure the cluster based on the output of the initialization command.

      mkdir -p $HOME/.kube
      cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      chown $(id -u):$(id -g) $HOME/.kube/config
      export KUBECONFIG=/etc/kubernetes/admin.conf
      source /etc/profile
      

      Check the health status.

      kubectl get cs
      

      The status of controller-manager and scheduler may be unhealthy. To rectify the fault, perform the following steps:
      Edit the configuration file.

      vi /etc/kubernetes/manifests/kube-controller-manager.yaml
      

      Comment out the following content: --port=0 Modify hostpath: Change all /usr/libexec/kubernetes/kubelet-plugins/volume/exec to /opt/libexec/...

      vi /etc/kubernetes/manifests/kube-scheduler.yaml
      

      Comment out the following content: --port=0

      After the modification is complete, check the health status again.

      Configuring the Network Plugin

      Configure the network plugin only on the master. However, you need to pull images on all hosts in advance. The commands for pulling images are as follows:

      isula pull calico/node:v3.19.3
      isula pull calico/cni:v3.19.3
      isula pull calico/kube-controllers:v3.19.3
      isula pull calico/pod2daemon-flexvol:v3.19.3
      

      Perform the following steps only on the master.
      Obtain the configuration file.

      wget https://docs.projectcalico.org/v3.19/manifests/calico.yaml
      

      Edit calico.yaml and change all /usr/libexec/... to /opt/libexec/.... Run the following command to install Calico:

      kubectl apply -f calico.yaml
      

      Run the kubectl get pod -n kube-system command to check whether Calico is successfully installed. Run the kubectl get pod -n kube-system command to check whether all Pods are in therunning status.

      Joining the Node to the Cluster

      Run the following command on the node to join the node to the cluster:

      kubeadm join 192.168.237.133:6443 --token j7kufw.yl1gte0v9qgxjzjw --discovery-
      token-ca-cert-hash
      sha256:73d337f5edd79dd4db997d98d329bd98020b712f8d7833c33a85d8fe44d0a4f5 --cri-
      socket=/var/run/isulad.sock
      

      Run the kubectl get node command to check whether the master and node statuses are ready.

      If yes, Kubernetes is successfully deployed.

      Using rpm-ostree

      Installing Software Packages Using rpm-ostree

      Install wget.

      rpm-ostree install wget
      

      Restart the system. During the startup, use the up and down arrow keys on the keyboard to enter system before or after the RPM package installation. ostree:0 indicates the version after the installation.

      systemctl reboot
      

      Check whether wget is successfully installed.

      rpm -qa | grep wget
      

      Manually Upgrading NestOS Using rpm-ostree

      Run the following command in NestOS to view the current rpm-ostree status and version:

      rpm-ostree status
      

      Run the check command to check whether a new version is available.

      rpm-ostree upgrade --check
      

      Preview the differences between the versions.

      rpm-ostree upgrade --preview
      

      In the latest version, the nano package is imported. Run the following command to download the latest ostree and RPM data without performing the deployment.

      rpm-ostree upgrade --download-only
      

      Restart NestOS. After the restart, the old and new versions of the system are available. Enter the latest version.

      rpm-ostree upgrade --reboot
      

      Comparing NestOS Versions

      Check the status. Ensure that two versions of ostree exist: LTS.20210927.dev.0 and LTS.20210928.dev.0.

      rpm-ostree status
      

      Compare the ostree versions based on commit IDs.

      rpm-ostree db diff 55eed9bfc5ec fe2408e34148
      

      Rolling Back the System

      When a system upgrade is complete, the previous NestOS deployment is still stored on the disk. If the upgrade causes system problems, you can roll back to the previous deployment.

      Temporary Rollback

      To temporarily roll back to the previous OS deployment, hold down Shift during system startup. When the boot load menu is displayed, select the corresponding branch from the menu.

      Permanent Rollback

      To permanently roll back to the previous OS deployment, log in to the target node and run the rpm-ostree rollback command. This operation sets the previous OS deployment as the default deployment to boot into. Run the following command to roll back to the system before the upgrade:

      rpm-ostree rollback
      

      Switching Versions

      NestOS is rolled back to an older version. You can run the following command to switch the rpm-ostree version used by NestOS to a newer version.

      rpm-ostree deploy -r 22.03.20220325.dev.0
      

      After the restart, check whether NestOS uses the latest ostree version.

      Using Zincati for Automatic Update

      Zincati automatically updates NestOS. Zincati uses the Cincinnati backend to check whether a new version is available. If a new version is available, Zincati downloads it using rpm-ostree.

      Currently, the Zincati automatic update service is disabled by default. You can modify the configuration file to set the automatic startup upon system startup for Zincati.

      vi /etc/zincati/config.d/95-disable-on-dev.toml
      

      Set updates.enabled to true. Create a configuration file to specify the address of the Cincinnati backend.

      vi /etc/zincati/config.d/update-cincinnati.toml
      

      Add the following content:

      [cincinnati]
      base_url="http://nestos.org.cn:8080"
      

      Restart the Zincati service.

      systemctl restart zincati.service
      

      When a new version is available, Zincati automatically detects the new version. Check the rpm-ostree status. If the status is busy, the system is being upgraded.

      After a period of time, NestOS automatically restarts. Log in to NestOS again and check the rpm-ostree status. If the status changes to idle and the current version is 20220325, rpm-ostree has been upgraded.

      View the zincati service logs to check the upgrade process and system restart logs. In addition, the information "auto-updates logic enabled" in the logs indicates that the update is automatic.

      Customizing NestOS

      You can use the nestos-installer tool to customize the original NestOS ISO file and package the Ignition file to generate a customized NestOS ISO file. The customized NestOS ISO file can be used to automatically install NestOS after the system is started for easy installation.

      Before customizing NestOS, make the following preparations:

      • Downloading the NestOS ISO.
      • Preparing a config.ign File.

      Generating a Customized NestOS ISO File

      Setting Parameter Variables

      export COREOS_ISO_ORIGIN_FILE=nestos-22.03.20220324.x86_64.iso
      export COREOS_ISO_CUSTOMIZED_FILE=my-nestos.iso
      export IGN_FILE=config.ign
      

      Checking the ISO File

      Ensure that the original NestOS ISO file does not contain the Ignition configuration.

      $ nestos-installer iso ignition show $COREOS_ISO_ORIGIN_FILE 
      
      Error: No embedded Ignition config.
      

      Generating a Customized NestOS ISO File

      Package the Ignition file into the original NestOS ISO file to generate a customized NestOS ISO file.

      nestos-installer iso ignition embed $COREOS_ISO_ORIGIN_FILE --ignition-file $IGN_FILE $COREOS_ISO_ORIGIN_FILE --output $COREOS_ISO_CUSTOMIZED_FILE
      

      Checking the ISO File

      Ensure that the customized NestOS ISO file contains the Ignition configuration.

      nestos-installer iso ignition show $COREOS_ISO_CUSTOMIZED_FILE
      

      The previous command displays the Ignition configuration.

      Installing the Customized NestOS ISO File

      The customized NestOS ISO file can be used to directly boot the installation. NestOS is automatically installed based on the Ignition configuration. After the installation is complete, you can use nest/password to log in to NestOS on the VM console.

      Bug Catching

      Buggy Content

      Bug Description

      Submit As Issue

      It's a little complicated....

      I'd like to ask someone.

      PR

      Just a small problem.

      I can fix it online!

      Bug Type
      Specifications and Common Mistakes

      ● Misspellings or punctuation mistakes;

      ● Incorrect links, empty cells, or wrong formats;

      ● Chinese characters in English context;

      ● Minor inconsistencies between the UI and descriptions;

      ● Low writing fluency that does not affect understanding;

      ● Incorrect version numbers, including software package names and version numbers on the UI.

      Usability

      ● Incorrect or missing key steps;

      ● Missing prerequisites or precautions;

      ● Ambiguous figures, tables, or texts;

      ● Unclear logic, such as missing classifications, items, and steps.

      Correctness

      ● Technical principles, function descriptions, or specifications inconsistent with those of the software;

      ● Incorrect schematic or architecture diagrams;

      ● Incorrect commands or command parameters;

      ● Incorrect code;

      ● Commands inconsistent with the functions;

      ● Wrong screenshots.

      Risk Warnings

      ● Lack of risk warnings for operations that may damage the system or important data.

      Content Compliance

      ● Contents that may violate applicable laws and regulations or geo-cultural context-sensitive words and expressions;

      ● Copyright infringement.

      How satisfied are you with this document

      Not satisfied at all
      Very satisfied
      Submit
      Click to create an issue. An issue template will be automatically generated based on your feedback.
      Bug Catching
      编组 3备份