长期支持版本

    社区创新版本

      部署KubeEdge

      介绍

      KubeEdge

      KubeEdge 是一个致力于解决边缘场景问题的开源系统,它将容器化应用程序编排和设备管理的能力扩展到边缘设备。基于 Kubernetes ,KubeEdge 为网络、应用程序部署以及云侧与边缘侧之间的元数据同步提供核心基础设施支持。KubeEdge 支持 MQTT,并允许开发人员编写自定义逻辑,在边缘上启用资源受限的设备通信。Kubeedge由云部分和边缘部分组成,目前均已开源。

      https://kubeedge.io/

      iSulad

      iSulad 是一个轻量级容器 runtime 守护程序,专为 IOT 和 Cloud 基础设施而设计,具有轻便、快速且不受硬件规格和体系结构限制的特性,可以被更广泛地应用在云、IoT、边缘计算等多个场景。

      https://gitee.com/openeuler/iSulad

      准备

      组件版本

      组件版本
      OSopenEuler 21.09
      Kubernetes1.20.2-4
      iSulad2.0.9-20210625.165022.git5a088d9c
      KubeEdgev1.8.0

      节点规划(示例)

      节点位置组件
      9.63.252.224云侧(cloud)k8s(master)、isulad、cloudcore
      9.63.252.227边缘侧(edge)isulad、edgecore

      环境准备

      以下设置需要在cloud和edge端均配置。

      # 关闭防火墙
      $ systemctl stop firewalld
      $ systemctl disable firewalld
      
      # 禁用selinux
      $ setenforce 0
      
      # 网络配置,开启相应的转发机制
      $ cat >> /etc/sysctl.d/k8s.conf <<EOF
      net.bridge.bridge-nf-call-ip6tables = 1
      net.bridge.bridge-nf-call-iptables = 1
      net.ipv4.ip_forward = 1
      vm.swappiness=0
      EOF
      
      # 生效规则
      $ modprobe br_netfilter
      $ sysctl -p /etc/sysctl.d/k8s.conf
      
      # 查看是否生效
      $ cat /proc/sys/net/bridge/bridge-nf-call-ip6tables
      1
      $ cat /proc/sys/net/bridge/bridge-nf-call-iptables
      1
      
      # 关闭系统swap
      $ swapoff -a
      
      # 设置hostname
      # 云侧
      $ hostnamectl set-hostname cloud.kubeedge
      # 边缘侧
      $ hostnamectl set-hostname edge.kubeedge
      
      # 配置hosts文件(示例),按照用户实际情况设置
      $ cat >> /etc/hosts << EOF
      9.63.252.224 cloud.kubeedge
      9.63.252.227 edge.kubeedge
      EOF
      
      # 同步时钟,选择可以访问的NTP服务器即可
      $ ntpdate cn.pool.ntp.org
      
      # 安装 cri-tools 网络工具
      $ wget --no-check-certificate https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.20.0/crictl-v1.20.0-linux-amd64.tar.gz
      $ tar zxvf crictl-v1.20.0-linux-amd64.tar.gz -C /usr/local/bin
      
      # 安装 cni 网络插件
      $ wget --no-check-certificate https://github.com/containernetworking/plugins/releases/download/v0.9.0/cni-plugins-linux-amd64-v0.9.0.tgz
      $ mkdir -p /opt/cni/bin
      $ tar -zxvf cni-plugins-linux-amd64-v0.9.0.tgz -C /opt/cni/bin
      

      配置iSulad

      以下设置需要在cloud和edge端均配置。

      # 配置iSulad(只列出修改项)
      $ cat /etc/isulad/daemon.json
      {
              "registry-mirrors": [
                      "docker.io"
              ],
              "insecure-registries": [
                      "k8s.gcr.io",
                      "quay.io",
                      "hub.oepkgs.net"
              ],
              "pod-sandbox-image": "k8s.gcr.io/pause:3.2",
              "network-plugin": "cni",
              "cni-bin-dir": "/opt/cni/bin",
              "cni-conf-dir": "/etc/cni/net.d",
      }
      
      # 如果不能直接访问外网,则需要配置proxy,否则不需要
      $ cat /usr/lib/systemd/system/isulad.service
      [Service]
      Type=notify
      Environment="HTTP_PROXY=http://..."
      Environment="HTTPS_PROXY=http://..."
      
      # 重启iSulad并设置为开机自启
      $ systemctl daemon-reload && systemctl restart isulad
      

      部署k8s组件

      k8s组件只需要在云侧安装部署。

      # 安装k8s工具
      $ yum install kubernetes-master kubernetes-kubeadm kubernetes-client kubernetes-kubelet
      # 开机启动kubelet
      $ systemctl enable kubelet --now
      
      # 注意,init之前需要取消系统环境中的proxy
      $ unset `env | grep -iE "tps?_proxy" | cut -d= -f1`
      $ env | grep proxy
      
      # 使用kubeadm init
      $ kubeadm init --kubernetes-version v1.20.2 --pod-network-cidr=10.244.0.0/16 --upload-certs --cri-socket=/var/run/isulad.sock
      # 默认k8s组件镜像是gcr.k8s.io,可以使用--image-repository=xxx 来使用自定义镜像仓库地址(测试自己的k8s镜像)
      
      # 注意这里的pod-network-cidr网段不能和宿主机的网段重复,否则网络不通
      # 先init再配置网络
      Your Kubernetes control-plane has initialized successfully!
      
      To start using your cluster, you need to run the following as a regular user:
      
        mkdir -p $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config
      
      You should now deploy a pod network to the cluster.
      Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
        https://kubernetes.io/docs/concepts/cluster-administration/addons/
      
      Then you can join any number of worker nodes by running the following on each as root:
      ...
      
      # 根据提示执行
      mkdir -p $HOME/.kube
      cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      chown $(id -u):$(id -g) $HOME/.kube/config
      # 这个命令复制了admin.conf(kubeadm帮我们自动初始化好的kubectl配置文件)
      # 这里包含了认证信息等相关信息的非常重要的一些配置。
      
      # 如果init出现问题可以重置
      $ kubeadm reset
      
      # 如果出现 Unable to read config path "/etc/kubernetes/manifests"
      $ mkdir -p /etc/kubernetes/manifests
      

      配置网络

      calico网络插件在edge节点无法运行,所以这里使用flannel代替,已有用户在KubeEdge社区提交issue

      因为云侧和边缘侧为不同的网络环境,需要配置不同的亲和性,所以这里需要两份flannel配置文件。

      # 下载flannel网络插件
      $ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
      
      # 准备云侧网络配置
      $ cp kube-flannel.yml kube-flannel-cloud.yml
      
      # 修改云侧网络配置
      $ patch kube-flannel-cloud.yml - <<EOF
      index c7edaef..f3846b9 100644
      --- a/kube-flannel.yml
      +++ b/kube-flannel.yml
      @@ -134,7 +134,7 @@ data:
       apiVersion: apps/v1
       kind: DaemonSet
       metadata:
      -  name: kube-flannel-ds
      +  name: kube-flannel-cloud-ds
         namespace: kube-system
         labels:
           tier: node
      @@ -158,6 +158,8 @@ spec:
                       operator: In
                       values:
                       - linux
      +              - key: node-role.kubernetes.io/agent
      +                operator: DoesNotExist
             hostNetwork: true
             priorityClassName: system-node-critical
             tolerations:
      EOF
      
      # 准备边缘侧网络配置
      $ cp kube-flannel.yml kube-flannel-edge.yml
      
      # 修改边缘侧网络配置
      $ patch kube-flannel-edge.yml - <<EOF
      index c7edaef..66a5b5b 100644
      --- a/kube-flannel.yml
      +++ b/kube-flannel.yml
      @@ -134,7 +134,7 @@ data:
       apiVersion: apps/v1
       kind: DaemonSet
       metadata:
      -  name: kube-flannel-ds
      +  name: kube-flannel-edge-ds
         namespace: kube-system
         labels:
           tier: node
      @@ -158,6 +158,8 @@ spec:
                       operator: In
                       values:
                       - linux
      +              - key: node-role.kubernetes.io/agent
      +                operator: Exists
             hostNetwork: true
             priorityClassName: system-node-critical
             tolerations:
      @@ -186,6 +188,7 @@ spec:
               args:
               - --ip-masq
               - --kube-subnet-mgr
      +        - --kube-api-url=http://127.0.0.1:10550
               resources:
                 requests:
                   cpu: "100m"
      EOF
      
      # 这里的--kube-api-url为边缘侧edgecore监听地址
      
      # 配置flannel网络插件
      $ kubectl apply -f kube-flannel-cloud.yml
      $ kubectl apply -f kube-flannel-edge.yml
      
      # 查看节点状态
      $ kubectl get node -A
      NAME             STATUS   ROLES                  AGE     VERSION
      cloud.kubeedge   Ready    control-plane,master   4h11m   v1.20.2
      

      使用keadm部署

      如果使用keadm进行集群部署,则只需要在云侧和边缘侧均安装kubeedge-keadmrpm包即可。

      $ yum install kubeedge-keadm
      

      部署云侧

      初始化集群

      # --advertise-address填写云侧IP
      $ keadm init --advertise-address="9.63.252.224" --kubeedge-version=1.8.0
      Kubernetes version verification passed, KubeEdge installation will start...
      W0829 10:41:56.541877  420383 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
      W0829 10:41:57.253214  420383 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
      W0829 10:41:59.778672  420383 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
      W0829 10:42:00.488838  420383 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
      kubeedge-v1.8.0-linux-amd64.tar.gz checksum:
      checksum_kubeedge-v1.8.0-linux-amd64.tar.gz.txt content:
      [Run as service] start to download service file for cloudcore
      [Run as service] success to download service file for cloudcore
      kubeedge-v1.8.0-linux-amd64/
      kubeedge-v1.8.0-linux-amd64/edge/
      kubeedge-v1.8.0-linux-amd64/edge/edgecore
      kubeedge-v1.8.0-linux-amd64/cloud/
      kubeedge-v1.8.0-linux-amd64/cloud/csidriver/
      kubeedge-v1.8.0-linux-amd64/cloud/csidriver/csidriver
      kubeedge-v1.8.0-linux-amd64/cloud/admission/
      kubeedge-v1.8.0-linux-amd64/cloud/admission/admission
      kubeedge-v1.8.0-linux-amd64/cloud/cloudcore/
      kubeedge-v1.8.0-linux-amd64/cloud/cloudcore/cloudcore
      kubeedge-v1.8.0-linux-amd64/version
      
      KubeEdge cloudcore is running, For logs visit:  /var/log/kubeedge/cloudcore.log
      CloudCore started
      

      此时cloudcore正在运行,但是并未使用systemd管理,且没有开启dynamiccontroller(对应edgecore中的list watch功能),需要修改配置。

      修改配置

      # 修改/etc/kubeedge/config/cloudcore.yaml
      # 开启dynamic controller
      $ cd /etc/kubeedge/config
      $ patch cloudcore.yaml - <<EOF
      index 28235a9..313375c 100644
      --- a/cloudcore.yaml
      +++ b/cloudcore.yaml
      @@ -1,7 +1,3 @@
       apiVersion: cloudcore.config.kubeedge.io/v1alpha1
       commonConfig:
         tunnelPort: 10350
      @@ -67,7 +63,7 @@ modules:
           load:
             updateDeviceStatusWorkers: 1
         dynamicController:
      -    enable: false
      +    enable: true	# 开启dynamicController以支持edgecore的listwatch功能
         edgeController:
           buffer:
             configMapEvent: 1
      @@ -119,5 +115,3 @@ modules:
           restTimeout: 60
         syncController:
           enable: true
      EOF
      
      # 云侧cloudcore可以通过systemd管理
      # 拷贝cloudcore.service到/usr/lib/systemd/system
      $ cp /etc/kubeedge/cloudcore.service /usr/lib/systemd/system
      
      # 杀掉当前cloudcore进程后重启
      $ pkill cloudcore
      $ systemctl restart cloudcore
      
      # 查看cloudcore是否运行
      $ systemctl status cloudcore
      ● cloudcore.service
           Loaded: loaded (/usr/lib/systemd/system/cloudcore.service disabled; vendor preset: disabled)
           Active: active (running) since Sun 2021-08-29 10:50:14 CST; 4s ago
         Main PID: 424578 (cloudcore)
            Tasks: 36 (limit: 202272)
           Memory: 44.2M
              CPU: 112ms
           CGroup: /system.slice/cloudcore.service
                   └─424578 /usr/local/bin/cloudcore
      
      Aug 29 10:50:15 cloud.kubeedge cloudcore[424578]: I0829 10:50:15.845792  424578 upstream.go:121] start upstream controller
      Aug 29 10:50:15 cloud.kubeedge cloudcore[424578]: I0829 10:50:15.846586  424578 downstream.go:870] Start downstream devicecontroller
      Aug 29 10:50:15 cloud.kubeedge cloudcore[424578]: I0829 10:50:15.849475  424578 downstream.go:566] start downstream controller
      Aug 29 10:50:15 cloud.kubeedge cloudcore[424578]: I0829 10:50:15.946110  424578 server.go:243] Ca and CaKey don't exist in local directory, and will read from the secret
      Aug 29 10:50:15 cloud.kubeedge cloudcore[424578]: I0829 10:50:15.951806  424578 server.go:288] CloudCoreCert and key don't exist in local directory, and will read from the >
      Aug 29 10:50:15 cloud.kubeedge cloudcore[424578]: I0829 10:50:15.959661  424578 signcerts.go:100] Succeed to creating token
      Aug 29 10:50:15 cloud.kubeedge cloudcore[424578]: I0829 10:50:15.959716  424578 server.go:44] start unix domain socket server
      Aug 29 10:50:15 cloud.kubeedge cloudcore[424578]: I0829 10:50:15.959973  424578 uds.go:71] listening on: //var/lib/kubeedge/kubeedge.sock
      Aug 29 10:50:15 cloud.kubeedge cloudcore[424578]: I0829 10:50:15.966693  424578 server.go:64] Starting cloudhub websocket server
      Aug 29 10:50:17 cloud.kubeedge cloudcore[424578]: I0829 10:50:17.847150  424578 upstream.go:63] Start upstream devicecontroller
      

      至此,云侧的cloudcore已部署完成,接下来部署边缘侧edgecore 。

      纳管边缘节点

      在边缘侧使用 keadm join 命令纳管边缘节点。

      修改iSulad配置

      文件位置:/etc/isulad/daemon.json

      {
          # 设置"pod-sandbox-image"
          "pod-sandbox-image": "kubeedge/pause:3.1",
          # KubeEdge边缘侧edgecore监听端口`10350`与iSulad websocket端口冲突,边缘侧无法启动edgecore。解决办法:修改iSulad配置(`/etc/isulad/daemon.json`)中的`websocket-server-listening-port`的字段为`10351`:
          "websocket-server-listening-port": 10351,
      }
      

      修改完配置文件之后,使用 systemctl restart isulad 重启iSulad。

      加入云侧

      # 在云侧获取token
      $ keadm gettoken
      28c25d3b137593f5bbfb776cf5b19866ab9727cab9e97964dd503f87cd52cbde.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MzAyOTE4MTV9.aGUyCi9gdysVtMu0DQzrD5TcV_DcXob647YeqcOxKDA
      
      # 在边缘侧使用 keadm join 加入云侧
      # --cloudcore-ipport是必须要加入的参数,10000是cloudcore默认端口
      $ keadm join --cloudcore-ipport=9.63.252.224:10000 --kubeedge-version=1.8.0 --token=28c25d3b137593f5bbfb776cf5b19866ab9727cab9e97964dd503f87cd52cbde.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MzAyOTE4MTV9.aGUyCi9gdysVtMu0DQzrD5TcV_DcXob647YeqcOxKDA
      Host has mosquit+ already installed and running. Hence skipping the installation steps !!!
      kubeedge-v1.8.0-linux-amd64.tar.gz checksum:
      checksum_kubeedge-v1.8.0-linux-amd64.tar.gz.txt content:
      [Run as service] start to download service file for edgecore
      [Run as service] success to download service file for edgecore
      kubeedge-v1.8.0-linux-amd64/
      kubeedge-v1.8.0-linux-amd64/edge/
      kubeedge-v1.8.0-linux-amd64/edge/edgecore
      kubeedge-v1.8.0-linux-amd64/cloud/
      kubeedge-v1.8.0-linux-amd64/cloud/csidriver/
      kubeedge-v1.8.0-linux-amd64/cloud/csidriver/csidriver
      kubeedge-v1.8.0-linux-amd64/cloud/admission/
      kubeedge-v1.8.0-linux-amd64/cloud/admission/admission
      kubeedge-v1.8.0-linux-amd64/cloud/cloudcore/
      kubeedge-v1.8.0-linux-amd64/cloud/cloudcore/cloudcore
      kubeedge-v1.8.0-linux-amd64/version
      
      KubeEdge edgecore is running, For logs visit: journalctl -u edgecore.service -b
      

      此时,edgecore并未部署成功,因为默认配置中使用的是docker,我们需要修改其配置文件用于对接iSulad。

      修改配置

      # 边缘侧edgecore可以通过systemd管理
      # 拷贝edgecore.service到/usr/lib/systemd/system
      $ cp /etc/kubeedge/edgecore.service /usr/lib/systemd/system
      
      # 修改edgecore配置
      $ cd /etc/kubeedge/config
      $ patch edgecore.yaml - <<EOF
      index 165e24b..efbfd49 100644
      --- a/edgecore.yaml
      +++ b/edgecore.yaml
      @@ -32,7 +32,7 @@ modules:
             server: 9.63.252.224:10000
             writeDeadline: 15
      @@ -73,10 +73,10 @@ modules:
           podSandboxImage: kubeedge/pause:3.1
           registerNode: true
           registerNodeNamespace: default
      -    remoteImageEndpoint: unix:///var/run/dockershim.sock
      -    remoteRuntimeEndpoint: unix:///var/run/dockershim.sock
      +    remoteImageEndpoint: unix:///var/run/isulad.sock
      +    remoteRuntimeEndpoint: unix:///var/run/isulad.sock
           runtimeRequestTimeout: 2
      -    runtimeType: docker
      +    runtimeType: remote	# iSulad类型为remote
           volumeStatsAggPeriod: 60000000000
         eventBus:
           enable: true
      @@ -97,7 +97,7 @@ modules:
           enable: true
           metaServer:
             debug: false
      -      enable: false
      +      enable: true		# 开启listwatch
           podStatusSyncInterval: 60
           remoteQueryTimeout: 60
         serviceBus:
      EOF
      
      # 杀掉当前edgecore进程
      $ pkill edgecore
      
      # 重启edgecore
      $ systemctl restart edgecore
      

      检查边缘节点是否纳管成功

      # 回到云侧,发现已经有了边缘节点
      $ kubectl get node -A
      NAME             STATUS   ROLES                  AGE     VERSION
      cloud.kubeedge   Ready    control-plane,master   19h     v1.20.2
      edge.kubeedge    Ready    agent,edge             5m16s   v1.19.3-kubeedge-v1.8.0
      

      至此,使用keadm部署KubeEdge集群已经完成,接下来我们测试一下从云侧下发任务到边缘侧。

      部署应用

      部署nginx

      # KubeEdge提供了一个nginx模板,我们可以直接使用
      $ kubectl apply -f https://github.com/kubeedge/kubeedge/raw/master/build/deployment.yaml
      deployment.apps/nginx-deployment created
      
      # 查看是否部署到了边缘侧
      $ kubectl get pod -A -owide | grep nginx
      default       nginx-deployment-77f96fbb65-fnp7n        1/1     Running   0          37s   10.244.2.4     edge.kubeedge    <none>           <none>
      
      # 可以看到,已经成功部署到了edge节点
      

      测试功能

      # 测试功能是否正常
      # 进入边缘侧节点,curl nginx的IP:10.244.2.4
      $ curl 10.244.2.4:80
      <!DOCTYPE html>
      <html>
      <head>
      <title>Welcome to nginx!</title>
      <style>
          body {
              width: 35em;
              margin: 0 auto;
              font-family: Tahoma, Verdana, Arial, sans-serif;
          }
      </style>
      </head>
      <body>
      <h1>Welcome to nginx!</h1>
      <p>If you see this page, the nginx web server is successfully installed and
      working. Further configuration is required.</p>
      
      <p>For online documentation and support please refer to
      <a href="http://nginx.org/">nginx.org</a>.<br/>
      Commercial support is available at
      <a href="http://nginx.com/">nginx.com</a>.</p>
      
      <p><em>Thank you for using nginx.</em></p>
      </body>
      </html>
      

      至此,部署KubeEdge+iSulad已经全流程打通。

      使用二进制部署

      用户也可以使用二进制部署kubeedge集群。只需要使用两个RPM包: cloudcore(云侧) 和edgecore(边缘侧) RPM 。

      使用二进制部署KubeEdge进行测试,切勿在生产环境中使用这种方式。

      部署云侧

      进入云侧主机。

      安装 cloudcore RPM包

      $ yum install kubeedge-cloudcore
      

      创建CRDs

      $ kubectl apply -f /etc/kubeedge/crds/devices/
      $ kubectl apply -f /etc/kubeedge/crds/reliablesyncs/
      $ kubectl apply -f /etc/kubeedge/crds/router/
      

      准备配置文件

      $ cloudcore --defaultconfig > /etc/kubeedge/config/cloudcore.yaml
      

      并按照使用keadm部署修改cloudcore相关配置。

      运行cloudcore

      $ pkill cloudcore
      $ systemctl start cloudcore
      

      部署边缘侧

      进入边缘侧主机。

      安装 edgecore RPM包

      $ yum install kubeedge-edgecore
      

      准备配置文件

      $ edgecore --defaultconfig > /etc/kubeedge/config/edgecore.yaml
      

      并按照使用keadm部署修改edgecore相关配置。

      $ kubectl get secret -nkubeedge tokensecret -o=jsonpath='{.data.tokendata}' | base64 -d
      1c4ff11289a14c59f2cbdbab726d1857262d5bda778ddf0de34dd59d125d3f69.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MzE0ODM3MzN9.JY77nMVDHIKD9ipo03Y0mSbxief9qOvJ4yMNx1yZpp0
      
      # 将获取到的token添加到配置文件中
      sed -i -e "s|token: .*|token: ${token}|g" /etc/kubeedge/config/edgecore.yaml
      
      # token变量的值来自于之前步骤
      

      运行edgecore

      $ pkill edgecore
      $ systemctl start edgecore
      

      附录

      kube-flannel-cloud.yml

      # 使用场景:云侧
      ---
      apiVersion: policy/v1beta1
      kind: PodSecurityPolicy
      metadata:
        name: psp.flannel.unprivileged
        annotations:
          seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
          seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
          apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
          apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
      spec:
        privileged: false
        volumes:
        - configMap
        - secret
        - emptyDir
        - hostPath
        allowedHostPaths:
        - pathPrefix: "/etc/cni/net.d"
        - pathPrefix: "/etc/kube-flannel"
        - pathPrefix: "/run/flannel"
        readOnlyRootFilesystem: false
        # Users and groups
        runAsUser:
          rule: RunAsAny
        supplementalGroups:
          rule: RunAsAny
        fsGroup:
          rule: RunAsAny
        # Privilege Escalation
        allowPrivilegeEscalation: false
        defaultAllowPrivilegeEscalation: false
        # Capabilities
        allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
        defaultAddCapabilities: []
        requiredDropCapabilities: []
        # Host namespaces
        hostPID: false
        hostIPC: false
        hostNetwork: true
        hostPorts:
        - min: 0
          max: 65535
        # SELinux
        seLinux:
          # SELinux is unused in CaaSP
          rule: 'RunAsAny'
      ---
      kind: ClusterRole
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: flannel
      rules:
      - apiGroups: ['extensions']
        resources: ['podsecuritypolicies']
        verbs: ['use']
        resourceNames: ['psp.flannel.unprivileged']
      - apiGroups:
        - ""
        resources:
        - pods
        verbs:
        - get
      - apiGroups:
        - ""
        resources:
        - nodes
        verbs:
        - list
        - watch
      - apiGroups:
        - ""
        resources:
        - nodes/status
        verbs:
        - patch
      ---
      kind: ClusterRoleBinding
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: flannel
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: flannel
      subjects:
      - kind: ServiceAccount
        name: flannel
        namespace: kube-system
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: flannel
        namespace: kube-system
      ---
      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: kube-flannel-cfg
        namespace: kube-system
        labels:
          tier: node
          app: flannel
      data:
        cni-conf.json: |
          {
            "name": "cbr0",
            "cniVersion": "0.3.1",
            "plugins": [
              {
                "type": "flannel",
                "delegate": {
                  "hairpinMode": true,
                  "isDefaultGateway": true
                }
              },
              {
                "type": "portmap",
                "capabilities": {
                  "portMappings": true
                }
              }
            ]
          }
        net-conf.json: |
          {
            "Network": "10.244.0.0/16",
            "Backend": {
              "Type": "vxlan"
            }
          }
      ---
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        name: kube-flannel-cloud-ds
        namespace: kube-system
        labels:
          tier: node
          app: flannel
      spec:
        selector:
          matchLabels:
            app: flannel
        template:
          metadata:
            labels:
              tier: node
              app: flannel
          spec:
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                  - matchExpressions:
                    - key: kubernetes.io/os
                      operator: In
                      values:
                      - linux
                    - key: node-role.kubernetes.io/agent
                      operator: DoesNotExist
            hostNetwork: true
            priorityClassName: system-node-critical
            tolerations:
            - operator: Exists
              effect: NoSchedule
            serviceAccountName: flannel
            initContainers:
            - name: install-cni
              image: quay.io/coreos/flannel:v0.14.0
              command:
              - cp
              args:
              - -f
              - /etc/kube-flannel/cni-conf.json
              - /etc/cni/net.d/10-flannel.conflist
              volumeMounts:
              - name: cni
                mountPath: /etc/cni/net.d
              - name: flannel-cfg
                mountPath: /etc/kube-flannel/
            containers:
            - name: kube-flannel
              image: quay.io/coreos/flannel:v0.14.0
              command:
              - /opt/bin/flanneld
              args:
              - --ip-masq
              - --kube-subnet-mgr
              resources:
                requests:
                  cpu: "100m"
                  memory: "50Mi"
                limits:
                  cpu: "100m"
                  memory: "50Mi"
              securityContext:
                privileged: false
                capabilities:
                  add: ["NET_ADMIN", "NET_RAW"]
              env:
              - name: POD_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.name
              - name: POD_NAMESPACE
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.namespace
              volumeMounts:
              - name: run
                mountPath: /run/flannel
              - name: flannel-cfg
                mountPath: /etc/kube-flannel/
            volumes:
            - name: run
              hostPath:
                path: /run/flannel
            - name: cni
              hostPath:
                path: /etc/cni/net.d
            - name: flannel-cfg
              configMap:
                name: kube-flannel-cfg
      

      kube-flannel-edge.yml

      # 使用场景:边缘侧
      ---
      apiVersion: policy/v1beta1
      kind: PodSecurityPolicy
      metadata:
        name: psp.flannel.unprivileged
        annotations:
          seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
          seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
          apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
          apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
      spec:
        privileged: false
        volumes:
        - configMap
        - secret
        - emptyDir
        - hostPath
        allowedHostPaths:
        - pathPrefix: "/etc/cni/net.d"
        - pathPrefix: "/etc/kube-flannel"
        - pathPrefix: "/run/flannel"
        readOnlyRootFilesystem: false
        # Users and groups
        runAsUser:
          rule: RunAsAny
        supplementalGroups:
          rule: RunAsAny
        fsGroup:
          rule: RunAsAny
        # Privilege Escalation
        allowPrivilegeEscalation: false
        defaultAllowPrivilegeEscalation: false
        # Capabilities
        allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
        defaultAddCapabilities: []
        requiredDropCapabilities: []
        # Host namespaces
        hostPID: false
        hostIPC: false
        hostNetwork: true
        hostPorts:
        - min: 0
          max: 65535
        # SELinux
        seLinux:
          # SELinux is unused in CaaSP
          rule: 'RunAsAny'
      ---
      kind: ClusterRole
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: flannel
      rules:
      - apiGroups: ['extensions']
        resources: ['podsecuritypolicies']
        verbs: ['use']
        resourceNames: ['psp.flannel.unprivileged']
      - apiGroups:
        - ""
        resources:
        - pods
        verbs:
        - get
      - apiGroups:
        - ""
        resources:
        - nodes
        verbs:
        - list
        - watch
      - apiGroups:
        - ""
        resources:
        - nodes/status
        verbs:
        - patch
      ---
      kind: ClusterRoleBinding
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: flannel
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: flannel
      subjects:
      - kind: ServiceAccount
        name: flannel
        namespace: kube-system
      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: flannel
        namespace: kube-system
      ---
      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: kube-flannel-cfg
        namespace: kube-system
        labels:
          tier: node
          app: flannel
      data:
        cni-conf.json: |
          {
            "name": "cbr0",
            "cniVersion": "0.3.1",
            "plugins": [
              {
                "type": "flannel",
                "delegate": {
                  "hairpinMode": true,
                  "isDefaultGateway": true
                }
              },
              {
                "type": "portmap",
                "capabilities": {
                  "portMappings": true
                }
              }
            ]
          }
        net-conf.json: |
          {
            "Network": "10.244.0.0/16",
            "Backend": {
              "Type": "vxlan"
            }
          }
      ---
      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        name: kube-flannel-edge-ds
        namespace: kube-system
        labels:
          tier: node
          app: flannel
      spec:
        selector:
          matchLabels:
            app: flannel
        template:
          metadata:
            labels:
              tier: node
              app: flannel
          spec:
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                  - matchExpressions:
                    - key: kubernetes.io/os
                      operator: In
                      values:
                      - linux
                    - key: node-role.kubernetes.io/agent
                      operator: Exists
            hostNetwork: true
            priorityClassName: system-node-critical
            tolerations:
            - operator: Exists
              effect: NoSchedule
            serviceAccountName: flannel
            initContainers:
            - name: install-cni
              image: quay.io/coreos/flannel:v0.14.0
              command:
              - cp
              args:
              - -f
              - /etc/kube-flannel/cni-conf.json
              - /etc/cni/net.d/10-flannel.conflist
              volumeMounts:
              - name: cni
                mountPath: /etc/cni/net.d
              - name: flannel-cfg
                mountPath: /etc/kube-flannel/
            containers:
            - name: kube-flannel
              image: quay.io/coreos/flannel:v0.14.0
              command:
              - /opt/bin/flanneld
              args:
              - --ip-masq
              - --kube-subnet-mgr
              - --kube-api-url=http://127.0.0.1:10550
              resources:
                requests:
                  cpu: "100m"
                  memory: "50Mi"
                limits:
                  cpu: "100m"
                  memory: "50Mi"
              securityContext:
                privileged: false
                capabilities:
                  add: ["NET_ADMIN", "NET_RAW"]
              env:
              - name: POD_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.name
              - name: POD_NAMESPACE
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.namespace
              volumeMounts:
              - name: run
                mountPath: /run/flannel
              - name: flannel-cfg
                mountPath: /etc/kube-flannel/
            volumes:
            - name: run
              hostPath:
                path: /run/flannel
            - name: cni
              hostPath:
                path: /etc/cni/net.d
            - name: flannel-cfg
              configMap:
                name: kube-flannel-cfg
      

      cloudcore.service

      # 使用场景:云侧
      # 文件位置:/usr/lib/systemd/system/cloudcore.service
      [Unit]
      Description=cloudcore.service
      
      [Service]
      Type=simple
      ExecStart=/usr/local/bin/cloudcore
      Restart=always
      RestartSec=10
      
      [Install]
      WantedBy=multi-user.target
      

      cloudcore.yaml

      # 使用场景:云侧
      # 文件位置:/etc/kubeedge/config/cloudcore.yaml
      apiVersion: cloudcore.config.kubeedge.io/v1alpha1
      commonConfig:
        tunnelPort: 10351
      kind: CloudCore
      kubeAPIConfig:
        burst: 200
        contentType: application/vnd.kubernetes.protobuf
        kubeConfig: /root/.kube/config
        master: ""
        qps: 100
      modules:
        cloudHub:
          advertiseAddress:
          - 9.63.252.224
          dnsNames:
          - ""
          edgeCertSigningDuration: 365
          enable: true
          https:
            address: 0.0.0.0
            enable: true
            port: 10002
          keepaliveInterval: 30
          nodeLimit: 1000
          quic:
            address: 0.0.0.0
            enable: false
            maxIncomingStreams: 10000
            port: 10001
          tlsCAFile: /etc/kubeedge/ca/rootCA.crt
          tlsCAKeyFile: /etc/kubeedge/ca/rootCA.key
          tlsCertFile: /etc/kubeedge/certs/server.crt
          tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
          tokenRefreshDuration: 12
          unixsocket:
            address: unix:///var/lib/kubeedge/kubeedge.sock
            enable: true
          websocket:
            address: 0.0.0.0
            enable: true
            port: 10000
          writeTimeout: 30
        cloudStream:
          enable: false
          streamPort: 10003
          tlsStreamCAFile: /etc/kubeedge/ca/streamCA.crt
          tlsStreamCertFile: /etc/kubeedge/certs/stream.crt
          tlsStreamPrivateKeyFile: /etc/kubeedge/certs/stream.key
          tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt
          tlsTunnelCertFile: /etc/kubeedge/certs/server.crt
          tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key
          tunnelPort: 10004
        deviceController:
          buffer:
            deviceEvent: 1
            deviceModelEvent: 1
            updateDeviceStatus: 1024
          context:
            receiveModule: devicecontroller
            responseModule: cloudhub
            sendModule: cloudhub
          enable: true
          load:
            updateDeviceStatusWorkers: 1
        dynamicController:
          enable: true
        edgeController:
          buffer:
            configMapEvent: 1
            deletePod: 1024
            endpointsEvent: 1
            podEvent: 1
            queryConfigMap: 1024
            queryEndpoints: 1024
            queryNode: 1024
            queryPersistentVolume: 1024
            queryPersistentVolumeClaim: 1024
            querySecret: 1024
            queryService: 1024
            queryVolumeAttachment: 1024
            ruleEndpointsEvent: 1
            rulesEvent: 1
            secretEvent: 1
            serviceAccountToken: 1024
            serviceEvent: 1
            updateNode: 1024
            updateNodeStatus: 1024
            updatePodStatus: 1024
          context:
            receiveModule: edgecontroller
            responseModule: cloudhub
            sendModule: cloudhub
            sendRouterModule: router
          enable: true
          load:
            ServiceAccountTokenWorkers: 4
            UpdateRuleStatusWorkers: 4
            deletePodWorkers: 4
            queryConfigMapWorkers: 4
            queryEndpointsWorkers: 4
            queryNodeWorkers: 4
            queryPersistentVolumeClaimWorkers: 4
            queryPersistentVolumeWorkers: 4
            querySecretWorkers: 4
            queryServiceWorkers: 4
            queryVolumeAttachmentWorkers: 4
            updateNodeStatusWorkers: 1
            updateNodeWorkers: 4
            updatePodStatusWorkers: 1
          nodeUpdateFrequency: 10
        router:
          address: 0.0.0.0
          enable: false
          port: 9443
          restTimeout: 60
        syncController:
          enable: true
      

      edgecore.service

      # 使用场景:边缘侧
      # 文件位置:/etc/systemd/system/edgecore.service
      [Unit]
      Description=edgecore.service
      
      [Service]
      Type=simple
      ExecStart=/usr/local/bin/edgecore
      Restart=always
      RestartSec=10
      
      [Install]
      WantedBy=multi-user.target
      

      edgecore.yaml

      # 使用场景:边缘侧
      # 文件位置:/etc/kubeedge/config/edgecore.yaml
      apiVersion: edgecore.config.kubeedge.io/v1alpha1
      database:
        aliasName: default
        dataSource: /var/lib/kubeedge/edgecore.db
        driverName: sqlite3
      kind: EdgeCore
      modules:
        dbTest:
          enable: false
        deviceTwin:
          enable: true
        edgeHub:
          enable: true
          heartbeat: 15
          httpServer: https://9.63.252.224:10002
          projectID: e632aba927ea4ac2b575ec1603d56f10
          quic:
            enable: false
            handshakeTimeout: 30
            readDeadline: 15
            server: 9.63.252.227:10001
            writeDeadline: 15
          rotateCertificates: true
          tlsCaFile: /etc/kubeedge/ca/rootCA.crt
          tlsCertFile: /etc/kubeedge/certs/server.crt
          tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
          token: "" # 这里填写从云侧获取的token
          websocket:
            enable: true
            handshakeTimeout: 30
            readDeadline: 15
            server: 9.63.252.224:10000
            writeDeadline: 15
        edgeMesh:
          enable: false
          lbStrategy: RoundRobin
          listenInterface: docker0
          listenPort: 40001
          subNet: 9.251.0.0/16
        edgeStream:
          enable: false
          handshakeTimeout: 30
          readDeadline: 15
          server: 9.63.252.224:10004
          tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt
          tlsTunnelCertFile: /etc/kubeedge/certs/server.crt
          tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key
          writeDeadline: 15
        edged:
          cgroupDriver: cgroupfs
          cgroupRoot: ""
          cgroupsPerQOS: true
          clusterDNS: ""
          clusterDomain: ""
          cniBinDir: /opt/cni/bin
          cniCacheDirs: /var/lib/cni/cache
          cniConfDir: /etc/cni/net.d
          concurrentConsumers: 5
          devicePluginEnabled: false
          dockerAddress: unix:///var/run/docker.sock
          edgedMemoryCapacity: 7852396000
          enable: true
          enableMetrics: true
          gpuPluginEnabled: false
          hostnameOverride: edge.kubeedge
          imageGCHighThreshold: 80
          imageGCLowThreshold: 40
          imagePullProgressDeadline: 60
          maximumDeadContainersPerPod: 1
          networkPluginMTU: 1500
          nodeIP: 9.63.252.227
          nodeStatusUpdateFrequency: 10
          podSandboxImage: kubeedge/pause:3.1
          registerNode: true
          registerNodeNamespace: default
          remoteImageEndpoint: unix:///var/run/isulad.sock
          remoteRuntimeEndpoint: unix:///var/run/isulad.sock
          runtimeRequestTimeout: 2
          runtimeType: remote
          volumeStatsAggPeriod: 60000000000
        eventBus:
          enable: true
          eventBusTLS:
            enable: false
            tlsMqttCAFile: /etc/kubeedge/ca/rootCA.crt
            tlsMqttCertFile: /etc/kubeedge/certs/server.crt
            tlsMqttPrivateKeyFile: /etc/kubeedge/certs/server.key
          mqttMode: 2
          mqttQOS: 0
          mqttRetain: false
          mqttServerExternal: tcp://127.0.0.1:1883
          mqttServerInternal: tcp://127.0.0.1:1884
          mqttSessionQueueSize: 100
        metaManager:
          contextSendGroup: hub
          contextSendModule: websocket
          enable: true
          metaServer:
            debug: false
            enable: true
          podStatusSyncInterval: 60
          remoteQueryTimeout: 60
        serviceBus:
          enable: false
      

      daemon.json

      # 使用场景:云侧、边缘侧
      # 文件位置:/etc/isulad/daemon.json
      {
          "group": "isula",
          "default-runtime": "lcr",
          "graph": "/var/lib/isulad",
          "state": "/var/run/isulad",
          "engine": "lcr",
          "log-level": "ERROR",
          "pidfile": "/var/run/isulad.pid",
          "log-opts": {
              "log-file-mode": "0600",
              "log-path": "/var/lib/isulad",
              "max-file": "1",
              "max-size": "30KB"
          },
          "log-driver": "stdout",
          "container-log": {
              "driver": "json-file"
          },
          "hook-spec": "/etc/default/isulad/hooks/default.json",
          "start-timeout": "2m",
          "storage-driver": "overlay2",
          "storage-opts": [
              "overlay2.override_kernel_check=true"
          ],
          "registry-mirrors": [
              "docker.io"
          ],
          "insecure-registries": [
              "k8s.gcr.io",
              "quay.io",
              "hub.oepkgs.net"
          ],
          "pod-sandbox-image": "k8s.gcr.io/pause:3.2",	# 边缘侧时配置为 kubeedge/pause:3.1
          "websocket-server-listening-port": 10351,
          "native.umask": "secure",
          "network-plugin": "cni",
          "cni-bin-dir": "/opt/cni/bin",
          "cni-conf-dir": "/etc/cni/net.d",
          "image-layer-check": false,
          "use-decrypted-key": true,
          "insecure-skip-verify-enforce": false
      }
      

      文档捉虫

      “有虫”文档片段

      问题描述

      提交类型 issue

      有点复杂...

      找人问问吧。

      PR

      小问题,全程线上修改...

      一键搞定!

      问题类型
      规范和低错类

      ● 错别字或拼写错误;标点符号使用错误;

      ● 链接错误、空单元格、格式错误;

      ● 英文中包含中文字符;

      ● 界面和描述不一致,但不影响操作;

      ● 表述不通顺,但不影响理解;

      ● 版本号不匹配:如软件包名称、界面版本号;

      易用性

      ● 关键步骤错误或缺失,无法指导用户完成任务;

      ● 缺少必要的前提条件、注意事项等;

      ● 图形、表格、文字等晦涩难懂;

      ● 逻辑不清晰,该分类、分项、分步骤的没有给出;

      正确性

      ● 技术原理、功能、规格等描述和软件不一致,存在错误;

      ● 原理图、架构图等存在错误;

      ● 命令、命令参数等错误;

      ● 代码片段错误;

      ● 命令无法完成对应功能;

      ● 界面错误,无法指导操作;

      风险提示

      ● 对重要数据或系统存在风险的操作,缺少安全提示;

      内容合规

      ● 违反法律法规,涉及政治、领土主权等敏感词;

      ● 内容侵权;

      您对文档的总体满意度

      非常不满意
      非常满意
      提交
      根据您的反馈,会自动生成issue模板。您只需点击按钮,创建issue即可。
      文档捉虫
      编组 3备份