Appendix

DaemonSet Configuration Template

yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rubik
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["pods/eviction"]
    verbs: ["create"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rubik
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rubik
subjects:
  - kind: ServiceAccount
    name: rubik
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rubik
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rubik-config
  namespace: kube-system
data:
  config.json: |
    {
      "agent": {
        "logDriver": "stdio",
        "logDir": "/var/log/rubik",
        "logSize": 1024,
        "logLevel": "info",
        "cgroupRoot": "/sys/fs/cgroup",
        "enabledFeatures": [
          "preemption"
        ]
      },
      "preemption": {
        "resource": [
          "cpu"
        ]
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: rubik-agent
  namespace: kube-system
  labels:
    k8s-app: rubik-agent
spec:
  selector:
    matchLabels:
      name: rubik-agent
  template:
    metadata:
      namespace: kube-system
      labels:
        name: rubik-agent
    spec:
      serviceAccountName: rubik
      hostPID: true
      containers:
      - name: rubik-agent
        image: hub.oepkgs.net/cloudnative/rubik:latest
        imagePullPolicy: IfNotPresent
        env:
          - name: RUBIK_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
        securityContext:
          capabilities:
            add:
            - SYS_ADMIN
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: rubiklog
          mountPath: /var/log/rubik
          readOnly: false
        - name: runrubik
          mountPath: /run/rubik
          readOnly: false
        - name: sysfs
          mountPath: /sys/fs
          readOnly: false
        - name: devfs
          mountPath: /dev
          readOnly: false
        - name: config-volume
          mountPath: /var/lib/rubik
      terminationGracePeriodSeconds: 30
      volumes:
      - name: rubiklog
        hostPath:
          path: /var/log/rubik
      - name: runrubik
        hostPath:
          path: /run/rubik
      - name: sysfs
        hostPath:
          path: /sys/fs
      - name: devfs
        hostPath:
          path: /dev
      - name: config-volume
        configMap:
          name: rubik-config
          items:
          - key: config.json
            path: config.json

Dockerfile Template

dockerfile
FROM scratch
COPY ./build/rubik /rubik
ENTRYPOINT ["/rubik"]

Image Build Script

bash
#!/bin/bash
set -e

CURRENT_DIR=$(cd "$(dirname "$0")" && pwd)
BINARY_NAME="rubik"

RUBIK_FILE="${CURRENT_DIR}/build/rubik"
DOCKERFILE="${CURRENT_DIR}/Dockerfile"
YAML_FILE="${CURRENT_DIR}/rubik-daemonset.yaml"

# Get version and release number of rubik binary
VERSION=$(${RUBIK_FILE} -v | grep ^Version | awk '{print $NF}')
RELEASE=$(${RUBIK_FILE} -v | grep ^Release | awk '{print $NF}')
IMG_TAG="${VERSION}-${RELEASE}"

# Get rubik image name and tag
IMG_NAME_AND_TAG="${BINARY_NAME}:${IMG_TAG}"

# Build container image for rubik
docker build -f "${DOCKERFILE}" -t "${IMG_NAME_AND_TAG}" "${CURRENT_DIR}"

echo -e "\n"
# Check image existence
docker images | grep -E "REPOSITORY|${BINARY_NAME}"

# Modify rubik-daemonset.yaml file, set rubik image name
sed -i "/image:/s/:.*/: ${IMG_NAME_AND_TAG}/" "${YAML_FILE}"

Communication Matrix

  • The Rubik service process communicates with the Kubernetes API server as a client through the list-watch mechanism to obtain information about Pods.
Source IP AddressSource PortDestination IP AddressDestination PortProtocolPort DescriptionListening Port ModifiableAuthentication Method
Rubik node32768-61000api-server node443TCPKubernetes external resource portNoToken

File Permissions

  • All Rubik operations require root permissions.

  • Related file permissions are as follows:

PathPermissionsDescription
/var/lib/rubik750Directory generated after the RPM package is installed, which stores Rubik-related files
/var/lib/rubik/build550Directory for storing the Rubik binary file
/var/lib/rubik/build/rubik550Rubik binary file
/var/lib/rubik/rubik-daemonset.yaml550Rubik DaemonSet configuration template to be used for Kubernetes deployment
/var/lib/rubik/Dockerfile640Dockerfile template
/var/lib/rubik/build_rubik_image.sh550Rubik container image build script.
/var/log/rubik640Directory for storing Rubik log files (requires logDriver=file)
/var/log/rubik/rubik.log*600Rubik log files

Constraints

Specifications

  • Drive: More than 1 GB

  • Memory: More than 100 MB

Runtime

  • Only one Rubik instance can exist on a Kubernetes node.

  • Rubik cannot take any CLI parameters. Rubik will fail to be started if any CLI parameter is specified.

  • When the Rubik process is in the T (TASK_STOPPED or TASK_TRACED) OR D (TASK_UNINTERRUPTIBLE) state, the server is unavailable and does not respond. The service becomes available after the process recovers from the abnormal state.

Pod Priorities

  • Pod priorities cannot be raised. If the priority of service A is changed from -1 to 0, Rubik will report an error.

  • Adding or modifying annotations or re-applying Pod YAML configuration file does not trigger Pod rebuild. Rubik senses changes in Pod annotations through the list-watch mechanism.

  • After an online service is moved to the offline group, do not move it back to the online group, otherwise QoS exception may occur.

  • Do not add important system services and kernel processes to the offline group. Otherwise, they cannot be scheduled timely, causing system errors.

  • Online and offline configurations for the CPU and memory must be consistent to avoid QoS conflicts between the two subsystems.

  • In the scenario of hybrid service deployment, the original CPU share mechanism is restricted:

    • When both online and offline services run on a CPU, the CPU share of the offline service does not take effect.
    • If only an online or offline service runs on a CPU, its CPU share takes effect.
    • You are advised to set the Pod priority of the offline service to BestEffort.
  • Priority inversion of user-mode processes, SMT, cache, NUMA load balancing, and offline service load balancing are not supported.

Other

To prevent data inconsistency, do not manually modify cgroup or resctrl parameters of the pods, including:

  • CPU cgroup directory, such as /sys/fs/cgroup/cpu/kubepods/burstable/<PodUID>/<container-longid>

    • cpu.qos_level
    • cpu.cfs_burst_us
  • memory cgroup directory, such as /sys/fs/cgroup/memory/kubepods/burstable/<PodUID>/<container-longid>

    • memory.qos_level
    • memory.soft_limit_in_bytes
    • memory.force_empty
    • memory.limit_in_bytes
    • memory.high
  • RDT cgroup directory, such as /sys/fs/resctrl