Imperceptible Container Management Plane Offload
Overview
Moore's law ceases to apply in data center and cloud scenarios. The CPU computing power growth rate of general processing units is slowing down, while the network I/O speed and performance keep increasing. As a result, the processing capability of current general-purpose processors cannot meet the I/O processing requirements of the network and drives. In traditional data centers, more and more general-purpose CPU computing power is occupied by I/O and management planes. This part of resource loss is called data center tax. According to AWS statistics, the data center tax may account for more than 30% of the computing power of the data center.
The data processing unit (DPU) is developed to release the computing resources from the host CPU. The management plane, network, storage, and security capabilities are offloaded to DPUs for acceleration, reducing costs and improving efficiency. Mainstream cloud vendors, such as AWS, Alibaba Cloud, and Huawei Cloud, use self-developed processors to offload the management plane and related data plane, achieving 100% utilization of data center computing resources.
The management plane processes can be offloaded to the DPU by splitting the component source code. The source code is split into two parts that run independently on the host and DPU based on the function logic. In this way, the component is offloaded. However, this method has the following problems:
- The software compatibility of the component is affected. You need to maintain the component and related patches in subsequent version upgrades, which increases the maintenance workload.
- The offload cannot be inherited by other components. You need to split each component based on code logic analysis.
To solve these problems, openEuler introduces imperceptible DPU offload. The abstraction layer provided by the OS shields the cross-host access differences between the host and DPU, and enables service processes to be offloaded to the DPU with virtually zero modification. This part of work at the common layer of the OS and is irrelevant to upper-layer services. Other services can also inherit the offload to DPU.
Architecture
Imperceptible Container Management Plane DPU Offload Architecture
Figure 1 Imperceptible Container Management Plane DPU Offload Architecture
As shown in Figure 1, after the container management plane is offloaded, management processes such as dockerd and kubelet run on the DPU side, and container processes run on the host. The interaction between processes is ensured by the system layer.
Communication layer: DPUs and hosts can communicate with each other through PCIe interfaces or networks. A communication interface layer is provided based on underlying physical connections to provide communication interfaces for upper-layer services.
qtfs kernel shared file system: The container management plane components kubelet and dockerd interact with container processes through file systems. Management plane tools need to prepare data plane paths to rootfs and volume for container processes. In addition, the proc and cgroup file systems need to be used to control and monitor the resources and status of container processes. For details about qtfs, see qtfs Shared File System Introduction and Usage.
User-mode offload environment: You need to use qtfs to prepare the runtime environment for the offloaded management plane, and remotely mount the container management and runtime directories of the host to the DPU. System management file systems such as proc, sys, and cgroup need to be mounted. To prevent damage to the native system functions of the DPU, the preceding mounting operations are performed in the chroot environment. In addition, the management plane (running on the DPU) and container processes (running on the host) have invoking relationships. The rexec remote binary execution tool needs to be used to provide corresponding functions.
For details about how to offload container management plane, see the Deployment Guide.
NOTE:
In this user guide, modifications are performed to the container management plane components and the rexec tool of a specific version. You can modify other versions based on the actual execution environment. The patch provided in this document is for verification only and is not for commercial use.
