Container Resource Management

• Container Resource Management
  • Description
  • Sharing Resources
  • Restricting CPU Resources of a Running Container
  • Restricting the Memory Usage of a Running Container
  • Restricting I/O Resources of a Running Container
  • Restricting the Rootfs Storage Space of a Container
  • Restricting the Number of File Handles in a Container
  • Restricting the Number of Processes or Threads that Can Be Created in a Container
  • Configuring the ulimit Value in a Container

Description

You can use namespaces and cgroups to manage container resources. iSula can use cgroup v1 and cgroup v2 to restrict resources. cgroup v2 is an experimental feature and cannot be put into commercial use. When the system is configured to support only cgroup v2 and cgroup v2 is mounted to the /sys/fs/cgroup directory, iSula uses cgroup v2 for resource management. Whether cgroup v1 or cgroup v2 is used to manage container resources, iSula provides the same interface for users to implement resource restriction.

Sharing Resources

Description

Containers or containers and hosts can share namespace information mutually, including PID, network, IPC, and UTS information.

NOTE: When namespace information is shared with a host, the namespace isolation mechanism is unavailable. As a result, information on the host can be queried and operated in a container, causing security risks. For example, if --pid=host is used to share the PID namespace of a host, information about other processes on the host can be viewed, causing information leakage or even killing the host process. Exercise caution when using the shared host namespace function to ensure security.

Usage

When running the isula create/run command, you can set the namespace parameters to share resources. For details, see the following parameter description table.

Parameters

You can specify the following parameters when running the isula create/run command:

Example

If two containers need to share the same PID namespace, add --pid container:<containerID> when running the container. For example:

isula run -tid --name test_pid busybox sh
isula run -tid --name test --pid container:test_pid busybox sh

Restricting CPU Resources of a Running Container

Description

You can set parameters to restrict the CPU resources of a container.

Usage

When running the isula create/run command, you can set CPU-related parameters to limit the CPU resources of a container. For details about the parameters and values, see the following table.

Parameters

You can specify the following parameters when running the isula create/run command:

Example

To restrict a container to use a specific CPU, add --cpuset-cpus number when running the container. For example:

isula run -tid --cpuset-cpus 0,2-3 busybox sh

NOTE: You can check whether the configuration is successful. For details, see "Querying Information About a Single Container."

Restricting the Memory Usage of a Running Container

Description

You can set parameters to restrict the memory usage of a container.

Usage

When running the isula create/run command, you can set memory-related parameters to restrict memory usage of containers. For details about the parameters and values, see the following table.

Parameters

You can specify the following parameters when running the isula create/run command:

Example

To set the upper limit of the memory of a container, add --memory <number>[<unit>] when running the container. For example:

isula run -tid --memory 1G busybox sh

Restricting I/O Resources of a Running Container

Description

You can set parameters to limit the read/write speed of devices in the container.

Usage

When running the isula create/run command, you can set --device-read-bps/--device-write-bps <device-path>:<number>[<unit>] to limit the read/write speed of devices in the container.

Parameters

When running the isula create/run command, set --device-read/write-bps.

Example

To limit the read/write speed of devices in the container, add --device-write-bps/--device-read-bps <device-path>:<number>[<unit>] when running the container. For example, to limit the read speed of the device /dev/sda in the container busybox to 1 Mbit/s, run the following command:

isula run -tid --device-read-bps /dev/sda:1mb busybox sh

To limit the write speed, run the following command:

isula run -tid --device-write-bps /dev/sda:1mb busybox sh

Restricting the Rootfs Storage Space of a Container

Description

When the overlay2 storage driver is used on the EXT4 file system, the file system quota of a single container can be set. For example, the quota of container A is set to 5 GB, and the quota of container B is set to 10 GB.

This feature is implemented by the project quota function of the EXT4 file system. If the kernel supports this function, use the syscall SYS_IOCTL to set the project ID of a directory, and then use the syscall SYS_QUOTACTL to set the hard limit and soft limit of the corresponding project ID.

Usage

1. Prepare the environment.

   Ensure that the file system supports the Project ID and Project Quota attributes, the kernel version is 4.19 or later, and the version of the peripheral package e2fsprogs is 1.43.4-2 or later.

2. Before mounting overlayfs to a container, set different project IDs for the upper and work directories of different containers and set inheritance options. After overlayfs is mounted to a container, the project IDs and inherited attributes cannot be modified.

3. Set the quota as a privileged user outside the container.

4. Add the following configuration to daemon:

   -s overlay2 --storage-opt overlay2.override_kernel_check=true
   

5. Daemon supports the following options for setting default restrictions for containers:

   --storage-opt overlay2.basesize=128M specifies the default limit. If --storage-opt size is also specified when you run the isula run command, the value of this parameter takes effect. If no size is specified during the daemon process or when you run the isula run command, the size is not limited.

6. Enable the Project ID and Project Quota attributes of the file system.

   • Format and mount the file system.

     mkfs.ext4 -O quota,project /dev/sdb
     mount -o prjquota /dev/sdb /var/lib/isulad
     

Parameters

When running the create/run command, set --storage-opt.

Example

In the isula run/create command, use the existing parameter --storage-opt size=value to set the quota. The value is a positive number in the unit of [kKmMgGtTpP]?[iI]?[bB]?. If the value does not contain a unit, the default unit is byte.

$ [root@localhost ~]# isula run -ti --storage-opt size=10M busybox
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  10.0M     48.0K     10.0M   0% /
none                     64.0M         0     64.0M   0% /dev
none                     10.0M         0     10.0M   0% /sys/fs/cgroup
tmpfs                    64.0M         0     64.0M   0% /dev
shm                      64.0M         0     64.0M   0% /dev/shm
/dev/mapper/vg--data-ext41
                          9.8G     51.5M      9.2G   1% /etc/hostname
/dev/mapper/vg--data-ext41
                          9.8G     51.5M      9.2G   1% /etc/resolv.conf
/dev/mapper/vg--data-ext41
                          9.8G     51.5M      9.2G   1% /etc/hosts
tmpfs                     3.9G         0      3.9G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                     3.9G         0      3.9G   0% /proc/scsi
tmpfs                    64.0M         0     64.0M   0% /proc/fdthreshold
tmpfs                    64.0M         0     64.0M   0% /proc/fdenable
tmpfs                     3.9G         0      3.9G   0% /sys/firmware
/ # 
/ # dd if=/dev/zero of=/home/img bs=1M count=12 && sync
dm-4: write failed, project block limit reached.
10+0 records in
9+0 records out
10432512 bytes (9.9MB) copied, 0.011782 seconds, 844.4MB/s
/ # df -h | grep overlay
overlay                  10.0M     10.0M         0 100% /
/ # 

Constraints

1. The quota applies only to the rw layer.

   The quota of overlay2 is for the rw layer of the container. The image size is not included.

2. The kernel supports and enables this function.

   The kernel must support the EXT4 project quota function. When running mkfs, add -O quota,project. When mounting the file system, add -o prjquota. If any of the preceding conditions is not met, an error is reported when --storage-opt size=value is used.

   $ [root@localhost ~]# isula run -it --storage-opt size=10Mb busybox df -h
   Error response from daemon: Failed to prepare rootfs with error: time="2019-04-09T05:13:52-04:00" level=fatal msg="error creating read-
   write layer with ID "a4c0e55e82c55e4ee4b0f4ee07f80cc2261cf31b2c2dfd628fa1fb00db97270f": --storage-opt is supported only for overlay over
   xfs or ext4 with 'pquota' mount option"
   

3. Description of the limit of quota:

   1. If the quota is greater than the size of the partition where user root of iSulad is located, the file system quota displayed by running the df command in the container is the size of the partition where user root of iSulad is located, not the specified quota.
   2. --storage-opt size=0 indicates that the size is not limited and the value cannot be less than 4096. The precision of size is one byte. If the specified precision contains decimal bytes, the decimal part is ignored. For example, if size is set to 0.1, the size is not limited. (The value is restricted by the precision of the floating point number stored on the computer. That is, 0.999999999999999999999999999 is equal to 1. The number of digits 9 may vary according to computers. Therefore, 4095.999999999999999999999999999 is equal to 4096.) Note that running isula inspect displays the original command line specified format. If the value contains decimal bytes, you need to ignore the decimal part.
   3. If the quota is too small, for example,--storage-opt size=4k, the container may fail to be started because some files need to be created for starting the container.
   4. The -o prjquota option is added to the root partition of iSulad when iSulad is started last time. If this option is not added during this startup, the setting of the container with quota created during the last startup does not take effect.
   5. The value range of the daemon quota --storage-opt overlay2.basesize is the same as that of --storage-opt size.

4. When storage-opt is set to 4 KB, the lightweight container startup is different from that of Docker.

   Use the storage-opt size=4k and image rnd-dockerhub.huawei.com/official/ubuntu-arm64:latest to run the container.

   Docker fails to be started.

   [root@localhost ~]# docker run -itd --storage-opt size=4k rnd-dockerhub.huawei.com/official/ubuntu-arm64:latest
   docker: Error response from daemon: symlink /proc/mounts /var/lib/docker/overlay2/e6e12701db1a488636c881b44109a807e187b8db51a50015db34a131294fcf70-init/merged/etc/mtab: disk quota exceeded.
   See 'docker run --help'.
   

   The lightweight container is started properly and no error is reported.

   [root@localhost ~]# isula run -itd --storage-opt size=4k rnd-dockerhub.huawei.com/official/ubuntu-arm64:latest
   636480b1fc2cf8ac895f46e77d86439fe2b359a1ff78486ae81c18d089bbd728
   [root@localhost ~]# isula ps
   STATUS  PID   IMAGE                                                 COMMAND   EXIT_CODE RESTART_COUNT STARTAT       FINISHAT RUNTIME ID           NAMES                                                            
   running 17609 rnd-dockerhub.huawei.com/official/ubuntu-arm64:latest /bin/bash 0         0             2 seconds ago -        lcr     636480b1fc2c 636480b1fc2cf8ac895f46e77d86439fe2b359a1ff78486ae81c18d089bbd728 
   

   During container startup, if you need to create a file in the rootfs directory of the container, the image size exceeds 4 KB, and the quota is set to 4 KB, the file creation will fail.

   When Docker starts the container, it creates more mount points than iSulad to mount some directories on the host to the container, such as /proc/mounts and /dev/shm. If these files do not exist in the image, the creation will fail, therefore, the container fails to be started.

   When a lightweight container uses the default configuration during container startup, there are few mount points. The lightweight container is created only when the directory like /proc or /sys does not exist. The image rnd-dockerhub.huawei.com/official/ubuntu-arm64:latest in the test case contains /proc and /sys. Therefore, no new file or directory is generated during the container startup. As a result, no error is reported during the lightweight container startup. To verify this process, when the image is replaced with rnd-dockerhub.huawei.com/official/busybox-aarch64:latest, an error is reported when the lightweight container is started because /proc does not exist in the image.

   [root@localhost ~]# isula run -itd --storage-opt size=4k rnd-dockerhub.huawei.com/official/busybox-aarch64:latest
   8e893ab483310350b8caa3b29eca7cd3c94eae55b48bfc82b350b30b17a0aaf4
   Error response from daemon: Start container error: runtime error: 8e893ab483310350b8caa3b29eca7cd3c94eae55b48bfc82b350b30b17a0aaf4:tools/lxc_start.c:main:404 starting container process caused "Failed to setup lxc,
   please check the config file."
   

5. Other description:

   When using iSulad with the quota function to switch data disks, ensure that the data disks to be switched are mounted using the prjquota option and the mounting mode of the /var/lib/isulad/storage/overlay2 directory is the same as that of the /var/lib/isulad directory.

   NOTE: Before switching the data disk, ensure that the mount point of /var/lib/isulad/storage/overlay2 is unmounted.

Restricting the Number of File Handles in a Container

Description

You can set parameters to limit the number of file handles that can be opened in a container.

Usage

When running the isula create/run command, set the --files-limit parameter to limit the number of file handles that can be opened in a container.

Parameters

Set the --files-limit parameter when running the isula create/run command.

Example

When running the container, add --files-limit n. For example:

isula run -ti --files-limit 1024 busybox bash

Constraints

1. If the --files-limit parameter is set to a small value, for example, 1, the container may fail to be started.

   [root@localhost ~]# isula run -itd --files-limit 1 rnd-dockerhub.huawei.com/official/busybox-aarch64
   004858d9f9ef429b624f3d20f8ba12acfbc8a15bb121c4036de4e5745932eff4
   Error response from daemon: Start container error: Container is not running:004858d9f9ef429b624f3d20f8ba12acfbc8a15bb121c4036de4e5745932eff4
   

   Docker will be started successfully, and the value of files.limit cgroup is max.

   [root@localhost ~]# docker run -itd --files-limit 1 rnd-dockerhub.huawei.com/official/busybox-aarch64
   ef9694bf4d8e803a1c7de5c17f5d829db409e41a530a245edc2e5367708dbbab
   [root@localhost ~]# docker exec -it ef96 cat /sys/fs/cgroup/files/files.limit
   max
   

   The root cause is that the startup principles of the lxc and runc processes are different. After the lxc process creates the cgroup, the files.limit value is set, and then the PID of the container process is written into the cgroup.procs file of the cgroup. At this time, the process has opened more than one handle. As a result, an error is reported, and the startup fails. After you create a cgroup by running the runc command, the PID of the container process is written to the cgroup.procs file of the cgroup, and then the files.limit value is set. Because more than one handle is opened by the process in the cgroup, the file.limit value does not take effect, the kernel does not report any error, and the container is started successfully.

Restricting the Number of Processes or Threads that Can Be Created in a Container

Description

You can set parameters to limit the number of processes or threads that can be created in a container.

Usage

When creating or running a container, use the --pids-limit parameter to limit the number of processes or threads that can be created in the container.

Parameters

When running the create/run command, set the --pids-limit parameter.

Example

When running the container, add --pids-limit n. For example:

isula run -ti --pids-limit 1024 busybox bash

Constraints

During container creation, some processes are created temporarily. Therefore, the value cannot be too small. Otherwise, the container may fail to be started. It is recommended that the value be greater than 10.

Configuring the ulimit Value in a Container

Description

You can use parameters to control the resources for executed programs.

Usage

Set the --ulimit parameter when creating or running a container, or configure the parameter on the daemon to control the resources for executed programs in the container.

Parameters

Use either of the following methods to configure ulimit:

1. When running the isula create/run command, use --ulimit <type>=<soft>[:<hard>] to control the resources of the executed shell program.

   Parameter	Description	Value Range	Mandatory or Not
   --ulimit	Limits the resources of the executed shell program.	64-bit integer The value of the soft limit must be less than or equal to that of the hard limit. If only the soft limit is specified, the value of the hard limit is equal to that of the soft limit. Some types of resources do not support negative numbers. For details, see the following table.	No

2. Use daemon parameters or configuration files.

   For details, see --default-ulimits in Configuration Mode.

   --ulimit can limit the following types of resources:

   Type	Description	Value Range
   core	limits the core file size (KB)	64-bit integer, without unit. The value can be 0 or a negative number. The value -1 indicates no limit. Other negative numbers are forcibly converted into a large positive integer.
   cpu	max CPU time (MIN)
   data	max data size (KB)
   fsize	maximum filesize (KB)
   locks	max number of file locks the user can hold
   memlock	max locked-in-memory address space (KB)
   msgqueue	max memory used by POSIX message queues (bytes)
   nice	nice priority
   nproc	max number of processes
   rss	max resident set size (KB)
   rtprio	max realtime priority
   rttime	realtime timeout
   sigpending	max number of pending signals
   stack	max stack size (KB)
   nofile	max number of open file descriptors	64-bit integer, without unit. The value cannot be negative. A negative number is forcibly converted to a large positive number. In addition, "Operation not permitted" is displayed during the setting.

Example

When creating or running a container, add --ulimit <type>=<soft>[:<hard>]. For example:

isula create/run -tid --ulimit nofile=1024:2048 busybox sh

Constraints

The ulimit cannot be configured in the daemon.json and /etc/sysconfig/iSulad files (or the iSulad command line). Otherwise, an error is reported when iSulad is started.