Installation and Configuration
This section describes how to install, configure, upgrade, and uninstall iSulad.
Note: You need root permissions to install, upgrade, or uninstall iSulad.
Installation Methods
iSulad can be installed by running the yum or rpm command. The yum command is recommended because dependencies can be installed automatically.
This section describes two installation methods.
(Recommended) Run the following command to install iSulad:
sudo yum install -y iSuladIf the rpm command is used to install iSulad, you need to download and manually install the RMP packages of iSulad and all its dependencies. To install the RPM package of a single iSulad (the same for installing dependency packages), run the following command:
sudo rpm -ihv iSulad-xx.xx.xx-YYYYmmdd.HHMMSS.gitxxxxxxxx.aarch64.rpmisulad -l DEBUG
Deployment Configuration
After iSulad is installed, you can perform related configurations as required.
Configuration Mode
The iSulad server daemon isulad can be configured with a configuration file or by running the isulad --xxx command. The priority in descending order is as follows: CLI > configuration file > default configuration in code.
CLI
During service startup, configure iSulad using the CLI. To view the configuration options, run the following command:
# isulad --help lightweight container runtime daemon Usage: isulad [global options] GLOBAL OPTIONS: --authorization-plugin Use authorization plugin --cgroup-parent Set parent cgroup for all containers --cni-bin-dir The full path of the directory in which to search for CNI plugin binaries. Default: /opt/cni/bin --cni-conf-dir The full path of the directory in which to search for CNI config files. Default: /etc/cni/net.d --default-ulimit Default ulimits for containers (default []) -e, --engine Select backend engine -g, --graph Root directory of the iSulad runtime -G, --group Group for the unix socket(default is isulad) --help Show help --hook-spec Default hook spec file applied to all containers -H, --host The socket name used to create gRPC server --image-layer-check Check layer intergrity when needed --image-opt-timeout Max timeout(default 5m) for image operation --insecure-registry Disable TLS verification for the given registry --insecure-skip-verify-enforce Force to skip the insecure verify(default false) --log-driver Set daemon log driver, such as: file -l, --log-level Set log level, the levels can be: FATAL ALERT CRIT ERROR WARN NOTICE INFO DEBUG TRACE --log-opt Set daemon log driver options, such as: log-path=/tmp/logs/ to set directory where to store daemon logs --native.umask Default file mode creation mask (umask) for containers --network-plugin Set network plugin, default is null, suppport null and cni -p, --pidfile Save pid into this file --pod-sandbox-image The image whose network/ipc namespaces containers in each pod will use. (default "rnd-dockerhub.huawei.com/library/pause-${machine}:3.0") --registry-mirrors Registry to be prepended when pulling unqualified images, can be specified multiple times --start-timeout timeout duration for waiting on a container to start before it is killed -S, --state Root directory for execution state files --storage-driver Storage driver to use(default overlay2) -s, --storage-opt Storage driver options --tls Use TLS; implied by --tlsverify --tlscacert Trust certs signed only by this CA (default "/root/.iSulad/ca.pem") --tlscert Path to TLS certificate file (default "/root/.iSulad/cert.pem") --tlskey Path to TLS key file (default "/root/.iSulad/key.pem") --tlsverify Use TLS and verify the remote --use-decrypted-key Use decrypted private key by default(default true) -V, --version Print the version --websocket-server-listening-port CRI websocket streaming service listening port (default 10350)Example: Start iSulad and change the log level to DEBUG.
# isulad -l DEBUGConfiguration file
The iSulad configuration file is /etc/isulad/daemon.json*. The parameters in the file are described as follows.
Example:
# cat /etc/isulad/daemon.json { "group": "isulad", "default-runtime": "lcr", "graph": "/var/lib/isulad", "state": "/var/run/isulad", "engine": "lcr", "log-level": "ERROR", "pidfile": "/var/run/isulad.pid", "log-opts": { "log-file-mode": "0600", "log-path": "/var/lib/isulad", "max-file": "1", "max-size": "30KB" }, "log-driver": "stdout", "hook-spec": "/etc/default/isulad/hooks/default.json", "start-timeout": "2m", "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ], "registry-mirrors": [ "docker.io" ], "insecure-registries": [ "rnd-dockerhub.huawei.com" ], "pod-sandbox-image": "", "image-opt-timeout": "5m", "native.umask": "secure", "network-plugin": "", "cni-bin-dir": "", "cni-conf-dir": "", "image-layer-check": false, "use-decrypted-key": true, "insecure-skip-verify-enforce": false }
NOTICE:
The default configuration file /etc/isulad/daemon.json is for reference only. Configure it based on site requirements.
Storage Description
Constraints
In high concurrency scenarios (200 containers are concurrently started), the memory management mechanism of Glibc may cause memory holes and large virtual memory (for example, 10 GB). This problem is caused by the restriction of the Glibc memory management mechanism in the high concurrency scenario, but not by memory leakage. Therefore, the memory consumption does not increase infinitely. You can set MALLOC_ARENA_MAX to reducevirtual memory error and increase the rate of reducing physical memory. However, this environment variable will cause the iSulad concurrency performance to deteriorate. Set this environment variable based on the site requirements.
To balance performance and memory usage, set MALLOC_ARENA_MAX to 4. (The iSulad performance on the ARM64 server is affected by less than 10%.) Configuration method: 1. To manually start iSulad, run the export MALLOC_ARENA_MAX=4 command and then start iSulad. 2. If systemd manages iSulad, you can modify the /etc/sysconfig/iSulad file by adding MALLOC_ARENA_MAX=4.Precautions for specifying the daemon running directories
Take --root as an example. When /new/path/ is used as the daemon new root directory, if a file exists in /new/path/ and the directory or file name conflicts with that required by iSulad (for example, engines and mnt), iSulad may update the original directory or file attributes including the owner and permission.
Therefore, please note the impact of re-specifying various running directories and files on their attributes. You are advised to specify a new directory or file for iSulad to avoid file attribute changes and security issues caused by conflicts.
Log file management:
NOTICE:
Log function interconnection: logs are managed by systemd as iSulad is and then transmitted to rsyslogd. By default, rsyslog restricts the log writing speed. You can add the configuration item $imjournalRatelimitInterval 0 to the /etc/rsyslog.conf file and restart the rsyslogd service.Restrictions on command line parameter parsing
When the iSulad command line interface is used, the parameter parsing mode is slightly different from that of Docker. For flags with parameters in the command line, regardless of whether a long or short flag is used, only the first space after the flag or the character string after the equal sign (=) directly connected to the flag is used as the flag parameter. The details are as follows:
When a short flag is used, each character in the character string connected to the hyphen (-) is considered as a short flag. If there is an equal sign (=), the character string following the equal sign (=) is considered as the parameter of the short flag before the equal sign (=).
isula run -du=root busybox is equivalent to isula run -du root busybox, isula run -d -u=root busybox, or isula run -d -u root busybox. When isula run -du:root is used, as -: is not a valid short flag, an error is reported. The preceding command is equivalent to isula run -ud root busybox. However, this method is not recommended because it may cause semantic problems.
When a long flag is used, the character string connected to -- is regarded as a long flag. If the character string contains an equal sign (=), the character string before the equal sign (=) is a long flag, and the character string after the equal sign (=) is a parameter.
isula run --user=root busyboxor
isula run --user root busybox
After an iSulad container is started, you cannot run the isula run -i/-t/-ti and isula attach/exec commands as a non-root user.
When iSulad connects to an OCI container, only kata-runtime can be used to start the OCI container.
Daemon Multi-Port Binding
Description
The daemon can bind multiple UNIX sockets or TCP ports and listen on these ports. The client can interact with the daemon through these ports.
Port
Users can configure one or more ports in the hosts field in the /etc/isulad/daemon.json file, or choose not to specify hosts.
{
"hosts": [
"unix:///var/run/isulad.sock",
"tcp://localhost:5678",
"tcp://127.0.0.1:6789"
]
}
Users can also run the -H or --host command in the /etc/sysconfig/iSulad file to configure a port, or choose not to specify hosts.
OPTIONS='-H unix:///var/run/isulad.sock --host tcp://127.0.0.1:6789'
If hosts are not specified in the daemon.json file and iSulad, the daemon listens on unix:///var/run/isulad.sock by default after startup.
Restrictions
Users cannot specify hosts in the /etc/isulad/daemon.json and /etc/sysconfig/iSulad files at the same time. Otherwise, an error will occur and iSulad cannot be started.
unable to configure the isulad with file /etc/isulad/daemon.json: the following directives are specified both as a flag and in the configuration file: hosts: (from flag: [unix:///var/run/isulad.sock tcp://127.0.0.1:6789], from file: [unix:///var/run/isulad.sock tcp://localhost:5678 tcp://127.0.0.1:6789])If the specified host is a UNIX socket, the socket must start with unix:// followed by a valid absolute path.
If the specified host is a TCP port, the TCP port number must start with tcp:// followed by a valid IP address and port number. The IP address can be that of the local host.
A maximum of 10 valid ports can be specified. If more than 10 ports are specified, an error will occur and iSulad cannot be started.
Configuring TLS Authentication and Enabling Remote Access
Description
iSulad is designed in C/S mode. By default, the iSulad daemon process listens only on the local/var/run/isulad.sock. Therefore, you can run commands to operate containers only on the local client iSula. To enable iSula's remote access to the container, the iSulad daemon process needs to listen on the remote access port using TCP/IP. However, listening is performed only by simply configuring tcp ip:port. In this case, all IP addresses can communicate with iSulad by calling isula -H tcp://remote server IP address:port, which may cause security problems. Therefore, it is recommended that a more secure version, namely Transport Layer Security (TLS), be used for remote access.
Generating TLS Certificate
Example of generating a plaintext private key and certificate
#!/bin/bash set -e echo -n "Enter pass phrase:" read password echo -n "Enter public network ip:" read publicip echo -n "Enter host:" read HOST echo " => Using hostname: $publicip, You MUST connect to iSulad using this host!" mkdir -p $HOME/.iSulad cd $HOME/.iSulad rm -rf $HOME/.iSulad/* echo " => Generating CA key" openssl genrsa -passout pass:$password -aes256 -out ca-key.pem 4096 echo " => Generating CA certificate" openssl req -passin pass:$password -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem -subj "/C=CN/ST=zhejiang/L=hangzhou/O=Huawei/OU=iSulad/CN=iSulad@huawei.com" echo " => Generating server key" openssl genrsa -passout pass:$password -out server-key.pem 4096 echo " => Generating server CSR" openssl req -passin pass:$password -subj /CN=$HOST -sha256 -new -key server-key.pem -out server.csr echo subjectAltName = DNS:$HOST,IP:$publicip,IP:127.0.0.1 >> extfile.cnf echo extendedKeyUsage = serverAuth >> extfile.cnf echo " => Signing server CSR with CA" openssl x509 -req -passin pass:$password -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile.cnf echo " => Generating client key" openssl genrsa -passout pass:$password -out key.pem 4096 echo " => Generating client CSR" openssl req -passin pass:$password -subj '/CN=client' -new -key key.pem -out client.csr echo " => Creating extended key usage" echo extendedKeyUsage = clientAuth > extfile-client.cnf echo " => Signing client CSR with CA" openssl x509 -req -passin pass:$password -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cert.pem -extfile extfile-client.cnf rm -v client.csr server.csr extfile.cnf extfile-client.cnf chmod -v 0400 ca-key.pem key.pem server-key.pem chmod -v 0444 ca.pem server-cert.pem cert.pemExample of generating an encrypted private key and certificate request file
#!/bin/bash echo -n "Enter public network ip:" read publicip echo -n "Enter pass phrase:" read password # remove certificates from previous execution. rm -f *.pem *.srl *.csr *.cnf # generate CA private and public keys echo 01 > ca.srl openssl genrsa -aes256 -out ca-key.pem -passout pass:$password 2048 openssl req -subj '/C=CN/ST=zhejiang/L=hangzhou/O=Huawei/OU=iSulad/CN=iSulad@huawei.com' -new -x509 -days $DAYS -passin pass:$password -key ca-key.pem -out ca.pem # create a server key and certificate signing request (CSR) openssl genrsa -aes256 -out server-key.pem -passout pass:$PASS 2048 openssl req -new -key server-key.pem -out server.csr -passin pass:$password -subj '/CN=iSulad' echo subjectAltName = DNS:iSulad,IP:${publicip},IP:127.0.0.1 > extfile.cnf echo extendedKeyUsage = serverAuth >> extfile.cnf # sign the server key with our CA openssl x509 -req -days $DAYS -passin pass:$password -in server.csr -CA ca.pem -CAkey ca-key.pem -out server-cert.pem -extfile extfile.cnf # create a client key and certificate signing request (CSR) openssl genrsa -aes256 -out key.pem -passout pass:$password 2048 openssl req -subj '/CN=client' -new -key key.pem -out client.csr -passin pass:$password # create an extensions config file and sign echo extendedKeyUsage = clientAuth > extfile.cnf openssl x509 -req -days 365 -passin pass:$password -in client.csr -CA ca.pem -CAkey ca-key.pem -out cert.pem -extfile extfile.cnf # remove the passphrase from the client and server key openssl rsa -in server-key.pem -out server-key.pem -passin pass:$password openssl rsa -in key.pem -out key.pem -passin pass:$password # remove generated files that are no longer required rm -f ca-key.pem ca.srl client.csr extfile.cnf server.csr
APIs
{
"tls": true,
"tls-verify": true,
"tls-config": {
"CAFile": "/root/.iSulad/ca.pem",
"CertFile": "/root/.iSulad/server-cert.pem",
"KeyFile":"/root/.iSulad/server-key.pem"
}
}
Restrictions
The server supports the following modes:
- Mode 1 (client verified): tlsverify, tlscacert, tlscert, tlskey
- Mode 2 (client not verified): tls, tlscert, tlskey
The client supports the following modes:
- Mode 1 (verify the identity based on the client certificate, and verify the server based on the specified CA): tlsverify, tlscacert, tlscert, tlskey
- Mode 2 (server verified): tlsverify, tlscacert
Mode 1 is used for the server, and mode 2 for the client if the two-way authentication mode is used for communication.
Mode 2 is used for the server and the client if the unidirectional authentication mode is used for communication.
- If RPM is used for installation, the server configuration can be modified in the /etc/isulad/daemon.json and /etc/sysconfig/iSulad files.
- Two-way authentification is recommended as it is more secure than non-authentication or unidirectional authentication.
- GRPC open-source component logs are not taken over by iSulad. To view gRPC logs, set the environment variables gRPC_VERBOSITY and gRPC_TRACE as required.
Example
On the server:
isulad -H=tcp://0.0.0.0:2376 --tlsverify --tlscacert ~/.iSulad/ca.pem --tlscert ~/.iSulad/server-cert.pem --tlskey ~/.iSulad/server-key.pem
On the client:
isula version -H=tcp://$HOSTIP:2376 --tlsverify --tlscacert ~/.iSulad/ca.pem --tlscert ~/.iSulad/cert.pem --tlskey ~/.iSulad/key.pem
devicemapper Storage Driver Configuration
To use the devicemapper storage driver, you need to configure a thinpool device which requires an independent block device with sufficient free space. Take the independent block device /dev/xvdf as an example. The configuration method is as follows:
Configuring a thinpool
Stop the iSulad service.
# systemctl stop isuladCreate a logical volume manager (LVM) volume based on the block device.
# pvcreate /dev/xvdfCreate a volume group based on the created physical volume.
# vgcreate isula /dev/xvdf Volume group "isula" successfully created:Create two logical volumes named thinpool and thinpoolmeta.
# lvcreate --wipesignatures y -n thinpool isula -l 95%VG Logical volume "thinpool" created.# lvcreate --wipesignatures y -n thinpoolmeta isula -l 1%VG Logical volume "thinpoolmeta" created.Convert the two logical volumes into a thinpool and the metadata used by the thinpool.
# lvconvert -y --zero n -c 512K --thinpool isula/thinpool --poolmetadata isula/thinpoolmeta WARNING: Converting logical volume isula/thinpool and isula/thinpoolmeta to thin pool's data and metadata volumes with metadata wiping. THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.) Converted isula/thinpool to thin pool.
Modifying the iSulad configuration files
If iSulad has been used in the environment, back up the running data first.
# mkdir /var/lib/isulad.bk # mv /var/lib/isulad/* /var/lib/isulad.bkModify configuration files.
Two configuration methods are provided. Select one based on site requirements.
Edit the /etc/isulad/daemon.json file, set storage-driver to devicemapper, and set parameters related to the storage-opts field. For details about related parameters, see Parameter Description. The following lists the configuration reference:
{ "storage-driver": "devicemapper" "storage-opts": [ "dm.thinpooldev=/dev/mapper/isula-thinpool", "dm.fs=ext4", "dm.min_free_space=10%" ] }You can also edit /etc/sysconfig/iSulad to explicitly specify related iSulad startup parameters. For details about related parameters, see Parameter Description. The following lists the configuration reference:
OPTIONS="--storage-driver=devicemapper --storage-opt dm.thinpooldev=/dev/mapper/isula-thinpool --storage-opt dm.fs=ext4 --storage-opt dm.min_free_space=10%"
Start iSulad for the settings to take effect.
# systemctl start isulad
Parameter Description
For details about parameters supported by storage-opts, see Table 1.
Table 1 Parameter description
Precautions
When configuring devicemapper, if the system does not have sufficient space for automatic capacity expansion of thinpool, disable the automatic capacity expansion function.
To disable automatic capacity expansion, set both thin_pool_autoextend_threshold and thin_pool_autoextend_percent in the /etc/lvm/profile/isula-thinpool.profile file to 100.
activation { thin_pool_autoextend_threshold=100 thin_pool_autoextend_percent=100 }When devicemapper is used, use Ext4 as the container file system. You need to add --storage-opt dm.fs=ext4 to the iSulad configuration parameters.
If graphdriver is devicemapper and the metadata files are damaged and cannot be restored, you need to manually restore the metadata files. Do not directly operate or tamper with metadata of the devicemapper storage driver in Docker daemon.
When the devicemapper LVM is used, if the devicemapper thinpool is damaged due to abnormal power-off, you cannot ensure the data integrity or whether the damaged thinpool can be restored. Therefore, you need to rebuild the thinpool.
Precautions for Switching the devicemapper Storage Pool When the User Namespace Feature Is Enabled on iSula
- Generally, the path of the deviceset-metadata file is /var/lib/isulad/devicemapper/metadata/deviceset-metadata during container startup.
- If user namespaces are used, the path of the deviceset-metadata file is /var/lib/isulad/userNSUID.GID/devicemapper/metadata/deviceset-metadata.
- When you use the devicemapper storage driver and the container is switched between the user namespace scenario and common scenario, the BaseDeviceUUID content in the corresponding deviceset-metadata file needs to be cleared. In the thinpool capacity expansion or rebuild scenario, you also need to clear the BaseDeviceUUID content in the deviceset-metadata file. Otherwise, the iSulad service fails to be restarted.
