Accelerating Redis with Gazelle

Background

Gazelle is a high-performance user-mode protocol stack. Based on DPDK, it directly reads and writes NIC packets in user mode, transfers packets by sharing huge page memory, and uses a lightweight LwIP protocol stack. It can greatly improve the network I/O throughput of applications. It focuses on database network performance acceleration, such as MySQL and Redis.

Compared with the kernel protocol stack, Gazelle achieves a significant improvement in the Redis test. Taking the set and get tests of an Arm-architecture 8U32G VM with OVS + DPDK as an example, the test results are as follows: set is improved by about 1.7 times, and get is improved by about 1.5 times. The number of test connections is 1,000, and the packet length is the default packet length 3.

#kernel
redis-benchmark -h 192.168.1.127 -p 6379 -c 1000 -n 10000000 -r 10000000 -t set,get --threads 12
#set
Summary:
  throughput summary: 132402.98 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        7.474     1.376     7.207     9.399    14.255    30.879
#get
Summary:
  throughput summary: 142834.69 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        6.919     1.384     6.663     8.751    13.311    24.207

#gazelle
redis-benchmark -h 192.168.1.127 -p 6379 -c 1000 -n 10000000 -r 10000000 -t set,get --threads 12
#set
Summary:
  throughput summary: 359389.03 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        2.736     0.240     2.735     2.895     3.127     9.471
#get
Summary:
  throughput summary: 359401.97 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        2.752     0.488     2.751     2.903     3.135    16.671

NOTE

Different hardware configurations and networking environments affect the performance improvement. The preceding test results are for reference only.

Feature Constraints

Currently, only IPv4 is supported, and IPv6 is not supported.
The maximum number of concurrent connections is 20,000.
Currently, Gazelle multi-process is not supported, that is, multiple Redis servers cannot be started using Gazelle on one node.
In a Redis master-slave cluster, the kernel mode repeatedly prints the master-slave reconnection log, while the user mode prints the log only once. This is normal and has no functional impact. Cause: After a kernel-mode master node goes offline, the slave node attempts to connect to the master node and sends a SYN packet. The master node is in the TIME_WAIT state and replies with an RST packet to close the connection. This repeats. After a user-mode master node goes offline, because the connection is completely closed after gazelle+redis exits and is not in the TIME_WAIT state, the slave node attempts to connect to the master node and sends a SYN packet. The master node does not reply with an RST packet. Therefore, when you run gazellectl lstack show ip -c to query information, there is a connection in the SYN_SENT state.

Gazelle-Accelerated Redis Test Procedure

Environment Requirements

Hardware

The standalone test requires one server and one client.
The master-slave mode test requires at least two servers (one master and one slave) and one client.
The sentinel mode test requires at least two Redis servers (one master and one slave), two sentinel servers, and one client.
The cluster mode test requires at least six Redis servers (three masters and three slaves) and one client.

Software

Redis software package download. The version used in this test is redis-6.2.9.

Server Deployment

Disabling Test-Impacting Items

# Disable the firewall.
systemctl stop iptables
systemctl stop firewalld

Compiling and Installing Redis

tar zxvf redis-6.2.9.tar.gz 
cd redis-6.2.9/
make clean
make -j 32
make install

Deploying the Gazelle Runtime Environment

Install Gazelle and its dependencies.

yum -y install gazelle dpdk libconfig numactl libboundscheck libcap

Modify the parameters in the /etc/gazelle/lstack.conf configuration file as follows. Other configuration parameters can retain their default values.

Configuration Item	Value	Description
dpdk_args	["--socket-mem", "2400,0,0,0", "--huge-dir", "/mnt/hugepages-lstack", "--proc-type", "primary"]	Configures 2400 MB of memory for the NUMA where the CPU and NIC are located (which can be reduced based on the number of concurrent connections). If the CPU and NIC are not on the same NUMA, memory must be configured for both NUMAs. If the NIC is an mlx NIC, "-d" and "librte_net_mlx5.so" must be appended.
num_cpus	"2"	Selects one CPU to bind lstack.
mbuf_count_per_conn	34	Number of mbufs required for each connection.
tcp_conn_count	20000	Maximum number of concurrent connections in the Redis test.

# Allocate huge pages on the server.
mkdir -p /mnt/hugepages-lstack
chmod -R 700 /mnt/hugepages-lstack
mount -t hugetlbfs nodev /mnt/hugepages-lstack -o pagesize=2M # Do not perform this operation repeatedly. Otherwise, the huge pages are occupied and cannot be released.
echo 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages # Select the page size based on the actual situation.
cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/free_hugepages # Query the actually available huge page memory on the corresponding node.

# Load the ko on the server. (Skip this step for mlx NICs.)
modprobe vfio enable_unsafe_noiommu_mode=1
modprobe vfio-pci

# Bind the NIC to user mode on the server. (Skip this step for mlx NICs.)
ip link set enp4s0 down
dpdk-devbind -b vfio-pci enp4s0

# Gazelle deployment is complete. Wait for app deployment.

NOTE

For details about Gazelle deployment, see Gazelle User Guide.
For details about how to bind different NICs in user mode, see Gazelle NIC Support and Usage.

Deploying the Redis Server

Redis environment deployment includes standalone deployment, master-slave mode deployment, sentinel mode deployment, and cluster mode deployment. In all scenarios, the redis.conf file of the Redis server must be configured as follows:

# Disable protected mode.
protected-mode no
# Gazelle does not support this parameter for running in the background.
daemonize no
# Enable AOF persistence. This parameter is optional for the Redis standalone test.
appendonly yes

Redis Standalone Deployment

The Redis standalone test includes one server. After Gazelle and Redis are deployed, you can directly start the Gazelle+Redis service.

LD_PRELOAD=/usr/lib64/liblstack.so GAZELLE_BIND_PROCNAME=redis-server redis-server /root/redis-6.2.9/redis.conf

Redis Master-Slave Mode Deployment

Master-slave replication means copying the data of one Redis server to other Redis servers. The former is called the master node, and the latter is called the slave node. Data replication is one-way, from the master node to the slave node. The Redis master-slave mode includes at least two servers. There are two configuration methods:

Master-slave configuration method 1
Add the following configuration to the redis.conf file of the slave node, and then start the Redis master and slave nodes respectively.

#192.168.1.127 and 6379 are the IP address and port of the master node service.
slaveof 192.168.1.127 6379

LD_PRELOAD=/usr/lib64/liblstack.so GAZELLE_BIND_PROCNAME=redis-server redis-server /root/redis-6.2.9/redis.conf

Master-slave configuration method 2
After modifying the common configuration of the master and slave nodes, start the Redis master and slave nodes (the master-slave relationship has not been established yet), and run the following commands on the client.

redis-cli -h 192.168.1.127 slaveof NO ONE # Master node
redis-cli -h 192.168.1.128 slaveof 192.168.1.127 6379 # Slave node

Query the master-slave information.

[root@openEuler redis-6.2.9]# redis-cli -h 192.168.1.127 info Replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.1.128,port=6379,state=online,offset=780,lag=0
......

[root@openEuler redis-6.2.9]# redis-cli -h 192.168.1.128 info Replication
# Replication
role:slave
master_host:192.168.1.127
master_port:6379
master_link_status:up
......

Redis Sentinel Mode Deployment

The sentinel mode is based on the master-slave replication mode, except that sentinels are introduced to monitor and automatically handle faults. The master-slave switchover technology works as follows: After a server breaks down, a slave server needs to be manually switched to the master server. This requires manual intervention, which is time-consuming and labor-intensive and causes service unavailability for a period of time. The sentinel mechanism is introduced to resolve the disadvantages of master-slave replication. The Redis sentinel mode test requires at least two Redis servers and two sentinel servers.

Start the two Redis servers by following the master-slave mode deployment method.
Install and deploy Redis on the two sentinel servers, and modify the sentinel.conf configuration file on each server.

protected-mode no # Disable protected mode.
daemonize yes # Run in the background, and record logs in logfile.
logfile "/var/log/sentinel.log" # Specify the log storage path.
sentinel monitor mymaster 192.168.1.127 6379 1 # The name of the master node is mymaster. Monitor the IP address and port of the master. The value 1 indicates that at least one sentinel node must agree before the master node is determined to be faulty and a failover is performed.
sentinel down-after-milliseconds mymaster 30000 # Time period for determining that the server is down. The default value is 30000 milliseconds (30 seconds).
sentinel failover-timeout mymaster 50000 # Maximum timeout period of the faulty node, which is 50000.

Start the sentinels (in kernel mode) and query the sentinel information.

[root@openEuler redis-6.2.9]#redis-sentinel sentinel.conf
[root@openEuler redis-6.2.9]#ps -ef|grep redis-sentinel 
root       5961      1  0 13:36 ?        00:00:00 redis-sentinel *:26379 [sentinel]
[root@openEuler redis-6.2.9]#redis-cli -p 26379 info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.1.127:6379,slaves=2,sentinels=3

NOTE

The Redis server and the Redis sentinel cannot be deployed on the same node. Otherwise, the master/standby switchover cannot be performed properly.
Redis Sentinel does not support user-mode startup currently.

Redis Cluster Mode Deployment

The concurrency capability of a single-node Redis is limited. To further improve the concurrency capability of Redis, you need to set up a master-slave cluster, which provides a program set for sharing data among multiple Redis nodes. The Redis cluster test requires at least six Redis servers.

Install and deploy Redis on the six Redis servers, and modify the redis.conf configuration file on each server.

protected-mode no # Disable protected mode.
daemonize no # Run in the foreground.
bind 0.0.0.0
port 6379 # Redis is deployed on different VMs with different IP addresses, so the port can retain the default value.
appendonly yes # Enable AOF persistence.
cluster-enabled yes # Enable the cluster mode.
cluster-config-file nodes.conf # Name of the configuration file in cluster mode. It does not need to be manually created and is automatically maintained by the cluster.
cluster-node-timeout 5000 # Heartbeat timeout period between nodes in the cluster.

Start the six Redis servers respectively.

LD_PRELOAD=/usr/lib64/liblstack.so GAZELLE_BIND_PROCNAME=redis-server redis-server /root/redis-6.2.9/redis.conf

Run commands on the client to create a cluster and confirm the allocation of master and slave nodes in the cluster.

[root@openEuler redis-6.2.9]#redis-cli --cluster create --cluster-replicas 1 192.168.1.127:6379 192.168.1.128:6379 192.168.1.129:6379 192.168.1.130:6379 192.168.1.131:6379 192.168.1.132:6379
>>> Performing hash slots allocation on 6 nodes...
......
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...
>>> Performing Cluster Check (using node 192.168.1.127:6379)
......
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered. # If the hash slots are allocated successfully, the cluster is created successfully.

#redis-cli --cluster: cluster operation command. create: create a cluster. --cluster-replicas 1: specify the number of replicas of each master in the cluster as 1.
# In this case, the total number of nodes ÷ (replicas + 1) is the number n of masters. Therefore, the first n nodes in the node list are masters, and the other nodes are slave nodes, which are randomly allocated to different masters.

Query the cluster information.

Any running server in the cluster can be used as the entry point.

# View the cluster status information.
[root@openEuler redis-6.2.9]# redis-cli -h 192.168.1.127 cluster info
cluster_state:ok # If the value is fail, check whether the hash slots fail to be allocated.
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
......

# View the master-slave relationship of the cluster.
[root@openEuler redis-6.2.9]# redis-cli -h 192.168.1.128 cluster nodes
514b35aaa5035d489b60a0e8f8fb01d1c20734ce 192.168.1.129:6379@16379 slave 50aa44a1e4a6a0c75cf2f9b20055bfaa77d1b163 0 1724919619916 1 connected
50aa44a1e4a6a0c75cf2f9b20055bfaa77d1b163 192.168.1.127:6379@16379 master - 0 1724919617960 1 connected 0-5460
a94402ca747ead08e4b93ff975dfbe995068ecbf 192.168.1.130:6379@16379 slave 0a569e1ac4e373a22abcbf6ce6b8118fba3d4d6e 0 1724919618000 3 connected
8c4040a4fa8456044acad2518dc45b8236ba44c4 192.168.1.128:6379@16379 myself,slave 44a96161651c8383fb4966c6dde45d400fe2a203 0 1724919617000 2 connected
0a569e1ac4e373a22abcbf6ce6b8118fba3d4d6e 192.168.1.132:6379@16379 master - 0 1724919618975 3 connected 10923-16383
44a96161651c8383fb4966c6dde45d400fe2a203 192.168.1.131:6379@16379 master - 0 1724919619617 2 connected 5461-10922
# You can see that 127, 131, and 132 are master nodes and 128, 129, and 130 are slave nodes. You can find the corresponding master node based on the node ID following slave.

redis-benchmark Tool Deployment on the Client

Compile and install

redis-benchmark is a test tool provided with Redis. You can compile and install Redis in the same way as on the server.

Test commands

# Standalone, master-slave, and sentinel modes
#set,get
redis-benchmark -h 192.168.1.127 -p 6379 -c 1000 -n 10000000 -d 3 -r 10000000 -t set,get --threads 12
# -h: specify the Redis server address; -p: specify the Redis server port; -c: specify the number of concurrent client connections; -n: specify the total number of requests; -t: specify the test command; -d: specify the packet size.
# In master-slave mode and sentinel mode, because slave-read-only yes is configured by default, the slave node Redis can only execute the get command.

# Cluster mode
#set,get
redis-benchmark -h 192.168.1.127 -p 6379 -c 1000 -n 10000000 -d 3 -r 10000000 -t set,get --threads 12 --cluster
#-h only needs to specify any available node in the cluster.

Sample Logs of Successful Gazelle-Accelerated Redis

[root@openEuler redis-6.2.9]# LD_PRELOAD=/usr/lib64/liblstack.so GAZELLE_BIND_PROCNAME=redis-server redis-server redis.conf
LSTACK[2555459]: LD_PRELOAD ok
dpdk argv: --socket-mem 2400,0,0,0 --huge-dir /mnt/hugepages-lstack --proc-type primary
LSTACK[2555459]: cfg_init success
# DPDK startup logs are omitted, which vary with environments.
......
# If Gazelle prints init success, it is started successfully.
LSTACK: gazelle_stack_thread:533 stack_00 init success
LSTACK: gazelle_network_init:328 gazelle_network_init success
# Redis startup logs are omitted, which are the same as those of kernel-mode Redis startup.

After the startup is successful, you can ping the user-mode IP address configured in lstack.conf on the client. If the ping succeeds, Gazelle-accelerated Redis is deployed successfully.

Accelerating Redis with Gazelle ​

Background ​

Feature Constraints ​

Gazelle-Accelerated Redis Test Procedure ​

Environment Requirements ​

Hardware ​

Software ​

Server Deployment ​

Disabling Test-Impacting Items ​

Compiling and Installing Redis ​

Deploying the Gazelle Runtime Environment ​

Deploying the Redis Server ​

Redis Standalone Deployment ​

Redis Master-Slave Mode Deployment ​

Redis Sentinel Mode Deployment ​

Redis Cluster Mode Deployment ​

redis-benchmark Tool Deployment on the Client ​

Sample Logs of Successful Gazelle-Accelerated Redis ​

Accelerating Redis with Gazelle

Background

Feature Constraints

Gazelle-Accelerated Redis Test Procedure

Environment Requirements

Hardware

Software

Server Deployment

Disabling Test-Impacting Items

Compiling and Installing Redis

Deploying the Gazelle Runtime Environment

Deploying the Redis Server

Redis Standalone Deployment

Redis Master-Slave Mode Deployment

Redis Sentinel Mode Deployment

Redis Cluster Mode Deployment

redis-benchmark Tool Deployment on the Client

Sample Logs of Successful Gazelle-Accelerated Redis