Server

Version: 22.03 LTS SP4
Document Center
Server
Accelerating openGauss with Gazelle

Accelerating openGauss with Gazelle ​

Background ​

openGauss is a high-performance database. As a high-performance user-mode protocol stack, Gazelle can greatly improve the network I/O throughput of applications. Accelerating openGauss with Gazelle can effectively improve openGauss performance. In the openGauss standalone scenario, Gazelle improves performance by 15%. In the active/standby scenario, Gazelle improves performance by 5%.

Feature Constraints ​

  • The openGauss version used in the current verification is openGauss 6.0.0 (LTS). Other versions have not been verified and may not be supported.
  • Currently, Gazelle has been verified to enable acceleration in the openGauss standalone and active/standby scenarios. Other scenarios have not been verified and may not be supported.
  • Gazelle does not support port reuse: If a port is used on the IP address of a user-mode protocol stack, other IP addresses cannot use this port.

Setting Up the User-Mode Environment ​

Prerequisites ​

  1. The database has been installed and the kernel-mode database has been started properly. For details about the kernel mode, see the openGauss official website.

  2. Install Gazelle and its dependencies.

     ```sh
 yum -y install gazelle dpdk libconfig numactl libboundscheck libcap 
 ```

  3. For details about how to install and use benchmark, see Using BenchmarkSQL

Gazelle Environment Configuration ​

Run the following command to view the supported huge page sizes.

shell
ll /sys/kernel/mm/hugepages/

Generally, each NUMA node requires 2-3 GB of huge page memory. There are usually 4 NUMA nodes, which require 8-12 GB of huge page memory. Set the number of huge pages based on the supported huge page size.

For example, to add hugepages-1048576kB (1 GB) huge pages, 20 huge pages are selected. The corresponding memory is 20 GB, which is sufficient for allocation. The number must be a multiple of 4. Do not set this value too large, because an excessively large value occupies too much operating system memory.

shell
echo 20 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

# Create a local disk mount after the configuration is complete.
mkdir /mnt/hugepages-1G
mount -t hugetlbfs -o pagesize=1G nodev /mnt/hugepages-1G

# Load the ko.
modprobe vfio enable_unsafe_noiommu_mode=1
modprobe vfio-pci

NOTE

For details about Gazelle deployment, see Gazelle User Guide.
For details about how to bind different NICs in user mode, see Gazelle NIC Support and Usage.

lstack.conf File Configuration ​

Use the corresponding database user to write lstack.conf, that is, ensure that the user has the access permission on the file.

Descriptions of several key configuration items and the modification points:

  • dpdk_args

3096,3096,3096,3096 is the memory size set for each NUMA node. In this example, 3 GB is set.

Change /mnt/hugepages-1G to the path set in the huge page environment configuration.

  • num_cpus

30,31,62,63,94,95,126,127: network interrupt cores set for lstack. In this example, the last two CPU IDs of each NUMA node are set. A total of 8 interrupts are used.

  • app_bind_numa

This parameter determines whether to bind the epoll/poll threads to the corresponding NUMA node. Set it to 0, that is, do not bind. The Gazelle core binding policy conflicts with the Gauss core binding policy. Use the Gauss core binding policy here.

  • host_addr, mask_addr, gateway_addr, devices

Obtain the required NIC IP information. Generally, you can run the ip addr show command to obtain the required information. gateway_addr is generally the IP address with the last octet changed to 1. You can run route -n to view the gateway. NIC information to be configured:

host_addr: 20.20.20.119

mask_addr: 255.255.255.0

gateway_addr: 20.20.20.1

devices: 78:b4:6a:40:16:30

NIC information captured after the ip addr show command is executed:

sh
 8: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
     link/ether 78:b4:6a:40:16:30 brd ff:ff:ff:ff:ff:ff
     inet 20.20.20.149/24 brd 20.20.20.255 scope global enp7s0
        valid_lft forever preferred_lft forever
  • app_exclude_cpus

Excluded CPU IDs. openGauss uses CPU0 as the xlog write thread. Use app_exclude_cpus="0" to exclude CPU0 from core binding.

  • listen_shadow

Whether to use shadow fd listening. 0 indicates disabled, and 1 indicates enabled. This parameter is enabled when there is a single listen thread and multiple protocol stack threads. It needs to be enabled here.

Configuration file example:

shell
dpdk_args=["--socket-mem", "3096,3096,3096,3096", "--huge-dir", "/mnt/hugepages-1G", "--proc-type", "primary", "--legacy-mem"]

use_ltran=0
kni_switch=0

low_power_mode=0

#needed mbuf count = tcp_conn_count * mbuf_count_per_conn
tcp_conn_count = 1500
mbuf_count_per_conn = 350

# send ring size, default is 32, max is 2048
send_ring_size = 32

# 0: when send ring full, send return
# 1: when send ring full, alloc mbuf from mempool to send data
expand_send_ring = 0

#protocol stack thread per loop params
#read data form protocol stack into recv_ring
read_connect_number = 4
#process rpc msg number
rpc_number = 4
#read nic pkts number
nic_read_number = 128

#each cpu core start a protocol stack thread.
num_cpus="30,31,62,63,94,95,126,127"

#app worker thread bind to numa in epoll/poll.
app_bind_numa=0
#app main thread affinity set by dpdk.
main_thread_affinity=0

host_addr="20.20.20.119"
mask_addr="255.255.255.0"
gateway_addr="20.20.20.1"
devices="78:b4:6a:40:16:31"

udp_enable=0
#0: use rss rule
#1: use tcp tuple rule to specify packet to nic queue
tuple_filter=0

#tuple_filter=1, below cfg valid
num_process=1
process_numa="0,1"
process_idx=0

#tuple_filer=0, below cfg valid
listen_shadow=1
#exclude cpu
app_exclude_cpus="0"

Granting Permissions ​

Grant permissions to the gaussdb binary, liblstack.so, and the ls command.

shell
#Grant permissions to the gaussdb binary.
sudo setcap 'cap_chown,cap_dac_override,cap_dac_read_search,cap_sys_rawio,cap_net_admin,cap_net_raw,cap_sys_admin+eip' `which gaussdb`
#Grant permissions to liblstack.so.
sudo chmod u+s /usr/lib64/liblstack.so
#Grant permissions to the ls command.
setcap 'cap_chown,cap_dac_override,cap_dac_read_search,cap_sys_rawio,cap_net_admin,cap_net_raw,cap_sys_admin+eip' $(which ls)

The gaussdb binary to which permissions have been granted no longer searches for dynamic library paths in LD_LIBRARY_PATH. Therefore, you also need to configure the global dynamic library path: vim /etc/ld.so.conf.d/opengauss.conf. Write the actual dynamic library path content.

shell
/home/code/openGauss-server/dest/lib

Required Path Permissions ​

You need to change the permissions of the file path corresponding to the huge page memory and the permissions of liblstack.so to allow user access.

shell
chown -R 777 /mnt/
chmod 777 /usr/lib64/liblstack.so

You also need to ensure that the written lstack.conf file is also accessible to users.

Database Test ​

Preparation of Required Information ​

First, the database test requires at least one NVMe disk for data storage.

The NIC used in the local test is a HINIC NIC.

You can query some information using the following commands. Assume that the NIC name is enp8s0.

shell
# Query the PCI information:
 ethtool -i enp8s0 | grep bus-info

NIC Switchover to User Mode ​

This operation will make the NIC unavailable. Operate it on a BMC machine or in an environment with two NIC ports. Otherwise, the environment becomes unavailable. Use the root user to switch the NIC to a user-mode NIC. This operation stops the NIC and takes it over using DPDK. If an SSH connection is established using this NIC, the connection is directly disconnected.

shell
#PCI=0000:08:00.0
#NET_DEVICE=enp8s0
ifconfig enp8s0 down
sudo modprobe vfio enable_unsafe_noiommu_mode=1
sudo echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
sudo dpdk-devbind -b vfio-pci 0000:08:00.0

Database Startup ​

openGauss configuration file modification:

In the openGauss database configuration file 'postgresql.conf', append parameters: appended parameter file

The postgresql.conf file of the database needs to be modified. Modify the thread pool parameters based on the actual situation and add performance parameters to the data configuration file.

shell
# This parameter specifies the number of CPUs bound to openGauss. For example, if openGauss is bound to 1-29, 32-61, 64-93, and 96-125, that is, 4 NUMA partitions and 119 CPUs in total, the number of threads must be an integer multiple of the number of CPUs. For example, if the multiple is 4, the value is 476.
thread_pool_attr = '476,4,(cpubind:1-29,32-61,64-93,96-125)'

# Enable this option when Gazelle acceleration is used.
enable_gazelle_performance_mode=on

Environment cleanup: To prevent background caches and background semaphores from affecting the test, clear the corresponding data.

shell
ipcs -m | awk '$2 ~/[0-9]+/ {print $2}' | while read s; do ipcrm -m $s; done
ipcs -s | awk '$2 ~/[0-9]+/ {print $2}' | while read s; do ipcrm -s $s; done
echo 3 > /proc/sys/vm/drop_caches

Start the database:

shell
# Set the path of the Gazelle configuration file.
LSTACK_CONF_PATH=/usr1/gazelle/gazelle_conf/lstack.conf
# Start Gazelle to accelerate gaussdb. The following uses the started dn1 database node of openGauss as an example.
LD_PRELOAD=liblstack.so GAZELLE_BIND_PROCNAME=gaussdb LSTACK_CONF_PATH=$LSTACK_CONF_PATH /home/opengauss/software/openGauss/bin/gaussdb -D /home/opengauss/software/openGauss/data/dn1

TPCC Test Startup ​

The client needs to configure network interrupts based on the actual situation. The following uses the configuration of 20 interrupts as an example. The corresponding bind_irq.sh file is as follows:

shell
intf=enp4s0

ethtool -G ${intf} rx 1024 tx 1024
ethtool -K ${intf} lro on
ethtool -L ${intf} combined ${combined}

irq_list=`cat /proc/interrupts | grep $intf | awk {'print $1'} | tr -d ":"`
irq_array_net=($irq_list)

cpu_array_irq=(27 28 29 30 31 59 60 61 62 63 91 92 93 94 95 123 124 125 126 127)

for (( i=0;i<20;i++))
do
        echo "${cpu_array_irq[$i]}" > /proc/irq/${irq_array_net[$i]}/smp_affinity_list
done

for j in ${irq_array_net[@]}
do
        cat /proc/irq/$j/smp_affinity_list
done

The benchmark client needs to modify the number of concurrent connections based on the actual situation. Usually, modify it to a multiple of the number of cores specified by the cpubind parameter in the postgresql.conf file. For example, for 1-29,32-61,64-93,96-125, the number of cores is 119. If the multiple is 4, the value is 714, and the number of concurrent connections is changed to 714:

shell
terminals=714

Restoration ​

  1. Stop the database.
  2. Use the root user to switch the NIC back to kernel mode.

Switch the user-mode NIC to a kernel-mode NIC:

shell
#PCI=0000:08:00.0
#NET_DEVICE=enp8s0
#IP=20.20.20.119
sudo dpdk-devbind -u 0000:08:00.0
sudo dpdk-devbind -b hinic 0000:08:00.0
sudo ifconfig enp8s0 20.20.20.119 netmask 255.255.255.0

Test Conclusion ​

In the standalone test, the measured TPCC performance is improved by about 15%.

openGauss File Configuration Reference ​

postgresql.conf file: After opening the postgresql.conf file, append the following content to the end of the file (the IP address must be changed based on the actual IP address):

shell
max_connections = 4096
allow_concurrent_tuple_update = true
audit_enabled = off
checkpoint_segments = 1024
checkpoint_timeout = 15min
cstore_buffers = 16MB
enable_alarm = off
enable_codegen = false
enable_data_replicate = off
full_page_writes = off
max_files_per_process = 100000
max_prepared_transactions = 2048
shared_buffers = 350GB
#max_process_memory = 100GB
use_workload_manager = off
wal_buffers = 1GB
work_mem = 1MB
log_min_messages = FATAL
transaction_isolation = 'read committed'
default_transaction_isolation = 'read committed'
synchronous_commit = on
fsync = on
maintenance_work_mem = 2GB
vacuum_cost_limit = 10000
autovacuum = on
autovacuum_mode = vacuum
autovacuum_max_workers = 20
autovacuum_naptime = 5s
autovacuum_vacuum_cost_delay = 10
#xloginsert_locks = 48
update_lockwait_timeout = 20min

enable_mergejoin = off
enable_nestloop = off
#-----
enable_hashjoin = off
#enable_cbm_tracking = off
enable_bitmapscan = on
enable_material = off

wal_log_hints = off
log_duration = off
checkpoint_timeout = 15min
autovacuum_vacuum_scale_factor = 0.1
autovacuum_analyze_scale_factor = 0.02
enable_save_datachanged_timestamp = false

# enable_stat_send = off
# enable_commandid_send = off
log_timezone = 'PRC'
timezone = 'PRC'
lc_messages = 'C'
lc_monetary = 'C'
lc_numeric = 'C'
lc_time = 'C'

enable_thread_pool = on

enable_double_write = on

enable_incremental_checkpoint = on
enable_opfusion = on
instr_unique_sql_count=0
advance_xlog_file_num = 100
numa_distribute_mode = 'all'

track_activities = off
enable_instr_track_wait = off
enable_instr_rt_percentile = off
track_counts = off
track_sql_count = off
enable_instr_cpu_timer = off

plog_merge_age = 0

# numa_excluded_cpus = '29,30,31,61,62,63,93,94,95,125,126,127'
# numa_excluded_cpus = '28,29,30,31,60,61,62,63,92,93,94,95,124,125,126,127'

#enable_crc_check = off
session_timeout = 0

enable_instance_metric_persistent = off
enable_logical_io_statistics = off
enable_page_lsn_check = off
enable_user_metric_persistent = off
enable_xlog_prune = off

enable_resource_track = off
instr_unique_sql_count=0

remote_read_mode = non_authentication
client_min_messages = ERROR
log_min_messages = FATAL
enable_asp = off
enable_bbox_dump = off
bgwriter_flush_after = 32
#gs_clean_timeout = '300s'
#minimum_pool_size = 200
wal_keep_segments = 1025


#scan_fusion
enable_bitmapscan = off
enable_seqscan = off

track_activities='off'
enable_resource_track='off'
enable_instr_rt_percentile='off'
enable_instr_cpu_timer='off'
# bypass_workload_manager='off'
enable_asp='off'

enable_thread_pool = on
xloginsert_locks=16
checkpoint_segments=8000

enable_stmt_track=false

data_replicate_buffer_size=128MB

#bgwriter_thread_num = 1

bgwriter_delay = 5s
incremental_checkpoint_timeout = 5min
pagewriter_thread_num = 2
candidate_buf_percent_target = 0.3
pagewriter_sleep = 100ms

standby_shared_buffers_fraction = 0.9

#replconninfo1 = 'localhost=20.20.20.58 localport=21131 localheartbeatport=21135 localservice=21134 remotehost=20.20.20.64 remoteport=21131 remoteheartbeatport=21135 remoteservice=21134'

# 2020-12-20
enable_incremental_checkpoint = on
checkpoint_segments = '4096'
checkpoint_timeout = 50min
#enable_thread_pool = off

max_process_memory = 240GB
shared_buffers = 180GB

wal_level = archive
hot_standby = off

wal_receiver_buffer_size = 256MB
walsender_max_send_size = 32MB


# 2021-03-16
xloginsert_locks = 8
wal_file_init_num = 30
#xlog_idle_flushes_before_sleep = 500000000
walwriter_sleep_threshold = 50000
wal_writer_delay = 150

undo_zone_count = 0

walwriter_cpu_bind = 0
checkpoint_segments = 32768

local_syscache_threshold = 16MB
enable_cachedplan_mgr = off
enable_global_syscache = off

time_record_level = 1
enable_beta_opfusion = on

#wal_file_init_num = 10000
#advance_xlog_file_num = 50000
light_comm = on
enable_indexscan_optimization = on

hot_standby = off

checkpoint_segments = 3000
advance_xlog_file_num = 100000
thread_pool_attr = '476,4,(cpubind:1-29,32-61,64-93,96-125)'
enable_gazelle_performance_mode=on

#------------------------------------------------------------------------------
wal_file_init_num = 40000
advance_xlog_file_num = 100000

版权所有 © 2026 openEuler 保留一切权利

京ICP备2020036654号-1

京公网安备 11030102011597 号

品牌隐私声明法律声明关于cookies