Instructions
nvme.conf.in Configuration File
By default, the HSAK configuration file is located in /etc/spdk/nvme.conf.in. You can modify the configuration file based on service requirements. The content of the configuration file is as follows:
- [Global]
- ReactorMask: cores used for I/O polling. The value is a hexadecimal number and cannot be set to core 0. The bits from the least significant one to the most significant one indicate different CPU cores. For example, 0x1 indicates core 0, and 0x6 indicates cores 1 and 2. This parameter supports a maximum of 34 characters, including the hexadecimal flag 0x. Each hexadecimal character can be F at most, indicating four cores. Therefore, a maximum of 128 (32 x 4) cores are supported.
- LogLevel: HSAK log print level (0: error; 1: warning; 2: notice; 3: info; 4: debug).
- MemSize: memory occupied by HSAK (The minimum value is 500 MB.)
- MultiQ: whether to enable multi-queue on the same block device.
- E2eDif: DIF type (1: half-way protection; 2: full protection). Drives from different vendors may have different DIF support capabilities. For details, see the documents provided by hardware vendors.
- IoStat: whether to enable the I/O statistics function. The options are Yes and No.
- RpcServer: whether to start the RPC listening thread. The options are Yes and No.
- NvmeCUSE: whether to enable the CUSE function. The options are Yes and No. After the function is enabled, the NVMe character device is generated in the /dev/spdk directory.
- [Nvme]
- TransportID: PCI address and name of the NVMe controller. The format is TransportID "trtype:PCIe traddr:0000:09:00.0" nvme0.
- RetryCount: number of retries upon an I/O failure. The value 0 indicates no retry. The maximum value is 255.
- TimeoutUsec: I/O timeout interval. If this parameter is set to 0 or left blank, no timeout interval is set. The unit is μs.
- ActionOnTimeout: I/O timeout behavior (None: prints information only; Reset: resets the controller; abort: aborts the command). The default value is None.
- [Reactor]
- BatchSize: number of I/Os that can be submitted in batches. The default value is 8, and the maximum value is 32.
Header File Reference
HSAK provides two external header files. Include the two files when using HSAK for development.
- bdev_rw.h: defines the macros, enumerations, data structures, and APIs of the user-mode I/O operations on the data plane.
- ublock.h: defines macros, enumerations, data structures, and APIs for functions such as device management and information obtaining on the management plane.
Service Running
After software development and compilation, you must run the setup.sh script to rebind the NVMe drive driver to the user mode before running the software. The script is located in /opt/spdk by default. Run the following commands to change the drive driver's binding mode from kernel to user and reserve 1024 x 2 MB huge pages:
[root@localhost ~]# cd /opt/spdk
[root@localhost spdk]# ./setup.sh
0000:3f:00.0 (8086 2701): nvme -> uio_pci_generic
0000:40:00.0 (8086 2701): nvme -> uio_pci_generic
Run the following commands to restore the drive driver's mode from user to kernel and free the reserved huge pages:
[root@localhost ~]# cd /opt/spdk
[root@localhost spdk]# ./setup.sh reset
0000:3f:00.0 (8086 2701): uio_pci_generic -> nvme
0000:40:00.0 (8086 2701): uio_pci_generic -> nvme
User-Mode I/O Read and Write Scenarios
Call HSAK APIs in the following sequence to read and write service data through the user-mode I/O channel:
Initialize the HSAK UIO module.
Call libstorage_init_module to initialize the HSAK user-mode I/O channel.Open a drive block device.
Call libstorage_open to open a specified block device. If multiple block devices need to be opened, call this API repeatedly.Allocate I/O memory.
Call libstorage_alloc_io_buf or libstorage_mem_reserve to allocate memory. libstorage_alloc_io_buf can allocate a maximum of 65 KB I/Os, and libstorage_mem_reserve can allocate unlimited memory unless there is no available space.Perform read and write operations on a drive.
You can call the following APIs to perform read and write operations based on service requirements:- libstorage_async_read
- libstorage_async_readv
- libstorage_async_write
- libstorage_async_writev
- libstorage_sync_read
- libstorage_sync_write
Free I/O memory.
Call libstorage_free_io_buf or libstorage_mem_free to free memory, which must correspond to the API used to allocate memory.Close a drive block device.
Call libstorage_close to close a specified block device. If multiple block devices are opened, call this API repeatedly to close them.API Description libstorage_init_module Initializes the HSAK module. libstorage_open Opens a block device. libstorage_alloc_io_buf Allocates memory from buf_small_pool or buf_large_pool of SPDK. libstorage_mem_reserve Allocates memory space from the huge page memory reserved by DPDK. libstorage_async_read Delivers asynchronous I/O read requests (the read buffer is a contiguous buffer). libstorage_async_readv Delivers asynchronous I/O read requests (the read buffer is a discrete buffer). libstorage_async_write Delivers asynchronous I/O write requests (the write buffer is a contiguous buffer). libstorage_async_wrtiev Delivers asynchronous I/O write requests (the write buffer is a discrete buffer). libstorage_sync_read Delivers synchronous I/O read requests (the read buffer is a contiguous buffer). libstorage_sync_write Delivers synchronous I/O write requests (the write buffer is a contiguous buffer). libstorage_free_io_buf Frees the allocated memory to buf_small_pool or buf_large_pool of SPDK. libstorage_mem_free Frees the memory space that libstorage_mem_reserve allocates. libstorage_close Closes a block device. libstorage_exit_module Exits the HSAK module.
Drive Management Scenarios
HSAK contains a group of C APIs, which can be used to format drives and create and delete namespaces.
Call the C API to initialize the HSAK UIO component. If the HSAK UIO component has been initialized, skip this operation.
libstorage_init_module
Call corresponding APIs to perform drive operations based on service requirements. The following APIs can be called separately:
libstorage_create_namespace
libstorage_delete_namespace
libstorage_delete_all_namespace
libstorage_nvme_create_ctrlr
libstorage_nvme_delete_ctrlr
libstorage_nvme_reload_ctrlr
libstorage_low_level_format_nvm
libstorage_deallocate_block
If you exit the program, destroy the HSAK UIO. If other services are using the HSAK UIO, you do not need to exit the program and destroy the HSAK UIO.
libstorage_exit_module
API Description libstorage_create_namespace Creates a namespace on a specified controller (the prerequisite is that the controller supports namespace management). libstorage_delete_namespace Deletes a namespace from a specified controller. libstorage_delete_all_namespace Deletes all namespaces from a specified controller. libstorage_nvme_create_ctrlr Creates an NVMe controller based on the PCI address. libstorage_nvme_delete_ctrlr Destroys an NVMe controller based on the controller name. libstorage_nvme_reload_ctrlr Automatically creates or destroys the NVMe controller based on the input configuration file. libstorage_low_level_format_nvm Low-level formats an NVMe drive. libstorage_deallocate_block Notifies NVMe drives of blocks that can be freed for garbage collection.
Data-Plane Drive Information Query
The I/O data plane of HSAK provides a group of C APIs for querying drive information. Upper-layer services can process service logic based on the queried information.
Call the C API to initialize the HSAK UIO component. If the HSAK UIO component has been initialized, skip this operation.
libstorage_init_module
Call corresponding APIs to query information based on service requirements. The following APIs can be called separately:
libstorage_get_nvme_ctrlr_info
libstorage_get_mgr_info_by_esn
libstorage_get_mgr_smart_by_esn
libstorage_get_bdev_ns_info
libstorage_get_ctrl_ns_info
If you exit the program, destroy the HSAK UIO. If other services are using the HSAK UIO, you do not need to exit the program and destroy the HSAK UIO.
libstorage_exit_module
API Description libstorage_get_nvme_ctrlr_info Obtains information about all controllers. libstorage_get_mgr_info_by_esn Obtains the management information of the drive corresponding to an ESN. libstorage_get_mgr_smart_by_esn Obtains the S.M.A.R.T. information of the drive corresponding to an ESN. libstorage_get_bdev_ns_info Obtains namespace information based on the device name. libstorage_get_ctrl_ns_info Obtains information about all namespaces based on the controller name.
Management-Plane Drive Information Query
The management plane component Ublock of HSAK provides a group of C APIs for querying drive information on the management plane.
Call the C API to initialize the HSAK Ublock server.
API Description init_ublock Initializes the Ublock function module. This API must be called before the other Ublock APIs. A process can be initialized only once because the init_ublock API initializes DPDK. The initial memory allocated by DPDK is bound to the process PID. One PID can be bound to only one memory. In addition, DPDK does not provide an API for freeing the memory. The memory can be freed only by exiting the process. ublock_init It is the macro definition of the init_ublock API. It can be considered as initializing Ublock to an RPC service. ublock_init_norpc It is the macro definition of the init_ublock API. It can be considered as initializing Ublock to a non-RPC service. Call the HSAK UIO component initialization API in another process based on service requirements.
Call the APIs listed in the following table on the Ublock server process or client process to query information.
API Description ublock_get_bdevs Obtains the device list. The obtained device list contains only PCI addresses and does not contain specific device information. To obtain specific device information, call the ublock_get_bdev API. ublock_get_bdev Obtains information about a specific device, including the device serial number, model, and firmware version. The information is stored in character arrays instead of character strings. ublock_get_bdev_by_esn Obtains the device information based on the specified ESN, including the serial number, model, and firmware version. ublock_get_SMART_info Obtains the S.M.A.R.T. information of a specified device. ublock_get_SMART_info_by_esn Obtains the S.M.A.R.T. information of the device corresponding to an ESN. ublock_get_error_log_info Obtains the error log information of a device. ublock_get_log_page Obtains information about a specified log page of a specified device. After obtaining the block device list, call the APIs listed in the following table to free resources.
API Description ublock_free_bdevs Frees the device list. ublock_free_bdev Frees device resources. If you exit the program, destroy the HSAK Ublock module (the destruction method on the server is the same as that on the client).
API Description ublock_fini Destroys the Ublock module. This API destroys the Ublock module and internally created resources. This API must be used together with the Ublock initialization API.
Log Management
HSAK logs are exported to /var/log/messages through syslog by default and managed by the rsyslog service of the OS. If a custom log directory is required, use rsyslog to configure the log directory.
Modify the /etc/rsyslog.conf configuration file.
if ($programname == 'LibStorage') then { action(type="omfile" fileCreateMode="0600" file="/var/log/HSAK/run.log") stop }
Restart the rsyslog service:
sysemctl restart rsyslog
Start the HSAK process. The log information is redirected to the target directory.
If redirected logs need to be dumped, manually configure log dump in the /etc/logrotate.d/syslog file.