hmdfs Distributed File System Overview

OpenHarmony distributed file system (hmdfs) distributed file system provides cross-device file access capabilities in the following scenarios:

• When two devices are deployed on a network, device A can transparently read and modify files on device B.

• The edge server can automatically synchronize file data from multiple embedded devices on the network.

The hmdfs provides a globally consistent access view across devices dynamically connected to a network via DSoftBus and allows you to implement high-performance read and write operations on files with low latency by using basic file system APIs.

It consists of the following core modules:

• distributed_file_daemon: user-mode daemon for distributed file management, which is responsible for access device networking, data transmission, and hmdfs mounting.

• hmdfs: core module of the distributed filesystem. It is a high-performance, kernel-mode, and layered file system for mobile distributed scenarios.

Constraints

Supported Interfaces

Distributed file management does not support or partially supports the following system calls of the Virtual File System (VFS):

• symlink is not supported.
• mmap supports read only.
• rename supports only operations within the same directory.

Specifications

• Maximum number of directory levels

  The value is the same as the overlaid file system, that is, the file system used by the data partition, such as ext4 and F2FS.

• Maximum file name length

  The smaller of 680 bytes and the length supported by the overlaid file system. For F2FS and ext4, the value is 255 bytes.

• Maximum size of a single file

  The smaller of $2^{64}$B and the size supported by the overlaid file system. The value is 16 TB for ext4 and 3.94 TB for F2FS.

Environment Restrictions

• The name of the wired NIC in the running environment must be eth0, and the name of the wireless NIC must be wlan0. You can run the ip a command to check the NIC name in the current environment. If eth0 or wlan0 does not exist, softbus_server fails to be started and the function is invalid. For details, see FAQs.

• The kernel version of openEuler must be 5.10.x. You can run the uname -r command to view the kernel version.

• All openEuler devices are in the same subnet, and the connections between the devices are normal. The firewall does not intercept data packets from DSoftBus.

Description

Installation

Note: If a step fails to be performed, rectify the fault by referring to FAQs.

1. Install the hmdfs and filemanagement_dfs_service software packages. Run the following command:

   sudo dnf install hmdfs filemanagement_dfs_service
   

2. Install the hmdfs file system. After the hmdfs software package is installed, the hmdfs.ko file is provided and stored in the /lib/modules/$(uname -r)/hmdfs directory. Insert the .ko file to install hmdfs.

   cd /lib/modules/$(uname -r)/hmdfs
   insmod hmdfs.ko
   

   Notice

   • If the hmdfs.ko file does not exist in the /lib/modules/$(uname -r)/hmdfs/ directory, the kernel version used for building the hmdfs project is not the same as the kernel version in the current environment. You can search for the hmdfs.ko file of other kernel versions in the /lib/modules directory.
   • In the Raspberry Pi environment, use hmdfs.ko compiled on openEuler 22.03 LTS SP3 raspberry-pi kernel.

Configuration

1. The startup of some services depends on the dynamic library libsec_shared.z.so, which is named libboundscheck.so in openEuler (provided by the libboundscheck software package). Therefore, you need to create a soft link to libsec_shared.z.so in /usr/lib64.

   ln -s /usr/lib64/libboundscheck.so /usr/lib64/libsec_shared.z.so
   

2. Configure the SN of each device. Currently, services such as softbus_sever use the SN set in the /etc/SN file to obtain the UDID of the device. Therefore, you need to set a unique SN for each openEuler device.

   echo "111" > /etc/SN # Set different values for different devices.
   

Usage

To use hmdfs, you need to mount the hmdfs directory and start the distributed_file_daemon service.

Notice

Perform the following steps on each openEuler device.

Mounting the hmdfs Directory

1. Run the mount command to mount the hmdfs directory. Ensure that the directory structure is the same as that of OpenHarmony. Mount /data/service/el2/100/non_account to /mnt/hmdfs/100/non_account.

   mkdir -p /data/service/el2/100/non_account
   mkdir -p /mnt/hmdfs/100/non_account
   sudo mount -t hmdfs -o merge,local_dst="/mnt/hmdfs/100/non_account" "/data/service/el2/100//non_account" "/mnt/hmdfs/100/non_account"
   

   After the directory is mounted, you can run the df -h command to view the mounted directory, which contains the device_view and merge_view directories.

   ├── device_view
   │   └── local
   └── merge_view
   

Starting the dfs_service Service

After filemanagement_dfs_service and its dependent software packages are installed, related executable binary files are stored in /system/bin/, and library files are stored in /system/lib64.

1. Start the dfs_service distributed file service:

   cd /system/bin
   ./start_services.sh dfs
   

2. Stop the dfs_service distributed file service:

   cd /system/bin
   ./stop_service.sh dfs
   

Function Usage

After distributed_file_daemon is started on each openEuler device, you can view the directories of the remote devices in /mnt/hmdfs/100/non_account. In this example, only two openEuler devices are connected.

├── device_view
│   ├── fceda1e26c36d1dd0ba65c00d71c1ab619fcf088ad2adf33cd1e2f396dc70ee2
│   └── local
└── merge_view

There are two file views in the directory: device_view, which contains local file view and remote file view; and merge_view, which is the merged file view, containing files of multiple devices.

If you need to perform cross-device file operations, simply perform operations on the files in the remote device directory in device_view.

FAQs

• When a service is started, the error message "Binder Driver died" is displayed.

  Cause: Binder is not enabled in the system. You can check whether the /dev/binder file exists. If the file does not exist, Binder is not enabled.

  Solution: Start Binder by referring to the communication_ipc repository README.

• The hmdfs.ko file cannot be inserted, and the error "insmod: ERROR: could not insert module hmdfs.ko: Invalid parameters" is reported.

  Cause: The kernel used for compiling hmdfs is different from that in the current environment.

  Solution 1: Compile a hmdfs.ko file that matches the kernel of the current environment, and then insert the .ko file.

  Solution 2: Use an openEuler version that has the same kernel as openEuler 22.03 LTS SP3.

• In the openEuler 22.03 LTS SP3 Raspberry Pi version, the hmdfs.ko file cannot be inserted and the error "insmod: ERROR: could not insert module hmdfs.ko: Invalid module format" is reported.

  Cause: The Raspberry Pi version uses a special kernel. However, the hmdfs software in the repository is built using the kernel of the openEuler 22.03 LTS SP3 server version. Therefore, the .ko file cannot be inserted.

  Solution: Use hmdfs.ko compiled on openEuler 22.03 LTS SP3 raspberry-pi kernel.

• The softbus_server service fails to be started, and the error message "GetNetworkIfIp ifName:eth0 fail" is displayed.

  Cause: Run the ip a command to view the name of the NIC in the current system and check whether the wired NIC eth0 exists. The softbus_server service obtains information such as the IP address through the wired NIC eth0. If eth0 does not exist, softbus_server cannot be started.

  Solution 1: Change the NIC name to eth0.

  Solution 2: Modify the softbus_server source code and change the name of the dependent wired NIC to that of the NIC in the current system.

• After the softbus_server service is started on multiple openEuler devices, the distributed_file_daemon service logs show that no online device is found.

  Cause: The network between devices is disconnected, or the firewall blocks DSoftBus data.

  Solution: Check whether the network is normal. (You can run systemctl stop firewalld.service to temporarily disable the firewall and test the network if services will not be affected.)