Distributed File System Overview

OpenHarmony distributed file system (hmdfs) distributed file system provides cross-device file access capabilities in the following scenarios:

When two devices are deployed on a network, device A can transparently read and modify files on device B.
The edge server can automatically synchronize file data from multiple embedded devices on the network.

The hmdfs provides a globally consistent access view across devices dynamically connected to a network via DSoftBus and allows you to implement high-performance read and write operations on files with low latency by using basic file system APIs.

It consists of the following core modules:

distributed_file_daemon: user-mode daemon for distributed file management, which is responsible for access device networking, data transmission, and hmdfs mounting.
hmdfs: core module of the distributed filesystem. It is a high-performance, kernel-mode, and layered file system for mobile distributed scenarios.

Constraints

Supported Interfaces

Distributed file management does not support or partially supports the following system calls of the Virtual File System (VFS):

symlink is not supported.
mmap supports read only.
rename supports only operations within the same directory.

Specifications

Maximum number of directory levels
The value is the same as the overlaid file system, that is, the file system used by the data partition, such as ext4 and F2FS.
Maximum file name length
The smaller of 680 bytes and the length supported by the overlaid file system. For F2FS and ext4, the value is 255 bytes.
Maximum size of a single file
The smaller of $2^{64}$B and the size supported by the overlaid file system. The value is 16 TB for ext4 and 3.94 TB for F2FS.

Environment Restrictions

The name of the wired NIC in the running environment must be eth0, and the name of the wireless NIC must be wlan0. You can run the ip a command to check the NIC name in the current environment. If eth0 or wlan0 does not exist, softbus_server fails to be started and the function is invalid. For details, see FAQs.
The dsoftbus and distributed-utils packages cannot be installed at the same time. dsoftbus contains DSoftBus 3.1.2, while distributed-utils contains DSoftBus 3.2. Uninstall dsoftbus first if you want to use DSoftBus 3.2.
Binder must be enabled for the running kernel. If Binder is not enabled, insert the .ko file of Binder or recompile the kernel. For details, see the communication_ipc repository README。
The kernel version of openEuler must be 5.10.x. You can run the uname -r command to view the kernel version.
All openEuler devices are in the same subnet, and the connections between the devices are normal. The firewall does not intercept data packets from DSoftBus.

Description

Installation

Note: If a step fails to be performed, rectify the fault by referring to FAQs.

Install the hmdfs, distributed-utils, and dfs_service software packages. Run the following command:
```
sudo dnf install hmdfs distributed-utils dfs_service
```
Install the hmdfs file system. After the hmdfs software package is installed, the hmdfs.ko file is provided and stored in the /lib/modules/$(uname -r)/hmdfs directory. Insert the .ko file to install hmdfs.
```
cd /lib/modules/$(uname -r)/hmdfs
insmod hmdfs.ko
```

Configuration

Each system ability (SA) is started by the sa_main binary file, which depends on service profiles to start services. Therefore, configure the service profile before starting a service.

Create the /system/profile directory and create XML files for device_manager, dfs_service, huks_service, and huks_service.

mkdir -p /system/profile
cd /system/profile
touch device_manager.xml distributedfiledaemon.xml huks_service.xml softbus_server.xml

Save the following content in the XML file to configure the services.

device_manager.xml

<?xml version="1.0" encoding="utf-8"?>
<info>
    <process>device_manager</process>
    <loadlibs>
        <libpath>libdevicemanagerservice.z.so</libpath>
    </loadlibs>
    <systemability>
        <name>4802</name>
        <libpath>libdevicemanagerservice.z.so</libpath>
        <run-on-create>true</run-on-create>
        <distributed>false</distributed>
        <dump-level>1</dump-level>
    </systemability>
</info>

softbus_server.xml

<?xml version="1.0" encoding="utf-8"?>
<info>
    <process>softbus_server</process>
    <loadlibs>
        <libpath>libsoftbus_server.z.so</libpath>
    </loadlibs>
    <systemability>
        <name>4700</name>
        <libpath>libsoftbus_server.z.so</libpath>
        <run-on-create>true</run-on-create>
        <distributed>false</distributed>
        <dump-level>1</dump-level>
    </systemability>
</info>

huks_service.xml

<?xml version="1.0" encoding="utf-8"?>
<info>
    <process>huks_service</process>
    <loadlibs>
        <libpath>libhuks_service.z.so</libpath>
    </loadlibs>
    <systemability>
        <name>3510</name>
        <libpath>libhuks_service.z.so</libpath>
        <run-on-create>true</run-on-create>
        <distributed>false</distributed>
        <dump-level>1</dump-level>
    </systemability>
</info>

distributedfiledaemon.xml

<?xml version="1.0" encoding="utf-8"?>
<info>
    <process>distributedfiledaemon</process>
    <loadlibs>
        <libpath>libdistributedfiledaemon.z.so</libpath>
    </loadlibs>
    <systemability>
        <name>5201</name>
        <libpath>libdistributedfiledaemon.z.so</libpath>
        <depend>4802;4700</depend>
        <depend-time-out>60000</depend-time-out>
        <run-on-create>true</run-on-create>
        <distributed>false</distributed>
        <dump-level>1</dump-level>
    </systemability>
</info>

The startup of some services depends on the dynamic library libsec_shared.z.so, which is named libboundscheck.so in openEuler (provided by the libboundscheck software package). Therefore, you need to create a soft link to libsec_shared.z.so in /usr/lib64.
```
ln -s /usr/lib64/libboundscheck.so /usr/lib64/libsec_shared.z.so
```
Configure the SN of each device. Currently, services such as softbus_sever use the SN set in the /etc/SN file to obtain the UDID of the device. Therefore, you need to set a unique SN for each openEuler device.
```
echo "111" > /etc/SN # Set different values for different devices.
```

Usage

To use hmdfs, you need to mount the hmdfs directory and start the distributed_file_daemon service. Perform the following steps on each openEuler device.

Mounting the hmdfs Directory

Run the mount command to mount the hmdfs directory. Ensure that the directory structure is the same as that of OpenHarmony. Mount /data/service/el2/100/non_account to /mnt/hmdfs/100/non_account.
```
mkdir -p /data/service/el2/100/non_account
mkdir -p /mnt/hmdfs/100/non_account
sudo mount -t hmdfs -o merge,local_dst="/mnt/hmdfs/100/non_account" "/data/service/el2/100//non_account" "/mnt/hmdfs/100/non_account"
```
After the directory is mounted, you can run the df -h command to view the mounted directory, which contains the device_view and merge_view directories.
```
├── device_view
│   └── local
└── merge_view
```

Starting the dfs_service Service

After libdistrbited-utils and dfs_service are installed, related executable binary files are stored in /system/bin/, and library files are stored in /system/lib64. Perform the following steps in /system/bin.

Start samgr.
```
cd /system/bin/
./samgr
```

Start huks_service.

./sa_main /system/profile/huks_service.xml

Start device_auth_service.
```
./deviceauth_service
```

Start softbus_server.

./sa_main /system/profile/softbus_server.xml

Start device_manager.

./sa_main /system/profile/device_manager.xml

Start distributed_file_daemon.

./sa_main /system/profile/distributedfiledaemon.xml

Function Usage

After distributed_file_daemon is started on each openEuler device, you can view the directories of the remote devices in /mnt/hmdfs/100/non_account. In this example, only two openEuler devices are connected.

├── device_view
│   ├── fceda1e26c36d1dd0ba65c00d71c1ab619fcf088ad2adf33cd1e2f396dc70ee2
│   └── local
└── merge_view

There are two file views in the directory: device_view, which contains local file view and remote file view; and merge_view, which is the merged file view, containing files of multiple devices.

If you need to perform cross-device file operations, simply perform operations on the files in the remote device directory in device_view.

FAQs

When a service is started, the error message "Binder Driver died" is displayed.
Cause: Binder is not enabled in the system. You can check whether the /dev/binder file exists. If the file does not exist, Binder is not enabled.
Solution: Start Binder by referring to the communication_ipc repository README.
The hmdfs.ko file cannot be inserted, and the error "insmod: ERROR: could not insert module hmdfs.ko: Invalid parameters" is reported.
Cause: The kernel used for compiling hmdfs is different from that in the current environment.
Solution 1: Compile a hmdfs.ko file that matches the kernel of the current environment, and then insert the .ko file.
Solution 2: Use an openEuler version that has the same kernel as openEuler 22.03 LTS SP2.
The softbus_server service fails to be started, and the error message "GetNetworkIfIp ifName:eth0 fail" is displayed.
Cause: Run the ip a command to view the name of the NIC in the current system and check whether the wired NIC eth0 exists. The softbus_server service obtains information such as the IP address through the wired NIC eth0. If eth0 does not exist, softbus_server cannot be started.
Solution 1: Change the NIC name to eth0.
Solution 2: Modify the softbus_server source code and change the name of the dependent wired NIC to that of the NIC in the current system.
After the softbus_server service is started on multiple openEuler devices, the distributed_file_daemon service logs show that no online device is found.
Cause: The network between devices is disconnected, or the firewall blocks DSoftBus data.
Solution: Check whether the network is normal. (You can run systemctl stop firewalld.service to temporarily disable the firewall and test the network if services will not be affected.)