sysTrace User Guide ​

Introduction ​

sysTrace is software used in AI training tasks. In AI training, training task faults often cause waste of training costs. The service pain points are as follows:

• Lack of regular monitoring and detection capabilities for AI training performance faults.
• Lack of full-stack tracing capabilities for slow AI tasks and card faults caused by host bound.

The sysTrace tool supports the following functions:

• Collect the call stack of Python functions at the torch_npu layer.
• Collect the memory usage at the CANN layer to determine whether an HBM OOM fault occurs.
• Collect the delivery/execution of communication operators of mspti to determine whether the operator slow problem occurs, thereby locating the slow card.

Installation ​

To install the sysTrace tool, the operating system must be openEuler 22.03 SP4. On a machine where the openEuler yum source has been configured, run the yum command to install the tool. This section describes how to install the sysTrace tool.

Environment Requirements ​

• Operating system: openEuler 22.03 SP4
• Ascend CANN version 8.0 RC3 or later
• Libunwind version 1.7 or later

Installation Procedure ​

Configure the openEuler yum source and run the yum command to install the tool.

yum install sysTrace

Usage ​

Collecting Data ​

After the sysTrace software package is installed, load the dynamic library to the AI training task by using the LD_PRELOAD method. (Note: The overhead of sysTrace is affected by AI training tasks. It is recommended that the training overhead fluctuation of the actual test task be less than 0.5%. For scenarios with large fluctuations, the test overhead may be greatly affected, and the overhead of sysTrace is amplified (about 2% in actual tests).)

LD_PRELOAD=/usr/local/lib/libunwind.so.8.2.0:/usr/local/lib/libunwind-aarch64.so.8.2.0:/home/ascend-toolkit-bak/ascend-toolkit/8.0.RC3.10/tools/mspti/lib64/libmspti.so:/usr/lib64/libsysTrace.so python ...

Converting Data ​

## Convert the flame graph (note: only the data of card 0 can be converted)
python /usr/bin/sysTrace/convert_mem_to_flamegraph.py <mem_trace_xxx_rank0.pb> <output_file>

python /usr/bin/sysTrace/convert_pytorch_to_timeline.py --output <output_file>

Display ​

Upload the final data to https://ui.perfetto.dev/ for display.