Using the Switchless Feature
Introduction to the Switchless Feature
Definition: Switchless is a technology that uses shared memory to reduce the number of context switches and data copies, and optimize the interaction performance between the REE and TEE.
Typical application scenarios: To reconstruct a traditional application for confidential computing, it is split into a CA in the REE and a TA in the TEE.
- When the CA frequently calls the TA interface, the calling duration is long, which severely affects the service performance.
- When large-block data is frequently exchanged between the CA and TA, multiple memory copies are performed during ECALLs. As a result, the performance is low.
In the preceding scenarios, switchless can be used to optimize interaction performance and reduce performance loss caused by confidential computing reconstruction. The optimal effect can be the same performance as that before the reconstruction.
Supported hardware platforms:
- Intel SGX
- ARM TrustZone Kunpeng 920
Constraints and Limitations
Although enabling switchless saves time spent on context switch and data copy between the REE and TEE, the feature requires additional threads to serve calls. If worker threads are busy waiting for messages, a large number of CPU resources are consumed. In addition, more worker threads usually compete for more CPU resources and cause more thread context switches, which may deteriorate performance. Therefore, based on the actual service model and performance commissioning, the optimal configuration of switchless is to balance the resource and performance requirements.
Feature Configuration Items
When calling cc_enclave_create to create an enclave, you need to pass the switchless configurations to the features parameter. The configuration items are as follows:
typedef struct {
uint32_t num_uworkers;
uint32_t num_tworkers;
uint32_t switchless_calls_pool_size;
uint32_t retries_before_fallback;
uint32_t retries_before_sleep;
uint32_t parameter_num;
uint32_t workers_policy;
uint32_t rollback_to_common;
} cc_sl_config_t;
The following table describes each configuration item.
| Item | Description |
|---|---|
| num_uworkers | Number of proxy worker threads in the REE, which are used to make switchless OCALLs. Currently, this field takes effect only on the SGX platform and can be configured on the ARM platform. However, because the ARM platform does not support OCALLs, the configuration does not take effect on the ARM platform. Specifications: ARM: maximum value: 512; minimum value: 1; and default value: 8 (used when num_uworkers is set to 0). SGX: maximum value: 4294967295; minimum value: 1. |
| num_tworkers | Number of proxy worker threads in the TEE, which are used to make switchless ECALLs. Specifications: ARM: maximum value: 512; minimum value: 1; and default value: 8 (used when num_tworkers is set to 0). SGX: maximum value: 4294967295; minimum value: 1. |
| switchless_calls_pool_size | Size of the switchless call pool. The pool can contain switchless_calls_pool_size x 64 switchless calls. For example, if switchless_calls_pool_size=1, 64 switchless calls are contained in the pool. Specifications: ARM: maximum value: 8; minimum value: 1; default value: 1 (used when switchless_calls_pool_size is set to 0) SGX: maximum value: 8; minimum value: 1; and default value: 1 (used when switchless_calls_pool_size is set to 0). |
| retries_before_fallback | After the pause assembly instruction is executed for retries_before_fallback times, if the switchless call is not made by the proxy worker thread on the other side, the system rolls back to the switch call mode. This field takes effect only on the SGX platform. Specifications: SGX: maximum value: 4294967295; minimum value: 1; and default value: 20000 (used when retries_before_fallback is set to 0). |
| retries_before_sleep | After the pause assembly instruction is executed for retries_before_sleep times, if the proxy worker thread does not receive any task, the proxy worker thread enters the sleep state. This field takes effect only on the SGX platform. Specifications: SGX: maximum value: 4294967295; minimum value: 1; and default value: 20000 (used when retries_before_sleep is set to 0). |
| parameter_num | Maximum number of parameters supported by the switchless function. This parameter takes effect only on the ARM platform. There is no such restriction on the SGX platform. Specifications: ARM: maximum value: 16; minimum value: 0. |
| workers_policy | Running mode of the switchless worker thread. This parameter takes effect only on the ARM platform. Specifications: ARM: WORKERS_POLICY_BUSY: The worker thread always occupies CPU resources regardless of whether there are tasks to be processed. This mode is suitable for scenarios that require high performance and have rich system software and hardware resources. WORKERS_POLICY_WAKEUP:The worker thread is woken up only when there is a task. After the task is processed, the worker thread enters the sleep state and waits to be woken up by a new task. |
| rollback_to_common | Whether to roll back to common invocation when the asynchronous invocation of switchless fails. This parameter takes effect only on the ARM platform. Specifications: ARM:0: No. An error code is returned upon failure. Other: Yes. Common invocation is used upon failure. |
Switchless Development Process
This section describes how to use the switchless feature to develop a switchless program in C language, helping you understand how to use the switchless feature to develop applications.
For details about the process of developing applications based on secGear APIs, see secGear Development Guide.
Compile an Enclave Definition Language (EDL) file.
Add the transition_using_threads flag to the switchless function.
enclave { include "secgear_urts.h" from "secgear_tstdc.edl" import *; from "secgear_tswitchless.edl" import *; trusted { public int get_string_switchless([out, size=32]char *buf) transition_using_threads; }; };Compile the top-level CMakeLists.txt file.
Compile the top-level CMakeLists.txt file, place it in the switchless working directory, and configure information such as the processor architecture and required EDL file for compilation.
EDL_FILE indicates the EDL file and needs to be specified by users. In this example, EDL_FILE is switchless.edl. DPATH is a dynamic library loaded in the TEE. The configuration is shown in the example.
cmake_minimum_required(VERSION 3.10 FATAL_ERROR) project(switchless C) set(CMAKE_C_STANDARD 99) if (NOT DEFINED ENCLAVE) set(ENCLAVE "SGX") endif() set(SGX_SDK_DEFAULT_PATH /opt/intel/sgxsdk) set(GP_SDK_DEFAULT_PATH /opt/itrustee_sdk) set(PL_SDK_DEFAULT_PATH /root/dev/sdk) set(SGX_SSL_DEFAULT_PATH /opt/intel/sgxssl) set(GP_SSL_DEFAULT_PATH /opt/itrustee_sdk/itrustee_sdk_ssl) set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib) if(${ENCLAVE} STREQUAL "GP") if (NOT DEFINED SDK_PATH) set(iTrusteeSDK ${GP_SDK_DEFAULT_PATH}) else() set(iTrusteeSDK ${SDK_PATH}) endif() message("Current Platform: ARM Trustzone, iTrustee SDK PATH:${iTrusteeSDK}") if(NOT IS_DIRECTORY ${iTrusteeSDK}) message(FATAL_ERROR "Please provide the correct iTrusteeSDK path") endif() set(CC_GP ON) endif() if(${ENCLAVE} STREQUAL "SGX") if (NOT DEFINED SDK_PATH) set(SGXSDK ${SGX_SDK_DEFAULT_PATH}) else() set(SGXSDK ${SDK_PATH}) endif() message("Current Platform: Intel SGX, SGX SDK PATH:${SGXSDK}") if(NOT IS_DIRECTORY ${SGXSDK}) message(FATAL_ERROR "Please provide the correct SGXSDK path") endif() set(CC_SGX ON) endif() if(${ENCLAVE} STREQUAL "PL") if (NOT DEFINED SDK_PATH) set(PLSDK ${PL_SDK_DEFAULT_PATH}) else() set(PLSDK ${SDK_PATH}) endif() message("Current Platform: RISC-V, Penglai SDK PATH:${PLSDK}") if(NOT IS_DIRECTORY ${PLSDK}) message(FATAL_ERROR "Please provide the correct Penglai SDK path") endif() set(CC_PL ON) endif() set(CURRENT_ROOT_PATH ${CMAKE_CURRENT_SOURCE_DIR}) #set edl name set(EDL_FILE switchless.edl) set(CODEGEN codegen) if(CC_GP) set(CODETYPE trustzone) set(UUID ebc87fc2-05dc-41b3-85b9-f9f0ef481bad) add_definitions(-DPATH="${LOCAL_ROOT_PATH_INSTALL}/data/${UUID}.sec") endif() if(CC_SGX) set(CODETYPE sgx) add_definitions(-DPATH="${CMAKE_CURRENT_BINARY_DIR}/enclave/enclave.signed.so") endif() add_subdirectory(${CURRENT_ROOT_PATH}/enclave) add_subdirectory(${CURRENT_ROOT_PATH}/host)Compile code and CMakeLists.txt for the REE.
3.1 Compile main.c.
When using cc_enclave_create to create the enclave context in the secure environment, you need to pass the switchless configurations to the features parameter. To enable the switchless feature, you need to create a call pool whose size is determined by switchless_call_pool_size, and create an Untrust/Trust worker thread pool based on num_uworkers/num_tworkers respectively.
#include <stdio.h> #include <unistd.h> #include <linux/limits.h> #include <sys/time.h> #include <string.h> #include "enclave.h" #include "secgear_uswitchless.h" #include "secgear_shared_memory.h" #include "switchless_u.h" #define BUF_LEN 32 int main() { int retval = 0; char *path = PATH; char buf[BUF_LEN]; cc_enclave_t *context = NULL; context = (cc_enclave_t *)malloc(sizeof(cc_enclave_t)); cc_enclave_result_t res = CC_FAIL; char real_p[PATH_MAX]; /* switchless configuration */ cc_sl_config_t sl_cfg = CC_USWITCHLESS_CONFIG_INITIALIZER; sl_cfg.num_tworkers = 2; /* 2 tworkers */ sl_cfg.sl_call_pool_size_qwords = 2; /* 2 * 64 tasks */ enclave_features_t features = {ENCLAVE_FEATURE_SWITCHLESS, (void *)&sl_cfg}; res = cc_enclave_create(real_p, AUTO_ENCLAVE_TYPE, 0, SECGEAR_DEBUG_FLAG, &features, 1, context); ... char *shared_buf = (char *)cc_malloc_shared_memory(context, BUF_LEN); ... /* switchless ecall */ res = get_string_switchless(context, &retval, shared_buf); if (res != CC_SUCCESS || retval != (int)CC_SUCCESS) { printf("Switchless ecall error\n"); } else { printf("shared_buf: %s\n", shared_buf); } res = cc_free_shared_memory(context, shared_buf); ... res = cc_enclave_destroy(context); ... return res; }Asynchronous invocation of switchless: The following changes are made to the invocation of the ecall funcion:
- Start the asynchronous invocation
/* async switchless ecall */ res = get_string_switchless_async(context, &task_id, &retval, shared_buf); ...- Query the result of the asynchronous invocation
/* Query the result based on task_id returned in the last step */ ret = cc_sl_get_async_result(context, task_id, &retval); ...3.2 Compile CMakeLists.txt for the REE.
# Configure compilation environment variables. #set auto code prefix set(PREFIX switchless) #set host exec name set(OUTPUT secgear_switchless) #set host src code set(SOURCE_FILE ${CMAKE_CURRENT_SOURCE_DIR}/main.c) # Use the code generation tool to generate auxiliary code. The CODEGEN and CODETYPE variables are defined in the top-level CMakeLists.txt file. --search-path specifies the path of other EDL files on which the switchless.edl file depends. # set auto code if(CC_GP) set(AUTO_FILES ${CMAKE_CURRENT_BINARY_DIR}/${PREFIX}_u.h ${CMAKE_CURRENT_BINARY_DIR}/${PREFIX}_u.c ${CMAKE_CURRENT_BINARY_DIR}/${PREFIX}_args.h) add_custom_command(OUTPUT ${AUTO_FILES} DEPENDS ${CURRENT_ROOT_PATH}/${EDL_FILE} COMMAND ${CODEGEN} --${CODETYPE} --untrusted ${CURRENT_ROOT_PATH}/${EDL_FILE} --search-path /usr/include/secGear) endif() if(CC_SGX) set(AUTO_FILES ${CMAKE_CURRENT_BINARY_DIR}/${PREFIX}_u.h ${CMAKE_CURRENT_BINARY_DIR}/${PREFIX}_u.c) add_custom_command(OUTPUT ${AUTO_FILES} DEPENDS ${CURRENT_ROOT_PATH}/${EDL_FILE} COMMAND ${CODEGEN} --${CODETYPE} --untrusted ${CURRENT_ROOT_PATH}/${EDL_FILE} --search-path /usr/include/secGear --search-path ${SGXSDK}/include) endif() # Configure compilation options and link options. set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fPIE -L/usr/lib64") set(CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS} -s") # Compile link reference directories. if(CC_GP) if(${CMAKE_VERSION} VERSION_LESS "3.13.0") link_directories(${CMAKE_LIBRARY_OUTPUT_DIRECTORY}) endif() add_executable(${OUTPUT} ${SOURCE_FILE} ${AUTO_FILES}) target_include_directories(${OUTPUT} PRIVATE ${CMAKE_BINARY_DIR}/host /usr/include/secGear ${CMAKE_CURRENT_BINARY_DIR}) if(${CMAKE_VERSION} VERSION_GREATER_EQUAL "3.13.0") target_link_directories(${OUTPUT} PRIVATE /usr/lib64 ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}) endif() endif() if(CC_SGX) if(${CMAKE_VERSION} VERSION_LESS "3.13.0") link_directories(${CMAKE_LIBRARY_OUTPUT_DIRECTORY}) endif() add_executable(${OUTPUT} ${SOURCE_FILE} ${AUTO_FILES}) target_include_directories(${OUTPUT} PRIVATE /usr/include/secGear /opt/intel/sgxsdk/include ${CMAKE_CURRENT_BINARY_DIR}) if(${CMAKE_VERSION} VERSION_GREATER_EQUAL "3.13.0") target_link_directories(${OUTPUT} PRIVATE ${CMAKE_LIBRARY_OUTPUT_DIRECTORY} ${SGXSDK}/lib64) endif() endif() if(CC_SIM) target_link_libraries(${OUTPUT} secgearsim pthread) else() if(CC_GP) target_link_libraries(${OUTPUT} secgear pthread) endif() if(CC_SGX) target_link_libraries(${OUTPUT} secgear pthread -Wl,--whole-archive -lsgx_uswitchless -Wl,--no-whole-archive -lsgx_urts) endif() endif() # Specify the binary installation directory. set_target_properties(${OUTPUT} PROPERTIES SKIP_BUILD_RPATH TRUE) if(CC_GP) install(TARGETS ${OUTPUT} RUNTIME DESTINATION ${LOCAL_ROOT_PATH_INSTALL}/vendor/bin/ PERMISSIONS OWNER_EXECUTE OWNER_WRITE OWNER_READ) endif() if(CC_SGX) install(TARGETS ${OUTPUT} RUNTIME DESTINATION ${CMAKE_BINARY_DIR}/bin/ PERMISSIONS OWNER_EXECUTE OWNER_WRITE OWNER_READ) endif()Compile the code, CMakeLists.txt, and configuration file for the TEE, and place them in the enclave directory.
4.1 Compile the TEE code enclave.c.
#include <stdio.h> #include <string.h> #include "switchless_t.h" #define TA_HELLO_WORLD "secgear hello world!" #define BUF_MAX 32 int get_string_switchless(char *shared_buf) { strncpy(shared_buf, TA_HELLO_WORLD, strlen(TA_HELLO_WORLD) + 1); return 0; }4.2 Compile CMakeLists.txt for the TEE.
#set auto code prefix set(PREFIX switchless) #set sign key set(PEM Enclave_private.pem) #set sign tool set(SIGN_TOOL sign_tool.sh) #set enclave src code set(SOURCE_FILES ${CMAKE_CURRENT_SOURCE_DIR}/enclave.c) #set log level set(PRINT_LEVEL 3) add_definitions(-DPRINT_LEVEL=${PRINT_LEVEL}) # WHITE_LIST_x specifies the binary whitelist of the iTrustee. It specifies the binary files in the REE that can invoke the dynamic library in the TEE. # WHITE_LIST_OWNER specifies the permission of the user who runs the binary files. Only this user can invoke the dynamic library in the TEE. if(CC_GP) #set signed output set(OUTPUT ${UUID}.sec) set(WHITE_LIST_0 ${LOCAL_ROOT_PATH_INSTALL}/vendor/bin/secgear_switchless) set(WHITE_LIST_OWNER root) set(WHITELIST WHITE_LIST_0) set(AUTO_FILES ${CMAKE_CURRENT_BINARY_DIR}/${PREFIX}_t.h ${CMAKE_CURRENT_BINARY_DIR}/${PREFIX}_t.c ${CMAKE_CURRENT_BINARY_DIR}/${PREFIX}_args.h) add_custom_command(OUTPUT ${AUTO_FILES} DEPENDS ${CURRENT_ROOT_PATH}/${EDL_FILE} COMMAND ${CODEGEN} --${CODETYPE} --trusted ${CURRENT_ROOT_PATH}/${EDL_FILE} --search-path /usr/include/secGear) endif() # SGX dynamic library signature for the TEE if(CC_SGX) set(OUTPUT enclave.signed.so) set(AUTO_FILES ${CMAKE_CURRENT_BINARY_DIR}/${PREFIX}_t.h ${CMAKE_CURRENT_BINARY_DIR}/${PREFIX}_t.c) add_custom_command(OUTPUT ${AUTO_FILES} DEPENDS ${CURRENT_ROOT_PATH}/${EDL_FILE} COMMAND ${CODEGEN} --${CODETYPE} --trusted ${CURRENT_ROOT_PATH}/${EDL_FILE} --search-path /usr/include/secGear --search-path ${SGXSDK}/include) endif() # Configure compilation options. set(COMMON_C_FLAGS "-W -Wall -Werror -fno-short-enums -fno-omit-frame-pointer -fstack-protector-strong \ -Wstack-protector --param ssp-buffer-size=4 -frecord-gcc-switches -Wextra -nostdinc -nodefaultlibs \ -fno-peephole -fno-peephole2 -Wno-main -Wno-error=unused-parameter -D_FORTIFY_SOURCE=2 -O2 \ -Wno-error=unused-but-set-variable -L/usr/lib64 -Wno-error=format-truncation=") set(COMMON_C_LINK_FLAGS "-Wl,-z,now -Wl,-z,relro -Wl,-z,noexecstack -Wl,-nostdlib -nodefaultlibs -nostartfiles") # The **manifest.txt** file needs to be generated for the iTrustee to specify the iTrustee compilation options and the search path of header files and link files. if(CC_GP) set(CMAKE_C_FLAGS "${COMMON_C_FLAGS} -march=armv8-a") set(CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS} -s -fPIC") set(CMAKE_SHARED_LINKER_FLAGS "${COMMON_C_LINK_FLAGS} -Wl,-s") set(ITRUSTEE_TEEDIR ${iTrusteeSDK}/) set(ITRUSTEE_LIBC ${iTrusteeSDK}/thirdparty/open_source/musl/libc) if(${CMAKE_VERSION} VERSION_LESS "3.13.0") link_directories(${CMAKE_BINARY_DIR}/lib/) endif() add_library(${PREFIX} SHARED ${SOURCE_FILES} ${AUTO_FILES}) target_include_directories( ${PREFIX} PRIVATE ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_BINARY_DIR}/enclave /usr/include/secGear ${ITRUSTEE_TEEDIR}/include/TA ${ITRUSTEE_TEEDIR}/include/TA/huawei_ext ${ITRUSTEE_LIBC}/arch/aarch64 ${ITRUSTEE_LIBC}/ ${ITRUSTEE_LIBC}/arch/arm/bits ${ITRUSTEE_LIBC}/arch/generic ${ITRUSTEE_LIBC}/arch/arm) if(${CMAKE_VERSION} VERSION_GREATER_EQUAL "3.13.0") target_link_directories(${PREFIX} PUBLIC /usr/lib64) endif() foreach(WHITE_LIST ${WHITELIST}) add_definitions(-D${WHITE_LIST}="${${WHITE_LIST}}") endforeach(WHITE_LIST) add_definitions(-DWHITE_LIST_OWNER="${WHITE_LIST_OWNER}") target_link_libraries(${PREFIX} secgear_tee) #for trustzone compiling, you should connact us to get config and private_key.pem for test, so we will not sign and install binary in this example # add_custom_command(TARGET ${PREFIX} POST_BUILD COMMAND bash ${SIGN_TOOL} -d sign -x trustzone -i ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/lib${PREFIX}.so -c ${CMAKE_CURRENT_SOURCE_DIR}/manifest.txt -m ${CMAKE_CURRENT_SOURCE_DIR}/config_cloud.ini -o ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/${OUTPUT}) install(FILES ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/${OUTPUT} #DESTINATION /data DESTINATION ${LOCAL_ROOT_PATH_INSTALL}/data PERMISSIONS OWNER_EXECUTE OWNER_WRITE OWNER_READ GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE) endif() if(CC_SGX) set(SGX_DIR ${SGXSDK}) set(CMAKE_C_FLAGS "${COMMON_C_FLAGS} -m64 -fvisibility=hidden") set(CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS} -s") set(LINK_LIBRARY_PATH ${SGX_DIR}/lib64) if(CC_SIM) set(Trts_Library_Name sgx_trts_sim) set(Service_Library_Name sgx_tservice_sim) else() set(Trts_Library_Name sgx_trts) set(Service_Library_Name sgx_tservice) endif() set(Crypto_Library_Name sgx_tcrypto) set(CMAKE_SHARED_LINKER_FLAGS "${COMMON_C_LINK_FLAGS} -Wl,-z,defs -Wl,-pie -Bstatic -Bsymbolic -eenclave_entry \ -Wl,--export-dynamic -Wl,--defsym,__ImageBase=0 -Wl,--gc-sections \ -Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/Enclave.lds") if(${CMAKE_VERSION} VERSION_LESS "3.13.0") link_directories(${LINK_LIBRARY_PATH}) endif() add_library(${PREFIX} SHARED ${SOURCE_FILES} ${AUTO_FILES}) target_include_directories(${PREFIX} PRIVATE ${CMAKE_CURRENT_BINARY_DIR} /usr/include/secGear ${SGX_DIR}/include/tlibc ${SGX_DIR}/include/libcxx ${SGX_DIR}/include) if(${CMAKE_VERSION} VERSION_GREATER_EQUAL "3.13.0") target_link_directories(${PREFIX} PRIVATE ${LINK_LIBRARY_PATH}) endif() target_link_libraries(${PREFIX} -Wl,--whole-archive -lsgx_tswitchless ${Trts_Library_Name} -Wl,--no-whole-archive -Wl,--start-group -lsgx_tstdc -lsgx_tcxx -l${Crypto_Library_Name} -l${Service_Library_Name} -Wl,--end-group) add_custom_command(TARGET ${PREFIX} POST_BUILD COMMAND umask 0177 COMMAND openssl genrsa -3 -out ${PEM} 3072 COMMAND bash ${SIGN_TOOL} -d sign -x sgx -i ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/lib${PREFIX}.so -k ${PEM} -o ${OUTPUT} -c ${CMAKE_CURRENT_SOURCE_DIR}/Enclave.config.xml) endif() if(NOT DEFINED CC_PL) set_target_properties(${PREFIX} PROPERTIES SKIP_BUILD_RPATH TRUE) endif()
In the SGX environment, after the switchless feature is enabled and an enclave is created, a core dump occurs when the enclave is destroyed and then used.
To enable switchless for SGX, perform the following steps:
- Pass the switchless feature parameter when calling cc_enclave_create.
- Initialize the switchless thread scheduling in the first ECALL.
If cc_enclave_destroy is directly called without calling the ECALL function, an exception occurs when the switchless scheduling thread is destroyed in the SGX library.
In the actual application scenario of the switchless feature, ECALLs are frequently made. Therefore, after the switchless feature is initialized, ECALLs are usually made, but no exception occurs.
Bug Catching
