llama.cpp User Guide

Introduction

llama.cpp is a LLaMA model interface based on C/C++. It allows users to deploy and use open source models on CPU machines.

It supports the deployment of multiple open source Chinese LLMs, such as LLaMA, LLaMA2, and Vicuna.

Software Architecture

The llama.cpp core architecture consists of two layers:

• Quantization layer: quantifies open source models to reduce the model size.
• Startup layer: starts a quantized model.

Features:

• Implemented in GGML-based C/C++.
• Accelerated CPU inference with int4/int8 quantization, optimized KV cache and parallel computing.
• Streaming generation with typewriter effect.
• Can run solely on CPUs, with no GPUs required

Installation Guide

Software and Hardware Requirements

Processor architecture: AArch64 and x86_64

OS: openEuler 23.09

Memory: at least 4 GB, depending on the size of different open source models.

Component Installation

Before using llama.cpp to deploy an LLM, install the llama.cpp software package. Before the installation, ensure that the openEuler yum source has been configured.

1. Install the package.

   yum install llama.cpp
   

2. Check whether the package is installed.

   llama_cpp_main -h
   

If the help information is displayed, the installation is successful.

Usage Instructions

Using Directly

1. Install the llama.cpp software package.

   yum install llama-cpp
   

2. Download an open source LLM, such as LLaMa or LLaMa2. Use llama_convert.py to quantify the downloaded open source LLM.

   python3 /usr/bin/llama_convert.py -i model_path/
   

   model_path indicates the path for storing the open source LLM.

3. Start the model and start a dialog.

   llama_cpp_main -m model_path --color --ctx_size 2048 -n -1 -ins -b 256 --top_k 10000 --temp 0.2 --repeat_penalty 1.1 -t 8
   

   model_path indicates the path for storing the quantified model.

   You can run the following command to view the command options:

   llama_cpp_main -h
   

Using With a Container

1. Pull the container image.

   docker pull hub.oepkgs.net/openeuler/llama_image
   

2. Run the container image and start a dialog.

   docker run -it --security-opt seccomp=unconfined hub.oepkgs.net/openeuler/llama_image
   

Normal Startup Page

Figure 1 shows the page after the LLaMA model is started.

Figure 1 LLaMA startup page

Specifications

This project allows you to deploy and infer LLMs on CPU-level machines. However, the model inference speed is hardware-dependent. If the hardware is set to the lowest settings, the inference speed may be slow, which can lower usage efficiency.

Table 1 lists reference inference speed in different machine configurations.

The table shows that Q4_0, Q4_1, Q5_0, and Q5_1 represent the quantization precision, while ms/token indicates the duration in milliseconds spent by each token inference, which is a measure of inference speed. A smaller value for ms/token indicates a higher inference speed.

Table 1 Test data of the LLaMA-7B model inference speed