Deploying A-Ops
1 Introduction to A-Ops
A-Ops is a service used to improve the overall security of hosts. It provides functions such as asset management, CVE management, exception detection, and configuration source tracing to identify and manage information assets, monitor software vulnerabilities, and rectify system faults on hosts, ensuring stable and secure running of hosts.
The following table describes the modules related to the A-Ops service.
| Module | Description |
|---|---|
| aops-ceres | Client of the A-Ops service. Collects host data and manages other data collectors (such as gala-gopher). Responds to the commands delivered by the management center and processes the requirements and operations of the management center. |
| aops-zeus | A-Ops management center, which interacts with other modules. The default port number is 11111. Provides basic host management functions, such as adding and deleting hosts and host groups. |
| aops-diana | Exception diagnosis module of A-Ops. The default port number is 11112. Identifies faults that occur on the host and rectifies the faults by analyzing the data on the target host. |
| aops-hermes | Provides a visualized operation interface for A-Ops to display data information to users. |
| aops-apollo | Vulnerability management module of A-Ops. The default port number is 11116. Identifies clients, and periodically obtains security notices released by the openEuler community and updates them to the vulnerability database. Detects vulnerabilities in the system and software by comparing the vulnerabilities with those in the vulnerability database. |
| aops-vulcanus | Basic tool library of A-Ops. Except the aops-ceres and aops-hermes modules, other modules must be installed and used together with this module. |
| aops-tools | Basic environment deployment script. The script is available in the /opt/aops/scripts directory after A-Ops is installed. |
| gala-ragdoll | Configuration source tracing module of A-Ops. Uses Git to monitor and record configuration file changes. The default port number is 11114. |
| dnf-hotpatch-plugin | DNF plug-in, which allows DNF to recognize hot patch information and provides hot patch scanning and application. |
2 Environment Requirements
You are advised to use three hosts running on openEuler 22.03 LTS SP1 for deployment. The deployment scheme is as follows:
- Host A is used to deploy MySQL, Elasticsearch, Kafka, and Prometheus. It provides data service support and deploys aops-diana running in diagnosis mode. The recommended memory is more than 8 GB.
- Host B is used to deploy the A-Ops server and provides service function support. The recommended memory is more than 6 GB.
- Host C is used to deploy the A-Ops client and is used as a host managed and monitored by the A-Ops service. aops-ceres and gala-gopher need to be deployed on host C. The recommended memory is more than 4 GB.
| Host | IP Address | Module |
|---|---|---|
| Host A | 192.168.1.1 | MySQL, Elasticsearch, Kafka, Prometheus, aops-diana, Redis |
| Host B | 192.168.1.2 | aops-zeus, aops-apollo, aops-diana, aops-hermes, gala-ragdoll |
| Host C | 192.168.1.3 | aops-ceres, gala-gopher |
3 Server Deployment
3.1 Host Management
To use the host management function, you need to deploy the aops-zeus, aops-hermes, and MySQL services.
Before deployment, disable the firewall on each host.
systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld
3.1.1 Node Information
| Host | IP Address | Module |
|---|---|---|
| Host A | 192.168.1.1 | MySQL, aops-tools, Prometheus, Redis |
| Host B | 192.168.1.2 | aops-zeus, aops-vulcanus, aops-hermes |
| Host C | 192.168.1.3 | aops-ceres, gala-gopher |
3.1.2 Deployment Procedure
3.1.2.1 Deploying MySQL
Install MySQL.
yum install mysql-server
Modify the MySQL configuration file.
vim /etc/my.cnf
Add bind-address and set it to the IP address of the local host.
[mysqld]
bind-address=192.168.1.1
Restart the MySQL service.
systemctl restart mysqld
Connect to the database, set the access permission of the root user, and create the aops database.
$ mysql
mysql> show databases;
mysql> use mysql;
mysql> select user,host from user;
+---------------+-----------+
| user | host |
+---------------+-----------+
| root | localhost | // If the value of host is localhost, only the local host can connect to the MySQL database. The external network and local software client cannot connect to the MySQL database.
| mysql.session | localhost |
| mysql.sys | localhost |
+---------------+-----------+
3 rows in set (0.00 sec)
mysql> update user set host = '%' where user='root'; // Allow the access of the root user using any IP address.
mysql> flush privileges;// Refresh the permissions.
mysql> create database aops default character set utf8mb4 collate utf8mb4_unicode_ci; // Create the aops database.
mysql> exit
3.1.2.2 Deploying Prometheus
Install Prometheus.
yum install prometheus2
Modify the Prometheus configuration file.
vim /etc/prometheus/prometheus.yml
Add the gala-gopher IP addresses of all clients to the monitored targets of Prometheus.
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090', '192.168.1.3:8888'] // Add the gala-gopher IP address of host C.
Start the service.
systemctl start prometheus
3.1.2.3 Deploying Redis
Install Redis.
yum install redis
Modify the Redis configuration file.
vim /etc/redis.conf
Bind IP addresses.
# It is possible to listen to just one or multiple selected interfaces using
# the "bind" configuration directive, followed by one or more IP addresses.
#
# Examples:
#
# bind 192.168.1.100 10.0.0.1
# bind 127.0.0.1 ::1
#
# ~~~ WARNING ~~~ If the computer running Redis is directly exposed to the
# internet, binding to all the interfaces is dangerous and will expose the
# instance to everybody on the internet. So by default we uncomment the
# following bind directive, that will force Redis to listen only into
# the IPv4 lookback interface address (this means Redis will be able to
# accept connections only from clients running into the same computer it
# is running).
#
# IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES
# JUST COMMENT THE FOLLOWING LINE.
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bind 127.0.0.1 192.168.1.1 // Add the actual IP address of host A.
Start the Redis service.
systemctl start redis
3.1.2.4 Deploying aops-zeus
Install aops-zeus.
yum install aops-zeus
Modify the configuration file.
vim /etc/aops/zeus.ini
Change the IP address of each service in the configuration file to the actual IP address. In this document, aops-zeus is deployed on host B. Therefore, you need to set the IP address to the IP address of host B.
[zeus]
ip=192.168.1.2 // Change the IP address to the actual IP address of host B.
port=11111
[uwsgi]
wsgi-file=manage.py
daemonize=/var/log/aops/uwsgi/zeus.log
http-timeout=600
harakiri=600
processes=2 // Generate a specified number of workers or processes.
gevent=100 // Number of gevent asynchronous cores
[mysql]
ip=192.168.1.1 // Change the IP address to the actual IP address of host A.
port=3306
database_name=aops
engine_format=mysql+pymysql://@%s:%s/%s
pool_size=100
pool_recycle=7200
[prometheus]
ip=192.168.1.1 // Change the IP address to the actual IP address of host A.
port=9090
query_range_step=15s
[agent]
default_instance_port=8888
[redis]
ip=192.168.1.1 // Change the IP address to the actual IP address of host A.
port=6379
[diana]
ip=192.168.1.2 // Change the IP address to the actual IP address of host B.
port=11112
[apollo]
ip=192.168.1.2 // Change the IP address to the actual IP address of host B.
port=11116
Start the aops-zeus service.
systemctl start aops-zeus
3.1.2.5 Deploying aops-hermes
Install aops-hermes.
yum install aops-hermes
Modify the configuration file. Because all services are deployed on host B, set the IP address of each service accessed by aops-web to the actual IP address of host B.
vim /etc/nginx/aops-nginx.conf
Some service configurations:
# Ensure that Nginx still uses index.html as the entry when the front-end route changes.
location / {
try_files $uri $uri/ /index.html;
if (!-e $request_filename){
rewrite ^(.*)$ /index.html last;
}
}
location /api/ {
proxy_pass http://192.168.1.2:11111/; // Change it to the actual IP address of the host where aops-zeus is deployed.
}
location /api/domain {
proxy_pass http://192.168.1.2:11114/; // Enter the IP address of gala-ragdoll.
rewrite ^/api/(.*) /$1 break;
}
location /api/check {
proxy_pass http://192.168.1.2:11112/; // Enter the IP address of aops-diana running in configurable mode.
rewrite ^/api/(.*) /$1 break;
}
location /api/vulnerability {
proxy_pass http://192.168.1.2:11116/; // Enter the IP address of aops-apollo.
rewrite ^/api/(.*) /$1 break;
}
Enable the aops-hermes service.
systemctl start aops-hermes
3.2 CVE Management
The CVE management function is implemented based on the aops-zeus service. Therefore, you need to deploy aops-zeus and then aops-apollo.
The running of the aops-apollo service requires the support of the MySQL and Elasticsearch databases.
3.2.1 Node Information
| Host | IP Address | Module |
|---|---|---|
| Host A | 192.168.1.1 | MySQL, Elasticsearch, Redis |
| Host B | 192.168.1.2 | aops-zeus, aops-apollo, aops-hermes, aops-vulcanus |
| Host C | 192.168.1.3 | aops-ceres |
3.2.2 Deployment Procedure
3.2.2.1 Deploying Basic Services
See 3.1 Host Management.
3.2.2.2 Deploying Elasticsearch
Configure the repository for Elasticsearch.
echo "[aops_elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md" > "/etc/yum.repos.d/aops_elascticsearch.repo"
Install Elasticsearch using Yum.
yum install elasticsearch-7.14.0-1
Modify the Elasticsearch configuration file.
vim /etc/elasticsearch/elasticsearch.yml
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
# Change the value to the actual IP address of host A.
network.host: 192.168.1.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]
# Cross-domain configurations
http.cors.enabled: true
http.cors.allow-origin: "*"
#
Restart the Elasticsearch service.
systemctl restart elasticsearch
3.2.2.3 Deploying aops-apollo
Install aops-apollo.
yum install aops-apollo
Modify the configuration file.
vim /etc/aops/apollo.ini
Change the IP address of each service in the apollo.ini to the actual IP address.
[apollo]
ip=192.168.1.2// Change it to the actual IP address of host B.
port=11116
host_vault_dir=/opt/aops
host_vars=/opt/aops/host_vars
[zeus]
ip=192.168.1.2 // Change it to the actual IP address of host B.
port=11111
# herms info is used to send mail.
[hermes]
ip=192.168.1.2 // Change it to the actual IP address of aops-hermes, for example, the IP address of host B.
port=54795 // Change it to the actual port of the hermes service.
[cve]
cve_fix_function=yum
# value between 0-23, for example, 2 means 2:00 in a day.
cve_scan_time=2
[mysql]
ip=192.168.1.1 // Change it to the actual IP address of host A.
port=3306
database_name=aops
engine_format=mysql+pymysql://@%s:%s/%s
pool_size=100
pool_recycle=7200
[elasticsearch]
ip=192.168.1.1 // Change it to the actual IP address of host A.
port=9200
max_es_query_num=10000000
[redis]
ip=192.168.1.1 // Change it to the actual IP address of host A.
port=6379
[uwsgi]
wsgi-file=manage.py
daemonize=/var/log/aops/uwsgi/apollo.log
http-timeout=600
harakiri=600
processes=2
gevent=100
Start the aops-apollo service.
systemctl start aops-apollo
3.3 Exception Detection
The exception detection function is implemented based on the aops-zeus service. Therefore, you need to deploy aops-zeus and then aops-diana.
Considering distributed deployment, the aops-diana service must be deployed on both host A and host B to act as the producer and consumer in the message queue, respectively.
The running of the aops-diana service requires the support of MySQL, Elasticsearch, Kafka, and Prometheus.
3.3.1 Node Information
| Host | IP Address | Module |
|---|---|---|
| Host A | 192.168.1.1 | MySQL, Elasticsearch, Kafka, Prometheus, aops-diana |
| Host B | 192.168.1.2 | aops-zeus, aops-diana, aops-hermes, aops-vulcanus |
| Host C | 192.168.1.3 | aops-ceres, gala-gopher |
3.3.2 Deployment Procedure
3.3.2.1 Deploying Basic Services
See 3.1 Host Management.
3.3.2.2 Deploying Elasticsearch
See 3.2.2.2 Deploying Elasticsearch.
3.3.2.3 Deploying Kafka
Kafka uses ZooKeeper to manage and coordinate agents. Therefore, you need to deploy ZooKeeper when deploying Kafka.
3.3.2.3.1 Installing ZooKeeper
Install ZooKeeper.
yum install zookeeper
Start the ZooKeeper service.
systemctl start zookeeper
3.3.2.3.2 Installing Kafka
Install Kafka.
yum install kafka
Modify the configuration file.
vim /opt/kafka/config/server.properties
Change the value of listeners to the IP address of the local host.
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://192.168.1.1:9092
Start the Kafka service.
cd /opt/kafka/bin
nohup ./kafka-server-start.sh ../config/server.properties &
tail -f ./nohup.out # Check all the outputs of nohup. If the IP address of host A and the Kafka startup success INFO are displayed, Kafka is started successfully.
3.3.2.4 Deploying aops-diana
The aops-diana installation process on host A is the same as that on host B.
Install aops-diana.
yum install aops-diana
Modify the configuration file. The aops-dianas on host A and host B play different roles, which are distinguished based on the differences in the configuration file.
vim /etc/aops/diana.ini
(1) Start aops-diana on host A in executor mode. It functions as the consumer in the Kafka message queue. The configuration file to be modified is as follows:
[diana]
ip=192.168.1.1 // Change the IP address to the actual IP address of host A.
port=11112
mode=executor // This mode is the executor mode. It is used as the executor in common diagnosis mode and functions as the consumer in Kafka.
timing_check=on
[default_mode]
period=60
step=60
[elasticsearch]
ip=192.168.1.1 // Change the IP address to the actual IP address of host A.
port=9200
max_es_query_num=10000000
[mysql]
ip=192.168.1.1 // Change the IP address to the actual IP address of host A.
port=3306
database_name=aops
engine_format=mysql+pymysql://@%s:%s/%s
pool_size=10000
pool_recycle=7200
[redis]
ip=192.168.1.1 // Change the IP address to the actual IP address of host A.
port=6379
[prometheus]
ip=192.168.1.1 // Change the IP address to the actual IP address of host A.
port=9090
query_range_step=15s
[agent]
default_instance_port=8888
[zeus]
ip=192.168.1.2 // Change the IP address to the actual IP address of host B.
port=11111
[consumer]
kafka_server_list=192.168.1.1:9092 // Change the IP address to the actual IP address of host A.
enable_auto_commit=False
auto_offset_reset=earliest
timeout_ms=5
max_records=3
task_name=CHECK_TASK
task_group_id=CHECK_TASK_GROUP_ID
result_name=CHECK_RESULT
[producer]
kafka_server_list = 192.168.1.1:9092 // Change the IP address to the actual IP address of host A.
api_version = 0.11.5
acks = 1
retries = 3
retry_backoff_ms = 100
task_name=CHECK_TASK
task_group_id=CHECK_TASK_GROUP_ID
[uwsgi]
wsgi-file=manage.py
daemonize=/var/log/aops/uwsgi/diana.log
http-timeout=600
harakiri=600
processes=2
threads=2
(2) Start aops-diana on host B in configurable mode. It functions as the producer in the Kafka message queue. The aops-diana port configuration in the aops-hermes file is subject to the IP address and port number of this host. The configuration file to be modified is as follows:
[diana]
ip=192.168.1.2 // Change the IP address to the actual IP address of host B.
port=11112
mode=configurable // This mode is the configurable mode. It is used as a scheduler in common diagnosis mode and functions as the producer.
timing_check=on
[default_mode]
period=60
step=60
[elasticsearch]
ip=192.168.1.1 // Change the IP address to the actual IP address of host A.
port=9200
max_es_query_num=10000000
[mysql]
ip=192.168.1.1 // Change the IP address to the actual IP address of host A.
port=3306
database_name=aops
engine_format=mysql+pymysql://@%s:%s/%s
pool_size=100
pool_recycle=7200
[redis]
ip=192.168.1.1 // Change the IP address to the actual IP address of host A.
port=6379
[prometheus]
ip=192.168.1.1 // Change the IP address to the actual IP address of host A.
port=9090
query_range_step=15s
[agent]
default_instance_port=8888
[zeus]
ip=192.168.1.2 // Change the IP address to the actual IP address of host B.
port=11111
[consumer]
kafka_server_list=192.168.1.1:9092 // Change the IP address to the actual IP address of host A.
enable_auto_commit=False
auto_offset_reset=earliest
timeout_ms=5
max_records=3
task_name=CHECK_TASK
task_group_id=CHECK_TASK_GROUP_ID
result_name=CHECK_RESULT
[producer]
kafka_server_list = 192.168.1.1:9092 // Change the IP address to the actual IP address of host A.
api_version = 0.11.5
acks = 1
retries = 3
retry_backoff_ms = 100
task_name=CHECK_TASK
task_group_id=CHECK_TASK_GROUP_ID
[uwsgi]
wsgi-file=manage.py
daemonize=/var/log/aops/uwsgi/diana.log
http-timeout=600
harakiri=600
processes=2
threads=2
Start the aops-diana service.
systemctl start aops-diana
3.4 Configuring Source Tracing
A-Ops configuration source tracing depends on gala-ragdoll based on aops-zeus. Therefore, you need to deploy aops-zeus and then gala-ragdoll.
3.4.1 Node Information
| Host | IP Address | Module |
|---|---|---|
| Host A | 192.168.1.1 | MySQL, aops-tools |
| Host B | 192.168.1.2 | aops-zeus, gala-ragdoll, aops-hermes, aops-vulcanus |
| Host C | 192.168.1.3 | aops-ceres |
3.4.2 Deployment Procedure
3.4.2.1 Deploying Basic Services
See 3.1 Host Management.
3.4.2.2 Deploying gala-ragdoll
Install gala-ragdoll.
yum install gala-ragdoll
Modify the configuration file.
vim /etc/ragdoll/gala-ragdoll.conf
Change the IP address in collect_address of the collect section to the IP address of host B, and change the values of collect_api and collect_port to the actual API and port number.
[git]
git_dir = "/home/confTraceTest"
user_name = "user_name"
user_email = "user_email"
[collect]
collect_address = "http://192.168.1.2" // Change it to the actual IP address of host B.
collect_api = "/manage/config/collect" // The value is an example. Change it to the actual value.
collect_port = 11111 // Change it to the actual port number of the aops-zeus service.
[sync]
sync_address = "http://192.168.1.2"
sync_api = "/demo/syncConf"
sync_port = 11114
[ragdoll]
port = 11114
Start the gala-ragdoll service.
systemctl start gala-ragdoll
3.5 One-Click Deployment
A-Ops supports one-click deployment based on the script provided by aops-tools. For details, see One-Click Deployment of A-Ops。
4 Client Deployment
The client depends on the aops-ceres module. Some data collection operations depend on the gala-gopher module. For details about how to deploy the aops-ceres module, see Deploying aops-ceres.
5 FAQs
- The aops-zeus management center communicates with other modules through HTTP.
- If the firewall cannot be disabled, enable all interfaces involved in the service deployment process. Otherwise, the services cannot be accessed, affecting the normal use of A-Ops.
- For security purposes, the client initiates the operation of adding new hosts in aops-zeus. For details, see section "Registering with aops-zeus" in Deploying aops-ceres.
- The aops-diana, gala-ragdoll, and aops-apollo modules are independent of each other. You can deploy them as required.
- All services need to be deployed on the server and client. For details about how to deploy the server, see the specific function requirements. For details about how to deploy the client, see aops-ceres and gala-gopher.
- Configure Elasticsearch cross-domain settings based on actual requirements.
- Some HTTP access interfaces are synchronous. When the execution time is long, the web client may return a 504 error. You can set a relatively large number for proxy_read_timeout in the Nginx configurations to reduce the probability of the error. 8.When hosts are added in batches, the service execution is restricted by the maximum number of connections (MaxStartups) of the sshd service on the aops-zeus host. If some hosts cannot be connected, consider temporarily increasing the value of MaxStartups by referring to this document.
