LTS

    Innovation Version

      Deploying A-Ops

      1 Introduction to A-Ops

      A-Ops is a service used to improve the overall security of hosts. It provides functions such as asset management, CVE management, exception detection, and configuration source tracing to identify and manage information assets, monitor software vulnerabilities, and rectify system faults on hosts, ensuring stable and secure running of hosts.

      The following table describes the modules related to the A-Ops service.

      ModuleDescription
      aops-ceresClient of the A-Ops service.
      Collects host data and manages other data collectors (such as gala-gopher).
      Responds to the commands delivered by the management center and processes the requirements and operations of the management center.
      aops-zeusA-Ops management center, which interacts with other modules. The default port number is 11111.
      Provides basic host management functions, such as adding and deleting hosts and host groups.
      aops-dianaException diagnosis module of A-Ops. The default port number is 11112.
      Identifies faults that occur on the host and rectifies the faults by analyzing the data on the target host.
      aops-hermesProvides a visualized operation interface for A-Ops to display data information to users.
      aops-apolloVulnerability management module of A-Ops. The default port number is 11116.
      Identifies clients, and periodically obtains security notices released by the openEuler community and updates them to the vulnerability database.
      Detects vulnerabilities in the system and software by comparing the vulnerabilities with those in the vulnerability database.
      aops-vulcanusBasic tool library of A-Ops. Except the aops-ceres and aops-hermes modules, other modules must be installed and used together with this module.
      aops-toolsBasic environment deployment script. The script is available in the /opt/aops/scripts directory after A-Ops is installed.
      gala-ragdollConfiguration source tracing module of A-Ops.
      Uses Git to monitor and record configuration file changes. The default port number is 11114.
      dnf-hotpatch-pluginDNF plug-in, which allows DNF to recognize hot patch information and provides hot patch scanning and application.

      2 Environment Requirements

      You are advised to use three hosts running on openEuler 22.03 LTS SP1 for deployment. The deployment scheme is as follows:

      • Host A is used to deploy MySQL, Elasticsearch, Kafka, and Prometheus. It provides data service support and deploys aops-diana running in diagnosis mode. The recommended memory is more than 8 GB.
      • Host B is used to deploy the A-Ops server and provides service function support. The recommended memory is more than 6 GB.
      • Host C is used to deploy the A-Ops client and is used as a host managed and monitored by the A-Ops service. aops-ceres and gala-gopher need to be deployed on host C. The recommended memory is more than 4 GB.
      HostIP AddressModule
      Host A192.168.1.1MySQL, Elasticsearch, Kafka, Prometheus, aops-diana, Redis
      Host B192.168.1.2aops-zeus, aops-apollo, aops-diana, aops-hermes, gala-ragdoll
      Host C192.168.1.3aops-ceres, gala-gopher

      3 Server Deployment

      3.1 Host Management

      To use the host management function, you need to deploy the aops-zeus, aops-hermes, and MySQL services.

      Before deployment, disable the firewall on each host.

      systemctl stop firewalld
      systemctl disable firewalld
      systemctl status firewalld
      

      3.1.1 Node Information

      HostIP AddressModule
      Host A192.168.1.1MySQL, aops-tools, Prometheus, Redis
      Host B192.168.1.2aops-zeus, aops-vulcanus, aops-hermes
      Host C192.168.1.3aops-ceres, gala-gopher

      3.1.2 Deployment Procedure

      3.1.2.1 Deploying MySQL

      Install MySQL.

      yum install mysql-server
      

      Modify the MySQL configuration file.

      vim /etc/my.cnf
      

      Add bind-address and set it to the IP address of the local host.

      [mysqld]
      bind-address=192.168.1.1
      

      Restart the MySQL service.

      systemctl restart mysqld
      

      Connect to the database, set the access permission of the root user, and create the aops database.

      $ mysql
      
      mysql> show databases;
      mysql> use mysql;
      mysql> select user,host from user;
      
      +---------------+-----------+
      | user          | host      |
      +---------------+-----------+
      | root          | localhost | // If the value of host is localhost, only the local host can connect to the MySQL database. The external network and local software client cannot connect to the MySQL database.
      | mysql.session | localhost |
      | mysql.sys     | localhost |
      +---------------+-----------+
      3 rows in set (0.00 sec)
      
      mysql> update user set host = '%' where user='root'; // Allow the access of the root user using any IP address.
      mysql> flush privileges;// Refresh the permissions.
      mysql> create database aops default character set utf8mb4 collate utf8mb4_unicode_ci;  // Create the aops database.
      mysql> exit
      
      3.1.2.2 Deploying Prometheus

      Install Prometheus.

      yum install prometheus2
      

      Modify the Prometheus configuration file.

      vim /etc/prometheus/prometheus.yml
      

      Add the gala-gopher IP addresses of all clients to the monitored targets of Prometheus.

      # A scrape configuration containing exactly one endpoint to scrape:
      # Here it's Prometheus itself.
      scrape_configs:
        # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
        - job_name: 'prometheus'
      
          # metrics_path defaults to '/metrics'
          # scheme defaults to 'http'.
      
          static_configs:
          - targets: ['localhost:9090', '192.168.1.3:8888'] // Add the gala-gopher IP address of host C.
      

      Start the service.

      systemctl start prometheus
      
      3.1.2.3 Deploying Redis

      Install Redis.

      yum install redis
      

      Modify the Redis configuration file.

      vim /etc/redis.conf
      

      Bind IP addresses.

      # It is possible to listen to just one or multiple selected interfaces using
      # the "bind" configuration directive, followed by one or more IP addresses.
      #
      # Examples:
      #
      # bind 192.168.1.100 10.0.0.1
      # bind 127.0.0.1 ::1
      #
      # ~~~ WARNING ~~~ If the computer running Redis is directly exposed to the
      # internet, binding to all the interfaces is dangerous and will expose the
      # instance to everybody on the internet. So by default we uncomment the
      # following bind directive, that will force Redis to listen only into
      # the IPv4 lookback interface address (this means Redis will be able to
      # accept connections only from clients running into the same computer it
      # is running).
      #
      # IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES
      # JUST COMMENT THE FOLLOWING LINE.
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      bind 127.0.0.1 192.168.1.1 // Add the actual IP address of host A.
      

      Start the Redis service.

      systemctl start redis
      
      3.1.2.4 Deploying aops-zeus

      Install aops-zeus.

      yum install aops-zeus
      

      Modify the configuration file.

      vim /etc/aops/zeus.ini
      

      Change the IP address of each service in the configuration file to the actual IP address. In this document, aops-zeus is deployed on host B. Therefore, you need to set the IP address to the IP address of host B.

      [zeus]
      ip=192.168.1.2  // Change the IP address to the actual IP address of host B.
      port=11111
      
      [uwsgi]
      wsgi-file=manage.py
      daemonize=/var/log/aops/uwsgi/zeus.log
      http-timeout=600
      harakiri=600
      processes=2     // Generate a specified number of workers or processes.
      gevent=100      // Number of gevent asynchronous cores
      
      [mysql]
      ip=192.168.1.1  // Change the IP address to the actual IP address of host A.
      port=3306
      database_name=aops
      engine_format=mysql+pymysql://@%s:%s/%s
      pool_size=100
      pool_recycle=7200
      
      [prometheus]
      ip=192.168.1.1  // Change the IP address to the actual IP address of host A.
      port=9090
      query_range_step=15s
      
      [agent]
      default_instance_port=8888
      
      [redis]
      ip=192.168.1.1  // Change the IP address to the actual IP address of host A.
      port=6379
      [diana]
      ip=192.168.1.2  // Change the IP address to the actual IP address of host B.
      port=11112
      
      [apollo]
      ip=192.168.1.2  // Change the IP address to the actual IP address of host B.
      port=11116
      

      Start the aops-zeus service.

      systemctl start aops-zeus
      
      3.1.2.5 Deploying aops-hermes

      Install aops-hermes.

      yum install aops-hermes
      

      Modify the configuration file. Because all services are deployed on host B, set the IP address of each service accessed by aops-web to the actual IP address of host B.

      vim /etc/nginx/aops-nginx.conf
      

      Some service configurations:

      
              # Ensure that Nginx still uses index.html as the entry when the front-end route changes.
              location / {
                  try_files $uri $uri/ /index.html;
                  if (!-e $request_filename){
                      rewrite ^(.*)$ /index.html last;
                  }
              }
      
              location /api/ {
                  proxy_pass http://192.168.1.2:11111/; // Change it to the actual IP address of the host where aops-zeus is deployed.
              }
      
              location /api/domain {
                  proxy_pass http://192.168.1.2:11114/; // Enter the IP address of gala-ragdoll.
                  rewrite ^/api/(.*) /$1 break;
              }
      
              location /api/check {
                  proxy_pass http://192.168.1.2:11112/; // Enter the IP address of aops-diana running in configurable mode.
                  rewrite ^/api/(.*) /$1 break;
              }
      
              location /api/vulnerability {
                  proxy_pass http://192.168.1.2:11116/; // Enter the IP address of aops-apollo.
                  rewrite ^/api/(.*) /$1 break;
              }
      

      Enable the aops-hermes service.

      systemctl start aops-hermes
      

      3.2 CVE Management

      The CVE management function is implemented based on the aops-zeus service. Therefore, you need to deploy aops-zeus and then aops-apollo.

      The running of the aops-apollo service requires the support of the MySQL and Elasticsearch databases.

      3.2.1 Node Information

      HostIP AddressModule
      Host A192.168.1.1MySQL, Elasticsearch, Redis
      Host B192.168.1.2aops-zeus, aops-apollo, aops-hermes, aops-vulcanus
      Host C192.168.1.3aops-ceres

      3.2.2 Deployment Procedure

      3.2.2.1 Deploying Basic Services

      See 3.1 Host Management.

      3.2.2.2 Deploying Elasticsearch

      Configure the repository for Elasticsearch.

      echo "[aops_elasticsearch]
      name=Elasticsearch repository for 7.x packages
      baseurl=https://artifacts.elastic.co/packages/7.x/yum
      gpgcheck=1
      gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
      enabled=1
      autorefresh=1
      type=rpm-md" > "/etc/yum.repos.d/aops_elascticsearch.repo"
      

      Install Elasticsearch using Yum.

      yum install elasticsearch-7.14.0-1
      

      Modify the Elasticsearch configuration file.

      vim /etc/elasticsearch/elasticsearch.yml
      
      # ------------------------------------ Node ------------------------------------
      #
      # Use a descriptive name for the node:
      #
      node.name: node-1
      
      # ---------------------------------- Network -----------------------------------
      #
      # By default Elasticsearch is only accessible on localhost. Set a different
      # address here to expose this node on the network:
      #
      # Change the value to the actual IP address of host A.
      network.host: 192.168.1.1
      #
      # By default Elasticsearch listens for HTTP traffic on the first free port it
      # finds starting at 9200. Set a specific HTTP port here:
      #
      http.port: 9200
      #
      # For more information, consult the network module documentation.
      #
      
      # --------------------------------- Discovery ----------------------------------
      #
      # Pass an initial list of hosts to perform discovery when this node is started:
      # The default list of hosts is ["127.0.0.1", "[::1]"]
      #
      #discovery.seed_hosts: ["host1", "host2"]
      #
      # Bootstrap the cluster using an initial set of master-eligible nodes:
      #
      cluster.initial_master_nodes: ["node-1"]
      # Cross-domain configurations
      http.cors.enabled: true
      http.cors.allow-origin: "*"
      #
      

      Restart the Elasticsearch service.

      systemctl restart elasticsearch
      
      3.2.2.3 Deploying aops-apollo

      Install aops-apollo.

      yum install aops-apollo
      

      Modify the configuration file.

      vim /etc/aops/apollo.ini
      

      Change the IP address of each service in the apollo.ini to the actual IP address.

      [apollo]
      ip=192.168.1.2// Change it to the actual IP address of host B.
      port=11116
      host_vault_dir=/opt/aops
      host_vars=/opt/aops/host_vars
      
      [zeus]
      ip=192.168.1.2 // Change it to the actual IP address of host B.
      port=11111
      
      # herms info is used to send mail.
      [hermes]
      ip=192.168.1.2  // Change it to the actual IP address of aops-hermes, for example, the IP address of host B.
      port=54795      // Change it to the actual port of the hermes service.
      
      [cve]
      cve_fix_function=yum
      # value between 0-23, for example, 2 means 2:00 in a day.
      cve_scan_time=2
      
      [mysql]
      ip=192.168.1.1 // Change it to the actual IP address of host A.
      port=3306
      database_name=aops
      engine_format=mysql+pymysql://@%s:%s/%s
      pool_size=100
      pool_recycle=7200
      
      [elasticsearch]
      ip=192.168.1.1 // Change it to the actual IP address of host A.
      port=9200
      max_es_query_num=10000000
      
      [redis]
      ip=192.168.1.1 // Change it to the actual IP address of host A.
      port=6379
      
      [uwsgi]
      wsgi-file=manage.py
      daemonize=/var/log/aops/uwsgi/apollo.log
      http-timeout=600
      harakiri=600
      processes=2
      gevent=100
      

      Start the aops-apollo service.

      systemctl start aops-apollo
      

      3.3 Exception Detection

      The exception detection function is implemented based on the aops-zeus service. Therefore, you need to deploy aops-zeus and then aops-diana.

      Considering distributed deployment, the aops-diana service must be deployed on both host A and host B to act as the producer and consumer in the message queue, respectively.

      The running of the aops-diana service requires the support of MySQL, Elasticsearch, Kafka, and Prometheus.

      3.3.1 Node Information

      HostIP AddressModule
      Host A192.168.1.1MySQL, Elasticsearch, Kafka, Prometheus, aops-diana
      Host B192.168.1.2aops-zeus, aops-diana, aops-hermes, aops-vulcanus
      Host C192.168.1.3aops-ceres, gala-gopher

      3.3.2 Deployment Procedure

      3.3.2.1 Deploying Basic Services

      See 3.1 Host Management.

      3.3.2.2 Deploying Elasticsearch

      See 3.2.2.2 Deploying Elasticsearch.

      3.3.2.3 Deploying Kafka

      Kafka uses ZooKeeper to manage and coordinate agents. Therefore, you need to deploy ZooKeeper when deploying Kafka.

      3.3.2.3.1 Installing ZooKeeper

      Install ZooKeeper.

      yum install zookeeper
      

      Start the ZooKeeper service.

      systemctl start zookeeper
      
      3.3.2.3.2 Installing Kafka

      Install Kafka.

      yum install kafka
      

      Modify the configuration file.

      vim /opt/kafka/config/server.properties
      

      Change the value of listeners to the IP address of the local host.

      ############################# Socket Server Settings #############################
      
      # The address the socket server listens on. It will get the value returned from
      # java.net.InetAddress.getCanonicalHostName() if not configured.
      #   FORMAT:
      #     listeners = listener_name://host_name:port
      #   EXAMPLE:
      #     listeners = PLAINTEXT://your.host.name:9092
      listeners=PLAINTEXT://192.168.1.1:9092
      

      Start the Kafka service.

      cd /opt/kafka/bin
      nohup ./kafka-server-start.sh ../config/server.properties &
      tail -f ./nohup.out  # Check all the outputs of nohup. If the IP address of host A and the Kafka startup success INFO are displayed, Kafka is started successfully.
      
      3.3.2.4 Deploying aops-diana

      The aops-diana installation process on host A is the same as that on host B.

      Install aops-diana.

      yum install aops-diana
      

      Modify the configuration file. The aops-dianas on host A and host B play different roles, which are distinguished based on the differences in the configuration file.

      vim /etc/aops/diana.ini
      

      (1) Start aops-diana on host A in executor mode. It functions as the consumer in the Kafka message queue. The configuration file to be modified is as follows:

      [diana]
      ip=192.168.1.1  // Change the IP address to the actual IP address of host A.
      port=11112
      mode=executor  // This mode is the executor mode. It is used as the executor in common diagnosis mode and functions as the consumer in Kafka.
      timing_check=on
      
      [default_mode]
      period=60
      step=60
      
      [elasticsearch]
      ip=192.168.1.1  // Change the IP address to the actual IP address of host A.
      port=9200
      max_es_query_num=10000000
      
      
      [mysql]
      ip=192.168.1.1  // Change the IP address to the actual IP address of host A.
      port=3306
      database_name=aops
      engine_format=mysql+pymysql://@%s:%s/%s
      pool_size=10000
      pool_recycle=7200
      
      [redis]
      ip=192.168.1.1  // Change the IP address to the actual IP address of host A.
      port=6379
      
      
      [prometheus]
      ip=192.168.1.1  // Change the IP address to the actual IP address of host A.
      port=9090
      query_range_step=15s
      
      [agent]
      default_instance_port=8888
      
      [zeus]
      ip=192.168.1.2  // Change the IP address to the actual IP address of host B.
      port=11111
      
      [consumer]
      kafka_server_list=192.168.1.1:9092  // Change the IP address to the actual IP address of host A.
      enable_auto_commit=False
      auto_offset_reset=earliest
      timeout_ms=5
      max_records=3
      task_name=CHECK_TASK
      task_group_id=CHECK_TASK_GROUP_ID
      result_name=CHECK_RESULT
      
      [producer]
      kafka_server_list = 192.168.1.1:9092  // Change the IP address to the actual IP address of host A.
      api_version = 0.11.5
      acks = 1
      retries = 3
      retry_backoff_ms = 100
      task_name=CHECK_TASK
      task_group_id=CHECK_TASK_GROUP_ID
      
      [uwsgi]
      wsgi-file=manage.py
      daemonize=/var/log/aops/uwsgi/diana.log
      http-timeout=600
      harakiri=600
      processes=2
      threads=2
      

      (2) Start aops-diana on host B in configurable mode. It functions as the producer in the Kafka message queue. The aops-diana port configuration in the aops-hermes file is subject to the IP address and port number of this host. The configuration file to be modified is as follows:

      [diana]
      ip=192.168.1.2  // Change the IP address to the actual IP address of host B.
      port=11112
      mode=configurable  // This mode is the configurable mode. It is used as a scheduler in common diagnosis mode and functions as the producer.
      timing_check=on
      
      [default_mode]
      period=60
      step=60
      
      [elasticsearch]
      ip=192.168.1.1  // Change the IP address to the actual IP address of host A.
      port=9200
      max_es_query_num=10000000
      
      [mysql]
      ip=192.168.1.1  // Change the IP address to the actual IP address of host A.
      port=3306
      database_name=aops
      engine_format=mysql+pymysql://@%s:%s/%s
      pool_size=100
      pool_recycle=7200
      
      [redis]
      ip=192.168.1.1  // Change the IP address to the actual IP address of host A.
      port=6379
      
      [prometheus]
      ip=192.168.1.1  // Change the IP address to the actual IP address of host A.
      port=9090
      query_range_step=15s
      
      [agent]
      default_instance_port=8888
      
      [zeus]
      ip=192.168.1.2  // Change the IP address to the actual IP address of host B.
      port=11111
      
      [consumer]
      kafka_server_list=192.168.1.1:9092  // Change the IP address to the actual IP address of host A.
      enable_auto_commit=False
      auto_offset_reset=earliest
      timeout_ms=5
      max_records=3
      task_name=CHECK_TASK
      task_group_id=CHECK_TASK_GROUP_ID
      result_name=CHECK_RESULT
      
      [producer]
      kafka_server_list = 192.168.1.1:9092  // Change the IP address to the actual IP address of host A.
      api_version = 0.11.5
      acks = 1
      retries = 3
      retry_backoff_ms = 100
      task_name=CHECK_TASK
      task_group_id=CHECK_TASK_GROUP_ID
      
      [uwsgi]
      wsgi-file=manage.py
      daemonize=/var/log/aops/uwsgi/diana.log
      http-timeout=600
      harakiri=600
      processes=2
      threads=2
      

      Start the aops-diana service.

      systemctl start aops-diana
      

      3.4 Configuring Source Tracing

      A-Ops configuration source tracing depends on gala-ragdoll based on aops-zeus. Therefore, you need to deploy aops-zeus and then gala-ragdoll.

      3.4.1 Node Information

      HostIP AddressModule
      Host A192.168.1.1MySQL, aops-tools
      Host B192.168.1.2aops-zeus, gala-ragdoll, aops-hermes, aops-vulcanus
      Host C192.168.1.3aops-ceres

      3.4.2 Deployment Procedure

      3.4.2.1 Deploying Basic Services

      See 3.1 Host Management.

      3.4.2.2 Deploying gala-ragdoll

      Install gala-ragdoll.

      yum install gala-ragdoll
      

      Modify the configuration file.

      vim /etc/ragdoll/gala-ragdoll.conf
      

      Change the IP address in collect_address of the collect section to the IP address of host B, and change the values of collect_api and collect_port to the actual API and port number.

      [git]
      git_dir = "/home/confTraceTest"
      user_name = "user_name"
      user_email = "user_email"
      
      [collect]
      collect_address = "http://192.168.1.2"    // Change it to the actual IP address of host B.
      collect_api = "/manage/config/collect"    // The value is an example. Change it to the actual value.
      collect_port = 11111                      // Change it to the actual port number of the aops-zeus service.
      
      [sync]
      sync_address = "http://192.168.1.2"
      sync_api = "/demo/syncConf"
      sync_port = 11114
      
      
      [ragdoll]
      port = 11114
      

      Start the gala-ragdoll service.

      systemctl start gala-ragdoll
      

      3.5 One-Click Deployment

      A-Ops supports one-click deployment based on the script provided by aops-tools. For details, see One-Click Deployment of A-Ops

      4 Client Deployment

      The client depends on the aops-ceres module. Some data collection operations depend on the gala-gopher module. For details about how to deploy the aops-ceres module, see Deploying aops-ceres.

      5 FAQs

      1. The aops-zeus management center communicates with other modules through HTTP.
      2. If the firewall cannot be disabled, enable all interfaces involved in the service deployment process. Otherwise, the services cannot be accessed, affecting the normal use of A-Ops.
      3. For security purposes, the client initiates the operation of adding new hosts in aops-zeus. For details, see section "Registering with aops-zeus" in Deploying aops-ceres.
      4. The aops-diana, gala-ragdoll, and aops-apollo modules are independent of each other. You can deploy them as required.
      5. All services need to be deployed on the server and client. For details about how to deploy the server, see the specific function requirements. For details about how to deploy the client, see aops-ceres and gala-gopher.
      6. Configure Elasticsearch cross-domain settings based on actual requirements.
      7. Some HTTP access interfaces are synchronous. When the execution time is long, the web client may return a 504 error. You can set a relatively large number for proxy_read_timeout in the Nginx configurations to reduce the probability of the error. 8.When hosts are added in batches, the service execution is restricted by the maximum number of connections (MaxStartups) of the sshd service on the aops-zeus host. If some hosts cannot be connected, consider temporarily increasing the value of MaxStartups by referring to this document.

      Bug Catching

      Buggy Content

      Bug Description

      Submit As Issue

      It's a little complicated....

      I'd like to ask someone.

      PR

      Just a small problem.

      I can fix it online!

      Bug Type
      Specifications and Common Mistakes

      ● Misspellings or punctuation mistakes;

      ● Incorrect links, empty cells, or wrong formats;

      ● Chinese characters in English context;

      ● Minor inconsistencies between the UI and descriptions;

      ● Low writing fluency that does not affect understanding;

      ● Incorrect version numbers, including software package names and version numbers on the UI.

      Usability

      ● Incorrect or missing key steps;

      ● Missing prerequisites or precautions;

      ● Ambiguous figures, tables, or texts;

      ● Unclear logic, such as missing classifications, items, and steps.

      Correctness

      ● Technical principles, function descriptions, or specifications inconsistent with those of the software;

      ● Incorrect schematic or architecture diagrams;

      ● Incorrect commands or command parameters;

      ● Incorrect code;

      ● Commands inconsistent with the functions;

      ● Wrong screenshots.

      Risk Warnings

      ● Lack of risk warnings for operations that may damage the system or important data.

      Content Compliance

      ● Contents that may violate applicable laws and regulations or geo-cultural context-sensitive words and expressions;

      ● Copyright infringement.

      How satisfied are you with this document

      Not satisfied at all
      Very satisfied
      Submit
      Click to create an issue. An issue template will be automatically generated based on your feedback.
      Bug Catching
      编组 3备份