Page tree

For Moogsoft AIOps 7.2 and later, see
For Moogsoft AIOps 7.1 and prior, see

Skip to end of metadata
Go to start of metadata

Moogsoft AIOps Installation and Upgrade

YUM: Not installing correct version

If, when attempting to install Moogsoft AIOps, an incorrect or outdated version is offered then your YUM cache may need cleaning.

Try running:

yum clean all

and then attempt the install again.

YUM: HTTP Error 401 - Unauthorized

If an attempt to install Moogsoft AIOps fails with an error such as:

https://<username>:<password> [Errno 14] HTTP Error 401 - Unauthorized

then check your <username> and <password> credentials are correct in the configured Moogosft yum repository.

YUM: problem making ssl connection

If an attempt to install Moogsoft AIOps fails with an error such as:

https://<username>:<password> [Errno 14] problem making ssl connection
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: moogsoft-aiops. Please verify its path and try again

then you may need to update the nss packages on your server. Try running:

yum -y update nss

and then attempt the install again.

YUM: MySQL Conflict

If an attempt to install Moogsoft AIOps fails with an error such as:

Running rpm_check_debug
Running Transaction Test
Transaction Check Error:
  file /usr/lib64/mysql/ from install of mysql-community-libs-compat-5.7.22-2.el6.x86_64 conflicts with file from package compat-mysql51-5.1.54-1.el6.remi.x86_64
  file /usr/lib64/mysql/ from install of mysql-community-libs-compat-5.7.22-2.el6.x86_64 conflicts with file from package compat-mysql51-5.1.54-1.el6.remi.x86_64
Error Summary

then this is caused by a conflict with the MySQL libraries currently on the host.

Run the following bash commands to allow the product to be installed successfully:

echo "remove compat-mysql51" > /tmp/moog_yum_shell.txt
echo "install mysql-community-libs-compat-5.7.22" >> /tmp/moog_yum_shell.txt
echo "install mysql-community-client-5.7.22" >> /tmp/moog_yum_shell.txt
echo "install mysql-community-libs-5.7.22" >> /tmp/moog_yum_shell.txt
echo "install mysql-community-server-5.7.22" >> /tmp/moog_yum_shell.txt
echo "install mysql-community-common-5.7.22" >> /tmp/moog_yum_shell.txt
echo "groupinstall moogsoft" >> /tmp/moog_yum_shell.txt
echo "run" >> /tmp/moog_yum_shell.txt

cat /tmp/moog_yum_shell.txt | yum shell -y

The above error is most likely to occur on hosts which already have some MySQL components installed. i.e.the issue will most likely be seen when running the command yum install moogsoft-db on a system with an existing MySQL install.

Nginx install issues

Error: Package: moogsoft-ui-6.0.0-123.x86_64 (moogsoft-aiops) Requires: nginx >= 1.14.0

If you encounter the following error when attempting to install Moogsoft AIOps:

Requires: nginx >= 1.14.0
---> Package moogsoft-ui.x86_64 0:6.0.0-123 will be an update
--> Processing Dependency: nginx >= 1.14.0 for package: moogsoft-ui-6.0.0-123.x86_64
--> Finished Dependency Resolution
Error: Package: moogsoft-ui-6.0.0-123.x86_64 (moogsoft-aiops)
Requires: nginx >= 1.14.0
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest

then the nginx repo itself can be installed manually with the following command:

rpm -Uvh

The install can now take place.

Single-Host Installation for Non-Root 

If you cannot access the UI from your host machine, check your firewall and if you're listening on the right ports:

  1. To see if your firewall is enabled: 


    If your firewall is disabled it returns:

    SELinux status:                 disabled
  2.  To disable an active firewall run:

    setenforce 0
  3.  To check whether a port is open you can run:

    firewall-cmd --zone=public --query-port=8443/tcp
    firewall-cmd --zone=public --query-port=8080/tcp
  4. To open these ports run: 

    firewall-cmd --permanent --zone=public --add-port=8080/tcp
    firewall-cmd --permanent --zone=public --add-port=8443/tcp
    firewall-cmd --reload

Mobile Troubleshooting

Moogsoft AIOps includes a self-signed certificate by default. If you want to use Moogsoft AIOps for Mobile on an iPhone, you need to add a valid SSL certificate. This is because WebSockets do not work on iOS with self-signed certificates.

If a valid root CA certificate is not added, a 'Connection Error' appears at login and AIOps for Mobile does not work.

Nginx SSL Configuration

To apply a valid certificate to Nginx, go to the nginx config folder and edit moog-ssl.conf :

cd common/config/nginx vi moog-ssl.conf
vi moog-ssl.conf

Change the default self-signed certificate and key locations to point to the valid root certificate and key:

#ssl_certificate /etc/nginx/ssl/certificate.pem;
#ssl_certificate_key /etc/nginx/ssl/certificate.key;

ssl_certificate /etc/certificates/GeoTrust_Universal_CA.crt;
ssl_certificate_key /etc/certificates/GeoTrust_Universal_CA.key;

Restart Nginx with this command:

systemctl restart nginx

Moogsoft AIOps Processes

Required services for a functional production system

Service name



Web server that contains the servlets that provide the Moogsoft AIOps User interface

nginxWeb server that handles security, such as Moogsoft AIOps login, php and https/ssl implementation

Link Access Modules used for data ingestion. Service names may differ
At least one instance of a LAM is required for data feed. See Data ingestion
moogfarmdCore Moogsoft AIOps system application. See Event processing

Database containing Moogsoft AIOps data (moogdb and moog_reference schemas, etc.)


Message system for Moogsoft AIOps. See Core operation - MooMS


Elasticsearch service for UI search feature.

To check the status, stop, start or restart a service run one of the following:

service <service-name> status
service <service-name> stop
service <service-name> start
service <service-name> restart

Location of installation and log files

See Moogsoft AIOps Component Logs.

Generic Moogsoft AIOps process not starting

No space left on the disk

  • Check the file system with the following command:
    df -m
  • Look for partitions that are full

<Service> not found
Error while loading shared libraries

  • Environmental variables may not be properly set up for your shell
  • Run the environment and check the location set for $MOOGSOFT_HOME
  • For information on setting the AIOps shell environment, see Installation and Upgrade guide

moogfarmd not starting

Configuration parsing error

  • +|No config present|+ message in /var/log/moogsoft/moogfarmd.log
  • Points to a syntax error in $MOOGSOFT_HOME/config/moog_farmd.conf
  • Check the config file for punctuation mistakes

    Look for:
    • Missing commas
    • Unbalanced quotes
    • Missing {' or '} 
    Use # to comment out code instead of /* and */

  • Edit moog_farmd.conf and then restart the service

Rabbitmq related

Also see Troubleshooting MooMS deployment.

Rabbitmq not starting - "No such user"

  • No such user message in /var/log/rabbitmq/startup_err
  • Check /etc/passwd for user rabbitmq with the following command:
    grep rabbitmq /etc/passwd
  • If no user is found, add the following to /etc/passwd
    rabbitmq:x:491:488:RabbitMQ messaging server:/var/lib/rabbitmq:/bin/bash

Rabbitmq not starting - "Failed to create aux thread"

  • Failed to create aux thread message in /var/log/rabbitmq/startup_err
  • This is most likely a ulimit issue for the rabbitmq user

  • Check ulimits for the rabbitmq user by running (as root):

    [root@ldev04 ~]# su - rabbitmq
    bash-4.1$ ulimit -a
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 515675
    max locked memory       (kbytes, -l) 64
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 1024
    pipe size            (512 bytes, -p) 8
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 10240
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 1024
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited
  • The above example shows ulimits that are likely too low for RabbitMQ

  • As per instructions here it may be appropriate to increase ulimits for "open files" and "max user processes" to at least 4096 for Development/QA environments and 65536 for Production environments

Unable to create RabbitMQ connection

  • + Unable to create RabbitMQ connection+ message appearing in LAM, moogfarmd or apache-tomcat logs (see above for locations)
  • This indicates that the process unable to connect to the MooMS zone in rabbitmq
  • Check that the rabbitmq server is running with the following command:
    service rabbitmq-server status
  • If service is not running start it with the following command: 
    service rabbitmq-server start
  • If the service is running, check that the zone used in Moogsoft AIOps matches the vhost in rabbitmq
    List the zones (vhosts) added in rabbitmq with the following command:
    rabbitmqctl list_vhosts
    Check the MooMS section in /usr/share/moogsoft/config/system.conf for the zone used in Moogsoft AIOps
  • If the zone is missing, add the zone (vhost) to rabbitmq manually (see Troubleshooting MooMS deployment)
  • Restart the process affected

LAMs fail to start from command line

If LAMs run from the command line or as a service result in the following error:

[root@moogbox2 bin]# ./socket_lam 
./socket_lam: error while loading shared libraries: cannot open shared object file: No such file or directory

it may be because /usr/java/jdk1.8.0_171/jre/lib/amd64/server has not been added to the LD_LIBRARY_PATH.

  • To run the LAMs via a command line, a change t he  LD_LIBRARY_PATH  to be as follows (t he default  initd  files have this modification made ):
export LD_LIBRARY_PATH=$MOOGSOFT_HOME/lib:/usr/GNUstep/Local/Library/Libraries:/usr/GNUstep/System/Library/Libraries:$JAVA_HOME/jre/lib/amd64/server

Generic LAM not starting

Configuration parsing error

  • + |Unable to parse configuration file + message in /var/log/moogsoft/<lamd_name>.log
  • This indicates a syntax error in the LAM configuration file
  • Check the config file for syntax mistakes

    Look for:
    • Missing commas
    • Unbalanced quotes

  • Compare the config file to a default config file for the same LAM. Use the following command to locate differences in the files:
    diff -y <current_lam>.conf <default_lam>.conf | less
  • Edit <current_lam>.conf to resolve any syntax errors and restart the LAM

Connection refused error

  •  +failed to connect to [host:<port>]: Connection refused|+ message in /var/log/moogsoft/<lamd_name>.log
  • This means that the port specified in the LAM configuration file is already in use
  • Use the following command to check that the LAM is not already started: 
    ps -ef | grep <lamd_name>
  • Check that another process is not already bound to the port
  • If required, edit the port setting for the LAM in the config file and restart the LAM

Unresolvable hostname error

  • +|Host [hostname] unresolvable|+ message in /var/log/moogsoft/<lamd_name>.log
  • The LAM is unable to resolve the host, or the hostname is incorrectly set
  • In the LAM config file, check the address setting, and correct any errors
  • Check that /etc/hosts has an entry for the hostname specified

Failed to open file error

  • +|Failed to open file [<path_to_file>|+ message in /var/log/moogsoft/<lamd_name>.log
  • The LAM is unable to locate a file specified in the LAM configuration file
  • Locate the missing file in $MOOGSOFT_HOME/bots/lambots or $MOOGSOFT_HOME/contrib
  • Update the LAM config file with the correct file path, and restart the LAM

Generic LAM not processing

Syntax error in Presend filter

  • In the LAM configuration file, locate the presend filter file name
  • Either 
    With a javascript editor, check the code to locate any syntax errors,
    Or, if installed, run: 
    node $MOOGSOFT_HOME/bots/lambots/<path_to_filter_file>.js to locate the incorrect line in the code
  • Edit $MOOGSOFT_HOME/bots/lambots/<path_to_filter_file>.js to resolve the error, and restart the LAM

Empty columns in alert lists

  • Possibly indicates incorrect field mapping assignments
  • Check field mappings at the bottom of the LAM configuration file
  • Edit the config file to properly map the field to the column name, and then restart the LAM to test

Socket LAM not processing

Processing mode set incorrectly

  • The LAM may be set to Server mode rather than Client in the LAM configuration file. For an explanation of mode types, see Socket LAM 
  • Set the mode accordingly in the config file and restart the LAM

JSON feed not processing

  • The JSON string is incorrectly formatted. For example, event data contains nested JSON
  • Run the LAM in debug mode and look for nested JSON
  • To resolve, either modify the event data, or edit the presend filter to match values in the nested JSON

Logfile LAM not starting

No input file

  • +|Could not stat file [-1] error: [Bad file descriptor]|+ message in /var/log/moogsoft/<lamd_name>.log
  • The Logfile LAM cannot locate the log file to read
  • In the LAM configuration file, check the target setting and confirm the file path to the target log file

REST LAM not starting

Missing SSL path

  • +|No file path specified|+ message in /var/log/moogsoft/<lamd_name>.log
  • In the LAM configuration file, check the use_ssl  setting
  • If using SSL, (use_ssl set to true), then check the following are properly set:
  • If not using SSL, set use_ssl to false

Nginx fails on startup if IPv6 not configured

Comment out the following references in two conf files:

  1. go to /etc/nginx/conf.d 
  2. edit out IPv6 references with a  #:

    conf filesection
    moog-default.conf listen       80 default_server;
     #listen       [::]:80 default_server;
    moog-ssl.conf listen       443 ssl default_server;
    # listen       [::]:443 ssl;

User Interface (UI) issues

Unavailable UI Login Page

  • Check that port 443 is not being blocked by the firewall on the server.
  • Check that the nginx service is running with command:

    [root@moogbox2 ~]# service nginx status
    nginx (pid  42356) is running
  • Check that nginx is listening on port 443 - example expected output:

[root@moogbox2 ~]# netstat -anp|grep 443
tcp        0      0       *                   LISTEN      42356/nginx         
tcp        0      0 :::443                      :::*                        LISTEN      42356/nginx 

Login fails with "You could not be logged in. Please try again."

Apache-tomcat service not running

  • Check the apache-tomcat service is running with the following command:
    service apache-tomcat status

Communication problem between the UI and MySQL database

  • Check the MySQL service is running with the following command:
    service mysqld status
  • If MySQL is running on a different server, is it reachable from the Moog AIOps web server and have the required grants been applied?

Authentication problem between the UI and MySQL database

  • Does your user exist in the system?
    Check in the MySQL moogdb.users table
  • Is the username and password correct?

Search/Elasticsearch Troubleshooting

See Search and Indexing.

ElasticSearch not running or generating errors (such as MySQL connection problems)

  • Check that the elasticsearch service is running:

    [root@moogbox2 ~]# service elasticsearch status
    elasticsearch (pid  39596) is running
  • Any errors will be written to /var/log/elasticsearch/elasticsearch.log

Tomcat cannot connect to ElasticSearch

  • Check /usr/share/apache-tomcat/logs/catalina.out for any errors when attempting a search from the UI

Cron job errors

  • Check that cron job that runs the moog_indexer (created by the script to re-index against the Moogsoft AIOps database on a once-a-minute basis) exists and is not generating any warnings or errors
  • Running crontab -l will list the configured cron jobs
  • Errors are in /var/log/cron

  • Depending on the interval at which Elastic re-indexes against the Moogsoft AIOps database, it is possible that new Alerts, Situations, Thread or Comments have not yet been indexed, and so will not be searchable 

  • To change the interval manually, run the following command:
    crontab –ed

Elasticsearch fails to start with /tmp directory permission problems 

Elasticsearch fails to start with "java.lang. UnsatisfiedLinkError: /tmp/jna--<blah>" error. For example:

[2017-08-07T14:14:31,173][WARN ][o.e.b.Natives            ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna3872404023206022895.tmp: /tmp/jna--1985354563/jna3872404023206022895.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_171]
        at java.lang.ClassLoader.loadLibrary0( ~[?:1.8.0_171]
        at java.lang.ClassLoader.loadLibrary( ~[?:1.8.0_171]
        at java.lang.Runtime.load0( ~[?:1.8.0_171]
        at java.lang.System.load( ~[?:1.8.0_171]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath( ~[jna-4.2.2.jar:4.2.2 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary( ~[jna-4.2.2.jar:4.2.2 (b0)]
        at com.sun.jna.Native.<clinit>( ~[jna-4.2.2.jar:4.2.2 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_171]
        at java.lang.Class.forName( ~[?:1.8.0_171]
        at org.elasticsearch.bootstrap.Natives.<clinit>( [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives( [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.bootstrap.Bootstrap.setup( [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.bootstrap.Bootstrap.init( [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.bootstrap.Elasticsearch.init( [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.bootstrap.Elasticsearch.execute( [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.cli.SettingCommand.execute( [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling( [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.cli.Command.main( [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.bootstrap.Elasticsearch.main( [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.bootstrap.Elasticsearch.main( [elasticsearch-5.6.9.jar:5.6.9]

This is most likely due to the /tmp mount having the noexec directive.



a) Remove the noexec directive from the /tmp mount (if practical to do so)

sudo mount /tmp -o remount,exec


b) In file: /etc/sysconfig/elasticsearch set:


Restart the elasticsearch service after either of the above changes. 

Processing Issues

No Alerts created

License not applied

    • Ensure product license has been applied.

Rabbitmq not running

    • Check that the rabbitmq server is running with the following command:
      service rabbitmq-server status

AlertBuilder not started

    • Check that the run_on_startup setting for AlertBuilder in $MOOGSOFT_HOME/config/moog_farmd.conf is set to true
    • Check /var/log/moogsoft/moogfarmd.log 

AlertBuilder misconfiguration

    • Check the AlertBuilder Moolet for syntax errors or errors in logic

LAM misconfiguration

    • Check that the LAM is correctly parsing/mapping the data feed
    • Check that the LAM is not doing any post-event processing that may be filtering out the events out in the associated LAMBot

No Situations Created

License not applied

    • Ensure product license has been applied

Sigaliser Moolet not running

    • In $MOOGSOFT_HOME/config/moog_farmd.conf check that the run_on_startup setting for the sigaliser used is set to true

Incorrectly set process_ouput_of

    • In $MOOGSOFT_HOME/config/moog_farmd.conf check that the process_output_of setting for the sigaliser used is listing the correct Moolet (for example, AlertBuilder)
    • Also check that the Moolet listed under process_output_of is running

Sigaliser Moolet configuration is too restrictive or too open

If Moolet settings are too restrictive or too open they may not produce Situations. See Sigaliser Moolet for more information

Error Messages & Status Codes

HTTP Status Codes

The following list describes the HTTP status codes used by AIOps:



Application Usage



This status will be sent to the browser when a request has been processed ‘successfully’.



This status will be sent to the browser when the request was well-constructed but the server was unable to process it.


Bad Request

This status will be sent to the browser when the request parameters are invalid or missing.



This status code returned when the user is not authenticated.



This status code returned when the user is forbidden from performing a specific action.


Not Found

This status code will be sent to browser when a requested servlet path could not be found.



Some conflict has occurred that can potentially be resolved by the user e.g. change a template name


Internal Server Error

This status will be sent to the browser when there is some unexpected exception or problem on the server side.


Service Unavailable

Used when the server is overloaded (currently used by tool runner when no execution threads are available)

Application Status Codes

The application status codes present in the response payload are defined in the following ranges:

Please note: One range may map to more than one HTTP status

Code Range


HTTP Status


System Errors

500, 503


Validation Errors



Security Errors

401, 403


Conflict Errors


System Error Status Codes

The following system error status codes are currently defined:



HTTP Status


General server error



Service Unavailable



DB Unavailable



Service Busy


Validation Error Status Codes

The following validation error status codes are currently defined:



HTTP Status


General validation error.  The actual parameter specific error will be defined in the additional body.



Parameter missing (not included in post/get) or no value provided (parameter included but empty value)



Parameter format error (a value that could not be converted to desired type)



Illegal value (a value that is not in the list of allowed values)



Parameter out of range (a value that is outside the range of allowed values)


Security Error Status Codes

The following security error status codes are currently defined:



HTTP Status


General security error



User not authenticated



User not permitted to perform action



Header auth token not authenticated


Conflict Error Status Codes

The following conflict error status codes are currently defined:



HTTP Status


General conflict error



Duplicate name e.g. Duplicate template name, Invite already sent


Content Error Status Codes

The following content error status codes are currently defined:



HTTP Status


General content error



Content not supported



Error decoding content



Content accepted but cached, processing not guaranteed



Server being ahead of the UI (in terms of Moog Version)



UI being ahead of the UI (in terms of Moog Version)


JSON Response Body (Non-HTTP 200)

The response body will contain the detailed application error status information.  The format of the JSON is as follows:

Please note: The additional information is optional for some status codes

{ “message” : “User friendly message”,
“statusCode” : <appStatusCode>,
“additional” : <jsonAdditionalInfo> }
{ “message” : “User friendly message”,
“statusCode” : <appStatusCode> }

The high-level user friendly message will take the following format:

   “<operation> failed due to <something>”

   e.g. “Create situation failed due to invalid parameters”

JSON Addition Information

The format of the JSON additional information will depend on the the HTTP status code and application status code.  For certain status codes the additional information may not be present.  Although the HTTP status code (400 or 500) is enough to determine the additional information structure it is probably best to use the application status code (as in the future more than one range could apply to a single HTTP status code).

  • HTTP 400 -> Application Status Code 2000-2999

  • HTTP 409 -> Application Status Code 4000-4999

  • HTTP 500, 503 -> Application Status Code 1000-1999

Application Status Code [1000-1999]

Optional additional information

A debug message typically containing the exception message:

{ “debugMessage” : “ Connection refused”}

Application Status Code [2000-2999] or [4000-4999]

Mandatory additional information

List of the bad parameter(s) and the error information:

[ { “name” : “parameterName”,
   “value” : <badValue>,
  “errorCode” : <errorCode>},

Ngnix Error Messages

97: Address family not supported by protocol

This an IPv6 error. Either configure IPv6 or comment it out.

Comment out the following references in two conf files:

a. Go to /etc/nginx.conf.d

b. Edit out IPv6 references with a hash #:

Conf fileSection
moog-default.conflisten       80 default_server;
 #listen       [::]:80 default_server;
moog-ssl.conflisten       443 ssl default_server;
# listen       [::]:443 ssl;

502 Bad Gateway (nginx/1.14.0)

Situation: Fails to load resource and server responds with a status of 502 (Bad Gateway).

Resolution: Check that you have used the correct components for the version you are installing.

Please note: The SERVER 2 components changed slightly between AIOps versions 6.0.0 and 6.1.0 -z MY_ZONE -bz MY_ZONE -d SERVER1:3306 -sd SERVER1:3306 -bz MY_ZONE -d SERVER1:3306 -otwxfz MY_ZONE -d SERVER1:3306
  • No labels