PNP4Nagios installation and integration with Nagios @CIGNEX Datamatics

Hello guys,

I would like to share how to install PNP4Nagios and integrate with nagios to see graphs.
With nagios if you want to configure graphs for your any of the services you need performance data as part in output.

Performance Data: Plugins can return optional performance data in their output by sending the normal, human-readable text string that they usually would, followed by a pipe character (|), and then a string containing one or more performance data metrics. Let's take the check_ping plugin as an example and assume that it has been enhanced to return percent packet loss and average round trip time as performance data metrics. 

Sample output from the plugin might look like this:

PING ok - Packet loss = 0%, RTA = 0.80 ms | percent_packet_loss=0, rta=0.80

If you are new to performance data see below link for more information:
Performance Data

Installation and configuration of PNP4Nagios:

Step1: Install pnp4nagios and rrdtool-perl package

yum install pnp4nagios* rrdtool-perl

Step2: Change some parameters in nagios Configuration file 

Edit the performance data section of /usr/local/nagios/etc/nagios.cfg:

Enable performance data by adding following line
process_performance_data=1 

Define path where performance data files should be stored.
host_perfdata_file=/var/perfdatanagios/host-perfdata
service_perfdata_file=/var/perfdatanagios/service-perfdata

Create /var/perfdatanagios/ direcrtory and change permission.

mkdir /var/perfdatanagios/

chown nagios:apache /var/perfdatanagios/

chmod 775 /var/perfdatanagios 

Now add below performance data templates lines

host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tHOSTOUTPUT::$HOSTOUTPUT$

service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$\tSERVICEOUTPUT::$SERVICEOUTPUT$

Change below parameters
host_perfdata_file_mode=a 
service_perfdata_file_mode=a 

Change interval to process perfdata 
host_perfdata_file_processing_interval=15 
service_perfdata_file_processing_interval=15 

Add commands to check perfdata 
host_perfdata_file_processing_command=process-host-perfdata-file 
service_perfdata_file_processing_command=process-service-perfdata-file

Now create commands in commands.cfg file to process performance data as below:

define command {
        command_name    process-service-perfdata-file
        command_line    /usr/local/nagios/libexec/plugins/process_perfdata.pl --bulk=/var/perfdatanagios/service-perfdata
}
define command {
        command_name    process-host-perfdata-file
        command_line    /usr/local/nagios/libexec/plugins/process_perfdata.pl --bulk=/var/perfdatanagios/host-perfdata
}
Note: use your plugin directory path.

Download process_perfdata.pl
wget https://raw.github.com/willixix/WL-NagiosPlugins/master/graphing_templates/pnp4nagios/process_perfdata.pl

Change ownership and permissions as below.
chown nagios:nagios process_perfdata.pl
chmod +x  process_perfdata.pl

change conf array in process_perfdata.pl plugin according to your configuration in /etc/pnp4nagios/process_perfdata.cfg and /etc/pnp4nagios/config.php.

Change below parameters /etc/pnp4nagios/config.php

$conf['rrdtool'] = "/usr/bin/rrdtool";

$conf['rrdbase'] = "/var/perfdatanagios/";

$conf['page_dir'] = "/etc/pnp4nagios/pages/";

find below strings in process_perfdata.pl file and change its value according to your environment.

RRDPATH            => "/var/perfdatanagios/", 

RRA_CFG            => "/etc/pnp4nagios/rra.cfg",

STATS_DIR          => "/var/log/pnp4nagios/stats",

LOG_FILE           => "/var/log/pnp4nagios/perfdata.log",

ENCRYPTION         => 0,   

find below strings in process_perfdata.cfg file and change its value according to your environment.

RRDPATH = /var/perfdatanagios/ ## [ path of perfdata directory ]

RRA_CFG = /etc/pnp4nagios/rra.cfg ## [ path of rra.cfg file]

STATS_DIR = /var/log/pnp4nagios/stats ## [ create directory /var/log/pnp4nagios/stats ]

LOG_FILE = /var/log/pnp4nagios/pnp4nagios.log

Add below line 

ENCRYPTION=0

Move process_perfdata.pl plugin to you your plugin directory

mv process_perfdata.pl /usr/local/nagios/libexec/plugins/

Add below line in generic-host template in /etc/nagios/lconf/default-templates.cfg 
action_url /pnp4nagios/graph?host=$HOSTNAME$

Add below line in generic-service template in /etc/nagios/lconf/default-templates.cfg file
action_url /pnp4nagios/graph?host=$HOSTNAME$&srv=$SERVICEDESC$

Now Restart Nagios and Httpd service.

service nagios restart

service httpd restart

Now you can see new graph icon before service name in Nagios UI.

check_oracle_health

check_oracle_health

Monitoring Oracle Database

To Create monitoring of Oracle database Follow below steps.

Step 1: To Download Below Rpms Click Here.....

oracle-instantclient12.1-basic-12.1.0.1.0-1.x86_64.rpm
oracle-instantclient12.1-sqlplus-12.1.0.1.0-1.x86_64.rpm
oracle-instantclient12.1-devel-12.1.0.1.0-1.x86_64.rpm
oracle-instantclient12.1-jdbc-12.1.0.1.0-1.x86_64.rpm

Step 2: After Downloading rpms Install them on Nagios Server.

Fist install below pacages: libdbi-drivers and libdbi perl-DBI libaio
[root@nagios_master ~]# yum install libdbi-drivers libdbi perl-DBI libaio

Now install oracle packages.
[root@nagios_master ~]# rpm -ivh oracle-instantclient12.1-devel-12.1.0.1.0-1.x86_64.rpm oracle-instantclient12.1-jdbc-12.1.0.1.0-1.x86_64.rpm oracle-instantclient12.1-sqlplus-12.1.0.1.0-1.x86_64.rpm oracle-instantclient12.1-basic-12.1.0.1.0-1.x86_64.rpm

Now set ORACLE HOME
[root@nagios_master ~]# echo 'export ORACLE_HOME=/usr/lib/oracle/12.1/client64/' /etc/profile

[root@nagios_master ~]# echo 'export LD_LIBRARY_PATH=/usr/lib/oracle/12.1/client64/lib/' >> /etc/profile
[root@nagios_master ~]# mkdir -p /usr/lib/oracle/12.1/client64/lib/network/admin/ && echo 'names.directory_path = (TNSNAMES,EZCONNECT)' > /usr/lib/oracle/12.1/client64/lib/network/admin/sqlnet.ora
[root@nagios_master ~]# chmod 777 /tmp/check_oracle_health

Now execute cpan command prompt and install DBD-Oracle 
[root@nagios_master ~]# cpan
cpan[1]> install DBD::Oracle


Step 3: Now download Check_oracle_health plugin from below link:

http://labs.consol.de/download/shinken-nagios-plugins/check_oracle_health-1.7.8.1.tar.gz 

[root@nagios_master ~]# tar -xvf check_oracle_health-1.7.8.1.tar.gz
[root@nagios_master ~]# cd check_oracle_health-1.7.8.1
[root@nagios_master ~]# ./configure --with-statefiles-dir=/tmp/ --with-nagios-user=nagios --with-nagios-group=nagios --with-mymodules-dir=/usr/local/nagios/libexec/plugins/ --with-mymodules-dyn-dir=/usr/local/nagios/libexec/plugins/

[root@nagios_master ~]# make && make install

Then, create a dedicated user for monitoring, connect to your Oracle database and enter the following lines:

CREATE USER nagios IDENTIFIED BY nagios ;

GRANT CREATE SESSION TO nagios;GRANT SELECT any dictionary TO nagios;GRANT SELECT ON V_$SYSSTAT TO nagios;GRANT SELECT ON V_$INSTANCE TO nagios;GRANT SELECT ON V_$LOG TO nagios;GRANT SELECT ON SYS.DBA_DATA_FILES TO nagios;GRANT SELECT ON SYS.DBA_FREE_SPACE TO nagios;

[root@nagios_master ~]# /usr/local/nagios/libexec/plugins/check_oracle_health --connect=xxx.xxx.xxx.xxx/ORCL --username=nagios --password=nagios --tablespace=SUPERVISION  --mode tablespace-usage

CRITICAL - tbs SYSTEM usage is 90%

Syslog-ng Configuration to Send logs to Syslog server

Client IP:192.168.1.65
ServerIP:192.168.1.64

Client side add following lines into /etc/syslog-ng/syslog-ng.conf
##########################

source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); };
source s_msg { file("/var/log/messages.myhost"); };
destination d_loghost {tcp("192.168.1.64" port(514));};
log { source(s_local); source(s_msg); destination(d_loghost); };

###################################

Server side add following lines
To send logs to mysql need following packages
 libdbi0, libdbi0-dev, libdbd-mysql

##################################

  options {
        flush_lines (0);
    log_fifo_size (2048);
    create_dirs (yes);
    group (shankar);
    dir_group (shankar);
    perm (0640);
    dir_perm (0750);
    };


source s_remote { tcp(); };
destination d_clients { file("/var/log/HOSTS/$HOST/shankar.txt"); };
log { source(s_remote); destination(d_clients); };

# CREATES TABLE IF NOT IN MYSQL

destination d_sql {
sql(type(mysql)
host("localhost") username("root") password("") port("3306")
database("syslog")
table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")
columns("datetime", "host", "program", "pid", "message")
values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY")
indexes("datetime", "host", "program", "pid", "message"));
};


# TO SEND LOGS IN MYSQL
log { source(s_remote); destination(d_sql); };

###################################

SNMP Trap Monitoring in nagios

Introduction

To monitor SNMP supported devices like switches, router, firewall, UPS, Thermometer we need not to check actively the status of device. SNMP supported device provides trap incase of status change as described in MIB file of that device.

 SNMP Trap monitoring

The scenario is about nagios acting as a "manager" or (part of) a Network Management System (NMS), receiving messages from "Trap Agents" - conceptually similar to NSCA agents sending msgs to the NSCA Daemon with interacts with a Nagios passive service (add links), except that the values to be checked and the thresholds that trigger a message are handled with snmp tools.

Installation

To accept and process all traps from snmp supported devices we need to install few packages.

Following is the list of packages:

• snmpd
• snmp
• snmptt
• libnet-snmp-perl
• libsnmp-perl
• libsnmp-python
• libconfig-inifiles-perl
• libconfig-inifiles-perl

Installation methods:
All above mentioned packages are available in our repositories we can install it as follows

apt-get install snmpd snmp libnet-snmp-perl libsnmp-perl libsnmp-python libconfig-inifiles-perl libconfig-inifiles-perl snmptt

Now it is required to start snmpd service at boot time so we need to add it startup 

insserv snmpd

Configuration

[ Note :  * In configuration part you need to remove those parameters which written with RED color and add  those parameters which written with GREEN colour. ]

vim /etc/snmp/snmpd.conf

• agentAddress udp:127.0.0.1:161                     ###     (remove this line )
•  agentAddress  udp:192.168.1.66:161              ###    (add this line)

In  /etc/defaults/snmp update following parameters. 
==>Default TRAPD is disabled

• TRAPDRUN=no  

 Which is required to enable So enable it by changing "no" to "yes"

• TRAPDRUN=yes  
• TRAPDOPTS='-n -Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid'

==>Default SNMPD is enabled

• SNMPDRUN=yes

Which is not required to enable So disable it by changing "yes" to "no" 

• SNMPDRUN=no

In  /etc/snmp/snmptrapd.conf  update following parameters. Add following lines.

• disableAuthorization yes
•  traphandle default /usr/sbin/snmptthandler

In  /etc/snmp/snmptt.ini  update following parameters.

•  mode = daemon
•  net_snmp_perl_enable = 1
•  net_snmp_perl_best_guess = 2
•  translate_log_trap_oid = 2
•  remove_backslash_from_quotes = 1
•  description_mode = 2
•  date_time_format = %Y-%m-%d %H:%M:%S
•  unknown_trap_log_enable = 1
•  DEBUGGING_FILE = /var/log/snmptt.debug
•  DEBUGGING = 0 

 Update follwoing perameteres if you want to configure smtp with eventdb

•  db_translate_enterprise = 1
•  db_unknown_trap_format = 'Unknown Trap: $-*'
•  mysql_dbi_enable = 1
•  mysql_dbi_host = <eventdb database host)
•  mysql_dbi_database = <eventdb database name>
•  mysql_dbi_table = <eventdb table name for known traps>
•  mysql_dbi_table_unknown = <eventdb table name for known traps>
•  mysql_dbi_username = <Username for eventdb database access>
•  mysql_dbi_password = < Password for eventdb database access>
•  date_time_format_sql = %Y-%m-%d %H:%M:%S
•  stat_time_format_sql = %Y-%m-%d %H:%M:%S

Upload Trap definition for devices

Generate trap definition from mibs for all devices by following command and add those file names in /etc/snmp/snmptt.ini

snmpttconvertmib --in=MIBFILE --out=/etc/snmp/snmptt.conf --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $A TRAP <STATUS_NO_NAGIOS>' 

For Example:

snmpttconvertmib --in=MIBFILE --out=/etc/snmp/snmptt.conf --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $A TRAP 2' 

$A is the hostipaddress can be found in received traps

TRAP is the service defined on host 

2 is the status of service which you can have in nagios

Create Service in NAGIOS

Now create service (like. TRAP) in NAGIOS with following parameters.

EXAMPLE : Service name TRAP 

define service{ 
name trap-service 
register 0 
service_description TRAP 
is_volatile 1 
check_command check-host-alive 
max_check_attempts 1 
normal_check_interval 1 
retry_check_interval 1 
passive_checks_enabled 1 
check_period never
notification_interval 0 
contact_groups admins
check_period 24x7 
max_check_attempts 3 
normal_check_interval 15 
retry_check_interval 5 
active_checks_enabled 1 
passive_checks_enabled 0 
parallelize_check 1 
obsess_over_service 0 
check_freshness 0 
event_handler_enabled 0 
flap_detection_enabled 0 
process_perf_data 1 
retain_status_information 1 
retain_nonstatus_information 1 
notification_interval 60 
notification_period 24x7 
notification_options w,u,c,r 
notifications_enabled 1 
}

###################################

SUBMIT_CHECK_RESULT Plugin For nagios

###################################

#!/bin/bash
########################################################################################################
#Description  : This script will write a command to the Nagios command file to cause Nagios to process a passive service check result. Note: This script is intended to be run on the same host that is running Nagios  Main host and it will send those traps on that host to nagios instances if the trap found for the device configured on any of the nagios instance.
#################################################################s########################################
##Change Log  :
# Arguments:
#  $1 = ipaddress of host that the service is associated with
#  $2 = svc_description (Description of the service)
#  $3 = return_code (An integer that determines the state of the service check, 0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN).
#  $4 = plugin_output (A text string that should be used as the plugin output for the service check)
#
#################################################################s########################################
#binary paths
check_nrpe="/usr/lib/nagios/plugins/check_nrpe"
echocmd="/bin/echo"
db_name="nagios"
db_user="root"
db_passwd="password"

#Nagios CommandFile in which command will be passed to nagios 
CommandFile="/var/lib/nagios/rw/nagios.cmd"

#gets nagios instance and host_name of the source of trap from the ipaddress
host_name=`/usr/bin/mysql --skip-column-names -u $db_user -p$db_passwd $db_name -e"select instance_id,display_name from nagios_hosts where address=\"$1\"" | /usr/bin/expand -t 1`
instance=`echo $host_name|cut -d' ' -f1`
host_name=`echo $host_name|cut -d' ' -f2`

# get the current date/time in seconds since UNIX epoch
datetime=`date +%s`

# check thwe instance and forward nagios command to respected nagios instance
if [ $instance -eq 1 ]
then
        cmdline="$host_name;$2;$3;$4"
#       append the command to the end of the command file
        echo "[$datetime] PROCESS_SERVICE_CHECK_RESULT;$cmdline" >> $CommandFile
else
        print "Unknown. Host not found."
fi