How to read 2 files alternatively line by line

Run the same command on many Linux servers at once

Ever have to check a list of Linux servers for various things like what version of CentOS they’re running, maybe how long each has been running to get an uptime report? You can – and it’s very easy to get going with it.

gsh allows you to run commands on multiple hosts by adding tags to the gsh command.

gsh tag "remote command"

Important things to remember:

/etc/ghosts contains a list of all the servers and tags

gsh is a lot more fun once you’ve set up ssh keys to your servers Examples to use:

List uptime on all servers in the linux group:

gsh linux "uptime"

Check to see if an IP address was blocked with CSF by checking the csf and csfcluster groups/tags:

gsh csf+csfcluster "/usr/sbin/csf -g xxx.xxx.xxx.xxx"

Unblock an IP and remove from /etc/csf.deny from all csf and csfcluster machines

gsh csf+csfcluster "/usr/sbin/csf -dr xxx.xxx.xxx.xxx"

Check the linux kernel version on all VPS machines running centos 5

gsh centos5-baremetal "uname -r"

Check cpanel version on all cpanel machines

gsh cpanel "/usr/local/cpanel/cpanel -V"

The full readme is located here: http://outflux.net/unix/software/gsh/

Here’s an example /etc/ghosts file:

# Machines

#

# hostname                    OS-Version    Hardware    OS    cp    security

1.linuxbrigade.com        debian6        baremetal    linux    plesk    iptables

2.linuxbrigade.com        centos5        vps        linux    cpanel    csfcluster

3.linuxbrigade.com        debian7        baremetal    linux    plesk    iptables

4.linuxbrigade.com        centos6        vps        linux    cpanel    csfcluster

5.linuxbrigade.com        centos6        vps        linux    cpanel    csfcluster

6.linuxbrigade.com        centos6        vps        linux    nocp    denyhosts

7.linuxbrigade.com        debian6        baremetal    linux    plesk    iptables

8.linuxbrigade.com        centos6        baremetal    linux    cpanel    csf

9.linuxbrigade.com        centos5        vps        linux    cpanel    csf

centos5 vps linux cpanel csf

Configuring Gmail as a Sendmail email relay

Introduction

 In this configuration tutorial I will guide you through the process of configuring sendmail to be an email relay for your gmail or google apps account. This allows you to send email from your bash scripts, hosted website or from command line using mail command. Other examples where you can utilize this setting is for a notification purposes such or failed backups etc. Sendmail is just one of many utilities which can be configured to rely on gmail account where the others include postfix, exim , ssmpt etc. In this tutorial I will use Debian and sendmail for this task.

Install prerequisites

# apt-get install sendmail mailutils sendmail-bin 

Create Gmail Authentication file

# mkdir -m 700 /etc/mail/authinfo/
# cd /etc/mail/

next we need to create a auth file with a following content. File can have any name, in this example the name is gmail-auth:

AuthInfo: "U:root" "I:shankar.patel@gmail.com" "P:xxxxxxxxx"

Replace the above email with your gmail or google apps email.

Please note that in the above password example you need to keep 'P:' as it is not a part of the actual password.

In the next step we will need to create a hash map for the above authentication file:

# makemap hash gmail-auth < gmail-auth

Configure your sendmail 

Put bellow lines into your sendmail.mc configuration file right above first "MAILER" definition line:

define(`SMART_HOST',`[smtp.gmail.com]')dnl
define(`RELAY_MAILER_ARGS', `TCP $h 587')dnl
define(`ESMTP_MAILER_ARGS', `TCP $h 587')dnl
define(`confAUTH_OPTIONS', `A p')dnl
TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
FEATURE(`authinfo',`hash -o /etc/mail/gmail-auth.db')dnl

Do not put the above lines on the top of your sendmail.mc configuration file !

In the next step we will need to re-build sendmail's configuration. To do that execute:

# make -C /etc/mail

Reload sendmail service:

/etc/init.d/sendmail reload

and you are done.

Configuration test

Now you can send an email from your command line using mail command:

echo "Just testing my sendmail gmail relay" | mail -s "Sendmail gmail Relay" shankar.h.patel@yahoo.com

SNMP Trap Monitoring in nagios

Introduction

To monitor SNMP supported devices like switches, router, firewall, UPS, Thermometer we need not to check actively the status of device. SNMP supported device provides trap incase of status change as described in MIB file of that device.

 SNMP Trap monitoring

The scenario is about nagios acting as a "manager" or (part of) a Network Management System (NMS), receiving messages from "Trap Agents" - conceptually similar to NSCA agents sending msgs to the NSCA Daemon with interacts with a Nagios passive service (add links), except that the values to be checked and the thresholds that trigger a message are handled with snmp tools.

Installation

To accept and process all traps from snmp supported devices we need to install few packages.

Following is the list of packages:

• snmpd
• snmp
• snmptt
• libnet-snmp-perl
• libsnmp-perl
• libsnmp-python
• libconfig-inifiles-perl
• libconfig-inifiles-perl

Installation methods:
All above mentioned packages are available in our repositories we can install it as follows

apt-get install snmpd snmp libnet-snmp-perl libsnmp-perl libsnmp-python libconfig-inifiles-perl libconfig-inifiles-perl snmptt

Now it is required to start snmpd service at boot time so we need to add it startup 

insserv snmpd

Configuration

[ Note :  * In configuration part you need to remove those parameters which written with RED color and add  those parameters which written with GREEN colour. ]

vim /etc/snmp/snmpd.conf

• agentAddress udp:127.0.0.1:161                     ###     (remove this line )
•  agentAddress  udp:192.168.1.66:161              ###    (add this line)

In  /etc/defaults/snmp update following parameters. 
==>Default TRAPD is disabled

• TRAPDRUN=no  

 Which is required to enable So enable it by changing "no" to "yes"

• TRAPDRUN=yes  
• TRAPDOPTS='-n -Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid'

==>Default SNMPD is enabled

• SNMPDRUN=yes

Which is not required to enable So disable it by changing "yes" to "no" 

• SNMPDRUN=no

In  /etc/snmp/snmptrapd.conf  update following parameters. Add following lines.

• disableAuthorization yes
•  traphandle default /usr/sbin/snmptthandler

In  /etc/snmp/snmptt.ini  update following parameters.

•  mode = daemon
•  net_snmp_perl_enable = 1
•  net_snmp_perl_best_guess = 2
•  translate_log_trap_oid = 2
•  remove_backslash_from_quotes = 1
•  description_mode = 2
•  date_time_format = %Y-%m-%d %H:%M:%S
•  unknown_trap_log_enable = 1
•  DEBUGGING_FILE = /var/log/snmptt.debug
•  DEBUGGING = 0 

 Update follwoing perameteres if you want to configure smtp with eventdb

•  db_translate_enterprise = 1
•  db_unknown_trap_format = 'Unknown Trap: $-*'
•  mysql_dbi_enable = 1
•  mysql_dbi_host = <eventdb database host)
•  mysql_dbi_database = <eventdb database name>
•  mysql_dbi_table = <eventdb table name for known traps>
•  mysql_dbi_table_unknown = <eventdb table name for known traps>
•  mysql_dbi_username = <Username for eventdb database access>
•  mysql_dbi_password = < Password for eventdb database access>
•  date_time_format_sql = %Y-%m-%d %H:%M:%S
•  stat_time_format_sql = %Y-%m-%d %H:%M:%S

Upload Trap definition for devices

Generate trap definition from mibs for all devices by following command and add those file names in /etc/snmp/snmptt.ini

snmpttconvertmib --in=MIBFILE --out=/etc/snmp/snmptt.conf --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $A TRAP <STATUS_NO_NAGIOS>' 

For Example:

snmpttconvertmib --in=MIBFILE --out=/etc/snmp/snmptt.conf --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $A TRAP 2' 

$A is the hostipaddress can be found in received traps

TRAP is the service defined on host 

2 is the status of service which you can have in nagios

Create Service in NAGIOS

Now create service (like. TRAP) in NAGIOS with following parameters.

EXAMPLE : Service name TRAP 

define service{ 
name trap-service 
register 0 
service_description TRAP 
is_volatile 1 
check_command check-host-alive 
max_check_attempts 1 
normal_check_interval 1 
retry_check_interval 1 
passive_checks_enabled 1 
check_period never
notification_interval 0 
contact_groups admins
check_period 24x7 
max_check_attempts 3 
normal_check_interval 15 
retry_check_interval 5 
active_checks_enabled 1 
passive_checks_enabled 0 
parallelize_check 1 
obsess_over_service 0 
check_freshness 0 
event_handler_enabled 0 
flap_detection_enabled 0 
process_perf_data 1 
retain_status_information 1 
retain_nonstatus_information 1 
notification_interval 60 
notification_period 24x7 
notification_options w,u,c,r 
notifications_enabled 1 
}

###################################

SUBMIT_CHECK_RESULT Plugin For nagios

###################################

#!/bin/bash
########################################################################################################
#Description  : This script will write a command to the Nagios command file to cause Nagios to process a passive service check result. Note: This script is intended to be run on the same host that is running Nagios  Main host and it will send those traps on that host to nagios instances if the trap found for the device configured on any of the nagios instance.
#################################################################s########################################
##Change Log  :
# Arguments:
#  $1 = ipaddress of host that the service is associated with
#  $2 = svc_description (Description of the service)
#  $3 = return_code (An integer that determines the state of the service check, 0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN).
#  $4 = plugin_output (A text string that should be used as the plugin output for the service check)
#
#################################################################s########################################
#binary paths
check_nrpe="/usr/lib/nagios/plugins/check_nrpe"
echocmd="/bin/echo"
db_name="nagios"
db_user="root"
db_passwd="password"

#Nagios CommandFile in which command will be passed to nagios 
CommandFile="/var/lib/nagios/rw/nagios.cmd"

#gets nagios instance and host_name of the source of trap from the ipaddress
host_name=`/usr/bin/mysql --skip-column-names -u $db_user -p$db_passwd $db_name -e"select instance_id,display_name from nagios_hosts where address=\"$1\"" | /usr/bin/expand -t 1`
instance=`echo $host_name|cut -d' ' -f1`
host_name=`echo $host_name|cut -d' ' -f2`

# get the current date/time in seconds since UNIX epoch
datetime=`date +%s`

# check thwe instance and forward nagios command to respected nagios instance
if [ $instance -eq 1 ]
then
        cmdline="$host_name;$2;$3;$4"
#       append the command to the end of the command file
        echo "[$datetime] PROCESS_SERVICE_CHECK_RESULT;$cmdline" >> $CommandFile
else
        print "Unknown. Host not found."
fi

Recieving and Sending SNMP TRAP Configuration

SNMP Traps

Sending SNMP traps on a Remote Linux/Windows System 

Download SNMP Trap sender from following link.

http://trapgen.trapreceiver.com/

Now Edit input file and change ip address of remote host where you want to send snmp traps.

Change "-d 192.168.1.1" to your destination host.

and 

execute like as under

root@localhost:~#./trapgen -f input

 AND same time check the syslog file of remote SNMPTRAP manager syst

root@localhost:~# tail -f /var/log/syslog

Receive and handle SNMP traps on a Linux system

Scope

This is a quickstart guide on howto get an Ubuntu or Debian Linux system to listen for SNMP traps, and log them. In SNMP parlance, where' talking about configuring a linux system to be a "manager". We're not talking about how to monitor a linux host by means of SNMP.

We'll cover the basics of how to get it up and running, and provide some pointers to where to go next.

Prerequisites

It is assumed you have a basic understanding of what SNMP is, and are looking for a generic way to collect SNMP-generated messages (a.k.a. "trap's") from 1 or more devices onto a "manager" system, for centralized monitoring.

You might be interested in a generic, linux-based solution because this allows you to connect SNMP-based monitoring to other central monitoring solutions such as a central syslog server and/or a Nagios monitoring system.

It is assumed that you have a working linux system where you want to implement this. The following guide is based on Debian Squeeze and will most likele work as-is on other Debian bases server systems such as Ubuntu. It should also work on other linux sysems, though some details such as names of packages, commands and config files may be different.

Basic Setup

                                       apt-get install snmpd

this installs a number of snmp-related tools, including snmptrapd, which is the one we need to be able to listen for "traps"

snmpbulkget    snmpdf         snmpnetstat    snmptranslate  snmpwalk
snmpbulkwalk   snmpget        snmpset        snmptrap       
snmpconf       snmpgetnext    snmpstatus     snmptrapd      
snmpd          snmpinform     snmptable      snmpusm        
snmpdelta      snmpkey        snmptest       snmpvacm   

snmptrapd does not start up by default, you need to enable it in /etc/defaults/snmp ; it will then be started by /etc/init.d/snmpd (if /etc/snmp/snmptrapd.conf exists
/etc/defaults/snmp
                     TRAPDRUN=yes

we don't need to run snmpd if we only want to recive traps, rather than monitor this system by snmp, so we can disable it:
                     SNMPDRUN=no

Also have a look at the other default options : Debian sets them to "log to syslog" by default, so that traps will be reported in /var/log/syslog. This is OK for us (for now)

snmptrapd will not accept any traps unless they're "authorized" in /etc/snmp/snmptrapd.conf :

# simple test catch-all for any trap that has community=public
authCommunity log,execute,net public

Restart snmpd to get all the changes applied, then check if it works by sending a test trap (from the local host, and from a remote system) :

• local :

                                                                  snmptrap -v1 -c public localhost .1.3.6.1.6.3.1.1.5.2 0 0 "" "" .1.3.6.1.4.1 s "hello"

• remote:

                                                                        snmptrap -v1 -c public myserver .1.3.6.1.6.3.1.1.5.2 0 0 "" "" .1.3.6.1.4.1 s "hello" 

result : /var/log/syslog should now show te submitted trap msg :

May 1 12:35:00 localhost snmptrapd[1594]: 2013-05-01 12:35:00 0.0.0.0(via UDP: [127.0.0.1]:51633->[127.0.0.1]) TRAP, SNMP v1, community public#012#011.1.3.6.1.6.3.1.1.5.2 Cold Start Trap (0) Uptime: 0:26:53.33#012#011.1.3.6.1.4.1 = STRING: "hello"

this shows snmptrapd is working: it is capable of receiving snmp traps and handling them (in this case: logging them to syslog)

Further configuration

further config will involve :

1. define traps on the monitored hosts
2. add appropriate authorization config for those hosts in /etc/snmp/snmptrapd.conf  see  http://net-snmp.sourceforge.net/docs/man/snmptrapd.conf.html

More on Trap Handling

Syslog

Now you have snmptrapd logs in local syslog file. By configuring SYSLOG server you can integrate your snmp traps in a centralised syslog configuration.

Nagios

You can integrate SNMP trap with nagios to monitor traps using snmp traps and will improve your monitoring. See SNMP Trap monitoring in nagios

Nagios Perl Plugin (Service Check did not exit properly) issue

Hi Friends,

In this post I will talk about solving perl plugin related issue. Sometimes we are facing issue when we are creating plugin for nagios in perl language. Sometimes what happens we executing plugin on command prompt then it is working fine and giving properl result but when we are deploying that plugin in to production nagios environment then we are getting error in output.

If we are getting following error in nagios then 
 (Service Check did not exit properly) 
Add following lines in to beginning of the your perl script.

########################################################
use strict;
require 5.6.0;
use lib qw( /usr/lib/nagios/plugins );
use utils qw(%ERRORS $TIMEOUT &print_revision &support &usage);
############################################################################

Play with VIM editor

• To use mouse in vim editor copy and paste following line in to /etc/vim/vimrc file.Using mouse we can move cursor like we are moving in notepad.

set mouse=a             " Enable mouse usage (all modes)

• To make variables and system functions colorful add folowing line in to  /etc/vim/vimrc file.

syntax on

• By default in debian we opens a file in vim editor then vim is not saving our last position of the cursor.

      To enable that write following lines in to the /etc/vim/vimrc file.

if has("autocmd")

  au BufReadPost * if line("'\"") > 1 && line("'\"") <= line("$") | exe "normal! g'\"" | endif

endif

Continue.......

Date command in linux

Hi Friends,

We can convert Unix timestamp to date and date to unix timestamp using date command .

Unix timestamp to date conversion.


root@SHANKAR:~# date -d @1285977600
Fri Oct  1 17:00:00 PDT 2010


Date to Unix timestamp conversion.

root@SHANKAR:~# date -d "2010-10-30" "+%s"
1288422000

Rebuild your deb package by updating some files

#!/bin/bash

# script name: rebuild-deb.sh
# extracts/rebuilds a deb package
# Put your deb and the script in a new directory and run it from there
# 

echo "This script extracts and rebuilds a deb package. "
echo "It should called from the directory containing the original deb and run as:"
echo "./rebuild-deb.sh"
echo
ls
echo
echo "1.   Extract deb?"
echo "2.   Make deb?"
echo "Enter <1/2> :"
read REPLY
echo

   if [ "$REPLY" = "1" ]; then
ls

DEB=$(ls *.deb 2>&1)
mkdir package
cd ./package
ar -x ../$DEB

   rm debian-binary
   tar xvzf data.tar.gz && rm data.tar.gz
   mkdir DEBIAN && cd DEBIAN
   tar xvzf ../control.tar.gz && rm ../control.tar.gz

   echo "Extracted to './package'"
   echo "Make your changes. Make sure to update control and md5sum files"   
   elif [ "$REPLY" = "2" ]; then

PACKAGE=$(cat 'package/DEBIAN/control'|grep Package|sed 's|Package: ||')
VERSION=$(cat 'package/DEBIAN/control'|grep Version|sed 's|Version: ||')
ARCH=$(cat 'package/DEBIAN/control'|grep Architecture|sed 's|Architecture: ||')
fakeroot dpkg-deb -b package $PACKAGE\_$VERSION\_$ARCH.deb
   
   echo "Done"
fi