Good security, not just compliance requirements, encourages IT & Security staff to spend time reviewing security events.  There are many tools available for many security components of your network; this article will focus on host security and host log monitoring using OSSEC and Zabbix.

After having used many open and commercial  monitoring systems, we have settled on Zabbix because of it’s extensive features and expandability (especially with the new integrated API).  That and it doesn’t hurt that it is well designed for expanded enterprises and is OpenSource.

This article was originally posted with details for Zabbix 1.6 and was updated on April 10 to reflect how to set it up under Zabbix 1.8.

OSSEC is a great tool provided by Trend Micro and is also an OpenSource application. OSSEC provides a variety of tools for host based intrusion detection including:  log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response.  All of which support several compliance and basic security requirements. OSSEC is deployed in a client-server model with all alerting and active response features being controlled and sent through the central server.

To integrate OSSEC and Zabbix we will be using the active-response feature of OSSEC integrated with zabbix_sender to send the active response alert to the zabbix server.   Configuring for this integration requires a simple script, a quick change to the ossec.conf and the creation of an OSSEC template in the zabbix system.

We will start with the OSSEC changes.  First, we will edit the OSSEC/etc/ossec.conf file, where OSSEC is the path to your OSSEC installation.  In this file you will need to add the following items:

<command>
<name>zabbix-alert</name>
<executable>zabbix-alert.sh</executable>
<timeout_allowed>no</timeout_allowed>
<expect></expect>
</command>

<active-response>
<disabled>no</disabled>
<command>zabbix-alert</command>
<location>server</location>
<level>1</level>
</active-response>

The first <command> item defines the script we will be using for the zabbix-alert.  The <active-response> item defines when the system will use this script.  The defined configuration above has all alerts at and above level “1″ sent to the zabbix-alert command.  This can be modified for higher levels or specific rules or rule groups.  More information on this configuration can be found in the OSSEC manual.

Now the zabbix-alert.sh script needs to be put into the OSSEC/active-response/bin directory (be sure to watch for lines that are wrapped around but shouldn’t be).  You can download the script here: zabbix-alert.sh.

#!/bin/sh
#
# Submits an OSSEC alert as a passive service check result to zabbix.
#
# Author: David M. Zendzian
# ZZ Servers, LLC 2010
#
# Idea from Dave Stycos post: http://groups.google.com/group/ossec-dev/browse_thread/thread/e29c5d71926b8af5
#
# This script is Public Domain, and is provided AS-IS.  There is no
# warranty, and no support given for its contents.
#
# Version 1.0: Apr. 6, 2010
#

DEBUG=”false”
ACTION=$1
USER=$2
IP=$3
ALERTID=$4
RULEID=$5

LOCAL=dirname $0;
cd $LOCAL
cd ../
PWD=pwd
UNAME=uname

# Zabbix Sender
ZabbixSender=”/usr/bin/zabbix_sender”
#ZabbixSender=”/usr/sbin/zabbix_sender”

# Zabbix Server
ZabbixServer=<your zabbix server ip>

# Zabbix Port
ZabbixPort=10051

# All alerts will be processed by Zabbix under this key.
ZabbixKeyName=OSSEC

# Check that zabbix_sender file exists.
if [ ! -w $ZabbixSender ]; then
logger -p local0.err “$0: File $ZabbixSender not found.  Exiting.”
exit 1
fi

# Getting alert time
ALERTTIME=echo "$ALERTID" | cut -d  "." -f 1

# Getting end of alert
ALERTLAST=echo "$ALERTID" | cut -d  "." -f 2

# Getting full alert
ALERTTEXT=grep -A 10 "$ALERTTIME" $PWD/../logs/alerts/alerts.log | grep -v ".$ALERTLAST: " -A 10

# Extract host (agent) name from alert.
HOSTNAME=echo "$ALERTTEXT" | sed -n '1,1s/^.*\:[0-9][0-9]\:[0-9][0-9][^A-Za-z0-9_]*\([-A-Za-z0-9_]*\)\->.*$/\1/p'

# if hostname alert wasn’t from local host, the host value is “(hostname) ip”, which extracts differently
if [ "$HOSTNAME" = "" ]
then
HOSTNAME=echo "$ALERTTEXT" | sed -n '1,1s/^.*\:[0-9][0-9]\:[0-9][0-9] (\([-A-Za-z0-9_]*\)) .*\->.*$/\1/p'
fi
if [ "$HOSTNAME" = "" ]
then
exit 0
fi

# Extract alert level from alert.
ALERTLVL=echo "$ALERTTEXT" | sed -n '2,2s/^.*(level \([0-9]*\).*$/\1/p'

# Extract description from alert.
ALERTMSG=echo "$ALERTTEXT" | sed -n '5,5p'

# Create Alert message
# Short MSG version
#ZMSG=”$ALERTID | $ALERTLVL | $RULEID – $ALERTMSG”
# Full MSG Version
ZMSG=”AlertID: $ALERTID | User: $USER | IP: $IP | Level: $ALERTLVL | RuleID: $RULEID – $ALERTMSG”

# Send result to zabbix for logging and notification alerts.
$ZabbixSender –zabbix-server $ZabbixServer –port $ZabbixPort –host $HOSTNAME –key $ZabbixKeyName –value “$ZMSG”

if [ "$DEBUG" = "true" ]
then
echo “$ZabbixSender –zabbix-server $ZabbixServer –port $ZabbixPort –host $HOSTNAME –key $ZabbixKeyName –value ‘$ZMSG’” >> /tmp/zabbix-test.log
echo “ACTION: $ACTION” >> /tmp/zabbix-test.log
echo “USER: $USER”
echo “IP: $IP” >> /tmp/zabbix-test.log
echo “ALERTID: $ALERTID” >> /tmp/zabbix-test.log
echo “ALERTLVL: $ALERTLVL” >> /tmp/zabbix-test.log
echo “RULEID: $RULEID” >> /tmp/zabbix-test.log
echo “———————————” >> /tmp/zabbix-test.log
fi

With the script saved, you can restart OSSEC (OSSEC/bin/ossec-control restart) or wait until zabbix is setup then restart.

UPDATE 07-16-2010 – If you are using zabbix-proxies then you need to have the OSSEC alerts for proxy monitored hosts submitted through the proxy server.  This isn’t a problem with the existing script if the proxy server is also monitored through the proxy; just update the server IP to be the proxy not the central zabbix server.  If you monitor your proxy directly from the central zabbix server then the script needs to be updated to support sending proxy hosts though proxy and the host itself directly to zabbix.  The script can be found here; or below.  Again be sure to watch for broken wrapped lines:

#!/bin/sh
#
# Submits an OSSEC alert as a passive service check result to zabbix.
#
# Author: David M. Zendzian
#
# Idea from Dave Stycos post: http://groups.google.com/group/ossec-dev/browse_thread/thread/e29c5d71926b8af5
#
# Updated 7/15/10 – using 2 server hosts in case using proxies and local host is monitored directly by central server and not proxy.
#
# This script is Public Domain, and is provided AS-IS.  There is no
# warranty, and no support given for its contents.
#
# Version 1.1: Jul. 15, 2010
#

DEBUG=”true”
ACTION=$1
USER=$2
IP=$3
ALERTID=$4
RULEID=$5

if [ "$DEBUG" = "true" ]
then
echo “NOTICE: Starting Zabbix sender” >> /tmp/zabbix-test.log
fi

LOCAL=dirname $0;
cd $LOCAL
cd ../
PWD=pwd
UNAME=uname

# Zabbix Sender
ZabbixSender=”/usr/bin/zabbix_sender”
#ZabbixSender=”/usr/sbin/zabbix_sender”

# Zabbix Server
# Set server2 to be the same if all hosts monitored through proxy or the same server; otherwise
# set ZabbixServer to the proxy for non-localhost and then ZabbixServer2 to the host that the
# local proxy uses
ZabbixServer=<Server/Proxy>
ZabbixServer2=<Server for “Localhost”>

# Zabbix Port
ZabbixPort=10051

# All alerts will be processed by Zabbix under this key.
ZabbixKeyName=OSSEC

# Check that zabbix_sender file exists.
if [ ! -w $ZabbixSender ]; then
logger -p local0.err “$0: File $ZabbixSender not found.  Exiting.”
if [ "$DEBUG" = "true" ]
then
echo “ERROR: No Zabbix Sender” >> /tmp/zabbix-test.log
fi
exit 1
fi

# Getting alert time
ALERTTIME=echo "$ALERTID" | cut -d  "." -f 1

# Getting end of alert
ALERTLAST=echo "$ALERTID" | cut -d  "." -f 2

# Getting full alert
ALERTTEXT=grep -A 10 "$ALERTTIME" $PWD/../logs/alerts/alerts.log | grep -v ".$ALERTLAST: " -A 10

# Extract host (agent) name from alert.
HOSTNAME=echo "$ALERTTEXT" | sed -n '1,1s/^.*\:[0-9][0-9]\:[0-9][0-9][^A-Za-z0-9_]*\([-A-Za-z0-9_]*\)\->.*$/\1/p'

# if hostname alert wasn’t from local host, the host value is “(hostname) ip”, which extracts differently
if [ "$HOSTNAME" = "" ]
then
HOSTNAME=echo "$ALERTTEXT" | sed -n '1,1s/^.*\:[0-9][0-9]\:[0-9][0-9] (\([-A-Za-z0-9_]*\)) .*\->.*$/\1/p'
fi
if [ "$HOSTNAME" = "" ]
then
if [ "$DEBUG" = "true" ]
then
echo “ERROR: No Hostname” >> /tmp/zabbix-test.log
fi
exit 0
fi

# if the local host is a proxy then monitored items submitted through proxy, localhost probably monitored directly from central server (if not change comment this out)
LOCALHOSTNAME=hostname -s
if [ "$HOSTNAME" = "$LOCALHOSTNAME" ]
then
ZabbixServer=$ZabbixServer2
fi

if [ "$DEBUG" = "true" ]
then
echo “ZabbixServer: $ZabbixServer” >> /tmp/zabbix-test.log
fi

# Extract alert level from alert.
ALERTLVL=echo "$ALERTTEXT" | sed -n '2,2s/^.*(level \([0-9]*\).*$/\1/p'

# Extract description from alert.
ALERTMSG=echo "$ALERTTEXT" | sed -n '5,5p'

# Create Alert message
ZMSG=”$ALERTID | $ALERTLVL | $RULEID – $ALERTMSG”

# Send result to zabbix for logging and notification alerts.
$ZabbixSender –zabbix-server $ZabbixServer –port $ZabbixPort –host $HOSTNAME –key $ZabbixKeyName –value “$ZMSG”

if [ "$DEBUG" = "true" ]
then
echo “$ZabbixSender –zabbix-server $ZabbixServer –port $ZabbixPort –host $HOSTNAME –key $ZabbixKeyName –value ‘$ZMSG’” >> /tmp/zabbix-test.log
echo “ACTION: $ACTION” >> /tmp/zabbix-test.log
echo “USER: $USER” >> /tmp/zabbix-test.log
echo “IP: $IP” >> /tmp/zabbix-test.log
echo “ALERTID: $ALERTID” >> /tmp/zabbix-test.log
echo “ALERTLVL: $ALERTLVL” >> /tmp/zabbix-test.log
echo “RULEID: $RULEID” >> /tmp/zabbix-test.log
echo “———————————” >> /tmp/zabbix-test.log
fi

exit 0

UPDATE 09-24-2010 – If you happen to use full domain names, the regex for getting the name needs to allow “.”  – The script can be found zabbix-alert-201009; or below.  Again be sure to watch for broken wrapped lines:

#!/bin/sh
#
# Submits an OSSEC alert as a passive service check result to zabbix.
#
# Author: David M. Zendzian
#
# Idea from Dave Stycos post: http://groups.google.com/group/ossec-dev/browse_thread/thread/e29c5d71926b8af5
#
# Updated 7/15/10 – using 2 server hosts in case using proxies and local host is monitored directly by central server and not proxy.
#
# This script is Public Domain, and is provided AS-IS.  There is no
# warranty, and no support given for its contents.
#
# Version 1.1: Jul. 15, 2010
#

DEBUG=”true”
ACTION=$1
USER=$2
IP=$3
ALERTID=$4
RULEID=$5

if [ "$DEBUG" = "true" ]
then
echo “NOTICE: Starting Zabbix sender” >> /tmp/zabbix-test.log
fi

LOCAL=dirname $0;
cd $LOCAL
cd ../
PWD=pwd
UNAME=uname

# Zabbix Sender
ZabbixSender=”/usr/bin/zabbix_sender”
#ZabbixSender=”/usr/sbin/zabbix_sender”

# Zabbix Server
# Set server2 to be the same if all hosts monitored through proxy or the same server; otherwise
# set ZabbixServer to the proxy for non-localhost and then ZabbixServer2 to the host that the
# local proxy uses
ZabbixServer=<Server/Proxy>
ZabbixServer2=<Server for “Localhost”>

# Zabbix Port
ZabbixPort=10051

# All alerts will be processed by Zabbix under this key.
ZabbixKeyName=OSSEC

# Check that zabbix_sender file exists.
if [ ! -w $ZabbixSender ]; then
logger -p local0.err “$0: File $ZabbixSender not found.  Exiting.”
if [ "$DEBUG" = "true" ]
then
echo “ERROR: No Zabbix Sender” >> /tmp/zabbix-test.log
fi
exit 1
fi

# Getting alert time
ALERTTIME=echo "$ALERTID" | cut -d  "." -f 1

# Getting end of alert
ALERTLAST=echo "$ALERTID" | cut -d  "." -f 2

# Getting full alert
ALERTTEXT=grep -A 10 "$ALERTTIME" $PWD/../logs/alerts/alerts.log | grep -v ".$ALERTLAST: " -A 10

# Extract host (agent) name from alert.
HOSTNAME=echo "$ALERTTEXT" | sed -n '1,1s/^.*\:[0-9][0-9]\:[0-9][0-9][^A-Za-z0-9_]*\([-A-Za-z0-9_.]*\)\->.*$/\1/p'

# if hostname alert wasn’t from local host, the host value is “(hostname) ip”, which extracts differently
if [ "$HOSTNAME" = "" ]
then
HOSTNAME=echo "$ALERTTEXT" | sed -n '1,1s/^.*\:[0-9][0-9]\:[0-9][0-9] (\([-A-Za-z0-9_.]*\)) .*\->.*$/\1/p'
fi
if [ "$HOSTNAME" = "" ]
then
if [ "$DEBUG" = "true" ]
then
echo “ERROR: No Hostname” >> /tmp/zabbix-test.log
fi
exit 0
fi

# if the local host is a proxy then monitored items submitted through proxy, localhost probably monitored directly from central server (if not change comment this out)
LOCALHOSTNAME=hostname -s
if [ "$HOSTNAME" = "$LOCALHOSTNAME" ]
then
ZabbixServer=$ZabbixServer2
fi

if [ "$DEBUG" = "true" ]
then
echo “ZabbixServer: $ZabbixServer” >> /tmp/zabbix-test.log
fi

# Extract alert level from alert.
ALERTLVL=echo "$ALERTTEXT" | sed -n '2,2s/^.*(level \([0-9]*\).*$/\1/p'

# Extract description from alert.
ALERTMSG=echo "$ALERTTEXT" | sed -n '5,5p'

# Create Alert message
ZMSG=”$ALERTID | $ALERTLVL | $RULEID – $ALERTMSG”

# Send result to zabbix for logging and notification alerts.
$ZabbixSender –zabbix-server $ZabbixServer –port $ZabbixPort –host $HOSTNAME –key $ZabbixKeyName –value “$ZMSG”

if [ "$DEBUG" = "true" ]
then
echo “$ZabbixSender –zabbix-server $ZabbixServer –port $ZabbixPort –host $HOSTNAME –key $ZabbixKeyName –value ‘$ZMSG’” >> /tmp/zabbix-test.log
echo “ACTION: $ACTION” >> /tmp/zabbix-test.log
echo “USER: $USER” >> /tmp/zabbix-test.log
echo “IP: $IP” >> /tmp/zabbix-test.log
echo “ALERTID: $ALERTID” >> /tmp/zabbix-test.log
echo “ALERTLVL: $ALERTLVL” >> /tmp/zabbix-test.log
echo “RULEID: $RULEID” >> /tmp/zabbix-test.log
echo “———————————” >> /tmp/zabbix-test.log
fi

exit 0

For this integration to work, the host names used in OSSEC need to match the host names defined in Zabbix.  If they do not match, then zabbix_sender results will not make it into the hosts items correctly.

The easiest way to setup zabbix is to setup a template.  This template will define the application, item and triggers for OSSEC and can easily be linked to the hosts you are monitoring.

In zabbix goto Configure/Hosts and select ‘templates’.

Create a new template called OSSEC and be sure to add it to the Templates group.

Zabbix 1.6: Add OSSEC Template

Zabbix 1.8: Add OSSEC Template

With the template created, a new application needs to be created so the OSSEC items have a place to be organized.  Staying in Configuration / Hosts (for Zabbix 1.6 & 1.8), in the drop down on the upper right, select “Applications” and then from the group and host dropdown, select Templates / Template_OSSEC that was just created.  There should be no Applications in the list. If you do not see Template_OSSEC that was just created, go back to templates and edit the OSSEC item and be sure that it is in the “Templates” group.

Click on “Create application” and create “OSSEC Monitor” or whatever you want to call it.

Zabbix 1.6: Create OSSEC Template Application

Zabbix 1.8: Create OSSEC Application

Now we are ready to create the Item & Trigger for the OSSEC data.  In Zabbix 1.6, select “Configuration / Items” and select “Templates / Template_OSSEC” from the group and host selections and then click on “Create Item.”  The important item here is the Key which we will setup as OSSEC and is required to match the ZabbixKeyName in the zabbix-alert.sh script on the OSSEC server.

Zabbix 1.6: Create OSSEC Item

In Zabbix 1.8, remain in the “Configuration / Hosts” menu and in the upper right drop down select “Items”.  Click on “Create Item.”  When the new form is up, click on “Select” for the Host and select Template_OSSEC that we created above.  The same values will be set as with Zabbix 1.6.

Zabbix 1.8: Create OSSEC Item

As you can see, the item is a “Text” type getting data from a Zabbix Trapper event.  The things to not forget here are to enter your OSSEC server(s) in the Allowed Host line and to select the OSSEC Monitor application.

The next step is to create a trigger which will let us know when new data has arrived from OSSEC. Select “Configuration / Triggers” in Zabbix 1.6.

If you are using 1.8 then remain on the “Configuration / Hosts” page and select “Triggers” from the dropdown box on the upper right.

It should default to the Template_OSSEC host, but if it doesn’t then select Templates from groups and the Template_OSSEC host.

There are several ways to monitor the OSSEC text data, but I have selected to alert if there is new data in the last 10 minutes.  As you can see from the screenshot, the expression I used is {Template_OSSEC:OSSEC.nodata(600)}#1.  This works because the nodata(600) will return a 1 if no data is received in the time period specified (600 sec or 10 min).  So if it ever returns anything other than 1, we have new data.  For more information on trigger functions, consult the zabbix manual.

Now click on “Create Trigger” go create the trigger.

Zabbix 1.6: Create OSSEC Zabbix Trigger

Zabbix 1.8: Create OSSEC Zabbix Trigger

I have set the alert severity to “average” but you may want to change that depending on your needs.  The zabbix actions we will define will send all OSSEC alerts so the severity will not really matter.  One thing that is worth examining is to change the OSSEC item value to log instead of text which could allow for log severity and other values that could be used with the OSSEC alert levels; but that project is for another time.

Updated: 4/16/10 – What I have done for alerting based on level is to use the “short” ZMSG message type in the zabbix-alert.sh script and define a trigger such as:

({Template_OSSEC:OSSEC.nodata(600)}#1)&({Template_OSSEC:OSSEC.str( | 1 | )}#1)&({Template_OSSEC:OSSEC.str( | 2 | )}#1)&({Template_OSSEC:OSSEC.str( | 3 | )}#1)

What this trigger does is requires all 4 conditions to be met (&=”AND” between each item test).  The first is that there is new data within the last 10 minutes, the other 3 are requirements that the new data does not contain | 1 | or | 2 | or | 3 |, which would be OSSEC alert levels 1, 2 and 3.  If you use the longer ZMSG then the str values would be like: {Template_OSSEC:OSSEC.str( | Level: 3 | )}#1

Only 2 things left to do and the OSSEC/Zabbix integration is done.  These are to create actions for OSSEC events and to link the OSSEC template to the hosts you are monitoring with OSSEC.

In our local zabbix configuration I have created a “Security Administrator” group that receives IDS and other security events and will be using that to specify who receives the alerts.  You can modify these settings based on your local policy and zabbix configuration.

As you will also see in the following screenshot, I have modified the default message.  This allows me to receive the full data from the OSSEC event through {ITEM.LASTVALUE}.  I have also shortened the message so I can receive the details I want on my SMS alerts which have a smaller size than full emails.

I have tried to enable escalations for OSSEC alerts, however the way that zabbix handles items is that it will only look at the “active” triggers & items, what this means is that when a new OSSEC alert comes in and is added to the items database, the trigger is alerted but after 10 minutes it will “go away”.  There is no way, currently, to have a trigger depend on it’s being “Ack’d” which would be preferred for security, log and other events that just shouldn’t go away until an admin acks what happened.  There is a currently active zabbix feature request requesting this, so please go vote it up so we can see it added in the near future!

Zabbix 1.6: Create OSSEC Action

Zabbix 1.8: Create OSSEC Action

All that is left is to link your hosts to the OSSEC template.  The OSSEC alert submits data to zabbix based on the host names defined in OSSEC.  So once again, please be sure the names used match in both systems.

If you do not know how to link the OSSEC template, simply go to “Configuration / Hosts” and edit the hosts that are monitored by OSSEC.  You need to link every host as the alerts will be coming in directly to each unique host.  The example below is for one of our ossec servers, but the configuration should be the same for all OSSEC monitored hosts.

Zabbix 1.6: Host OSSEC Template Link

Zabbix 1.8: Host OSSEC Template Link

This should be it.  If you have already restarted OSSEC then you just need to create an event it will alert on (logging onto monitored host, creating “segfault” log messages: logger “segfault”, etc).  In my quick test, seen below, I did a failed logon (bad pw) and within a few seconds I had my jabber alert pop up and a sms message arrive on my phone!

If you have any problems, you can set DEBUG=true in the zabbix-alert.sh and it will log out what is being sent to zabbix into /tmp/zabbix-test.log.

If OSSEC is not running active-alerts, you may want to jump on #ossec on the openprojects IRC and get some assistance or search google.

Good luck!

David M. Zendzian | Managing Partner | ZZ Servers
268 Bush St. #4127 | San Francisco, CA 94104

Business Hosting Solutions | PCI | HIPAA
Managed Hosting Specialists