Monitoring Panasas with Opennms

System Summary

The Panasas cluster version I’m using is  5.5.1-1098440.68. The Opennms version I’m using is 1.12.9 running on RHEL 6.3.  I’ve installed:

net-snmp-utils-5.5-41.el6.x86_64
net-snmp-libs-5.5-41.el6.x86_64
net-snmp-5.5-41.el6.x86_64
jdk-7u67-linux-x64.rpm

Quick steps to adding new system

Once the Opennms system has been configured to monitor Panasas systems follow the steps below:

  • Only configure the director blade running the realm service RM President
  • Add the Director Blade IP address to:
    • PANASAS package in /opt/opennms/etc/collectd-configuration.xml
    • SNMPv1 section of /opt/opennms/etc/collectd-configuration.xml
  • In the /opt/opennms/etc/poller-configuration.xml file add the first and last IP address of the system’s configured range to:
    • An exclude range in the example1 package
    • An include range in the panasas package
  • Restart the Opennms service
  • Add a new node using the Director Blade’s IP address to the Panasas provisioning requisition group.
  • Click on the provisioning requisition group’s Synchronize button

Adding SNMP MIBs

One off Steps

To get the Panasas MIBs directly off the system you can get the entire Panasas SNMP MIBS via running the following command from the CLI:

[pancli] collect_logs match /pan/mibs/*

You can then scp them off and extract the resultant archive. Or via the GUI Configuration > SNMP > click on Download MIB definition button

Via the GUI to enable SNMP: Configuration > SNMP Set the community name I used public.

I then used the Opennms MIB compiler: Admin > SNMP MIB Compiler

Uploaded the MIBs and compiled them. The formatting wasn’t perfect for Opennms for the following MIB files:

PANASAS-HW-MIB-V1.mib
PANASAS-PERFORMANCE-MIB-V1.mib
PANASAS-NOTIFY-MIB-V1.mib

To cut a long story short there were some missing commas, changed the case of LACP to lacp, added a missing import of RFC1213-MIB, removed extra quote marks, removed extra commas and corrected some object references to obsolete OBJECT names, phew!

Once they’d been compiled OK Opennms had created the required data collection group files in /opt/opennms/etc/datacollection

PANASAS-BLADESET-MIB-V1.xml
PANASAS-PERFORMANCE-MIB-V1.xml
PANASAS-SYSTEM-MIB-V1.xml
PANASAS-VOLUMES-MIB-V1.xml

It had also create the snmp graph config files in /opt/opennms/etc/snmp-graph.droperties.d/ 

PANASAS-BLADESET-MIB-V1-graph.properties
PANASAS-PERFORMANCE-MIB-V1-graph.properties
PANASAS-SYSTEM-MIB-V1-graph.properties
PANASAS-VOLUMES-MIB-V1-graph.properties

Configure Opennms for Panasas data collection

One Off Steps

All the above configuration files created by the MIB compiler will not start collecting anything even after the system has been added. You need to add some more configurations:

  • SNMP collection
  • Data collection Groups (created by the MIB compiler)
  • System Definitions

SNMP collection

Via the GUI go to:

Admin > Manage SNMP Collections and Data Collection Groups > SNMP Collections Tab

Click on Add SNMP Collection button

Enter in a relevant name, Panasas

Set SNMP Storage Flag to select

In the Include Collections click Add and select and add all the relevant Panasas collection groups:

snmp_coll

Click Save

This will add the configuration to the /opt/opennms/etc/datacollection-config.xml file

System Definitions

systemDef or System Definition are configurations to the various data collection groups that have been created. No SNMP data will be polled without these added to the data collection groups.

You can do this via the GUI rather than manually adding the systemDef configs to the bottom of the relevant /opt/opennms/etc/datacollection/<whatever>.xml files by hand. Go to:

Admin > Manage SNMP Collections and Data Collection Groups > Data Collection Groups Tab

In the Select Data Collection Group File locate one of the Panasas xml files that was created. Then click on the System Definitions Tab below:

sys_def_panasas01

Click on the Add System Definition button.

Enter in a relevant Group Name, this is to relate to the MIB groups within the collection group added below.

The System OID/Mask was the the key part of the entire process. If this isn’t right nothing will work. For all of the Panasas monitoring I used a Mask based on the System OID. To get this info I used the following command taken from the SNMP Reports How-to:

snmpget -v2c -On -c <community> <host> sysObjectID.0

For example:

[root@vinz opennms]# snmpget -v 1 -On -c public 10.7.18.16 sysObjectID.0 .1.3.6.1.2.1.1.2.0 = OID: .1.3.6.1.4.1.10159.1.3.0
[root@vinz opennms]#

Initially you would take the entire OID returned with a trailing full stop, to test that it works run the following (I pipe to more so I can Ctrl+C out if it works):

snmpwalk -v 1 -c public 10.7.18.16 .1.3.6.1.4.1.10159.1.3.0 | more

This didn’t work but dropping the last number did:

[root@vinz opennms]# snmpwalk -v 1 -c public 10.7.18.16 .1.3.6.1.4.1.10159.1.3.0 | more
[root@vinz opennms]# snmpwalk -v 1 -c public 10.7.18.16 .1.3.6.1.4.1.10159.1.3 | more
SNMPv2-SMI::enterprises.10159.1.3.1.1.0 = INTEGER: 100
SNMPv2-SMI::enterprises.10159.1.3.1.2.1.1.1 = INTEGER: 1
SNMPv2-SMI::enterprises.10159.1.3.1.2.1.1.2 = INTEGER: 2
<SNIP>

So use the following for the mask = .1.3.6.1.4.1.10159.1.3.

UPDATE using the above mask can miss lots of OIDs that could be useful. For example the OID for the human readable name for a blade contains .1.3.6.1.4.1.10159.1.2. so would never be captured. The recommended mask is therefore .1.3.6.1.4.1.10159.1.

In the MIB Groups pull down list choose the required MIB group and then click on Add Group

sys_def_panasas02

Click Save

This will then add the follwoing to the data collection group’s xml file:

<systemDef name="Blade Set">
<sysoidMask>.1.3.6.1.4.1.10159.1.3.</sysoidMask>
<collect>
<includeGroup>panBSetTable</includeGroup>
</collect>
</systemDef>
</datacollection-group>

Repeat this for all required data collection groups and MIB groups.

Configuring the collection service

The following section configures Opennms’ collectd service to use the newly created SNMP collection and it’s collection groups and against what. This config within the /opt/opennms/etc/collectd-configuration.xml needs to be updated whenever a new Panasas system is added. Below is the configuration with two systems, panasas01 and panasas-ota:

<package name="PANASAS">
<filter>IPADDR != '0.0.0.0'</filter>
<specific>10.7.18.16</specific>
<specific>10.10.4.2</specific>
<service name="SNMP" interval="300000" user-defined="false" status="on">
<parameter key="collection" value="Panasas"/>
</service>
</package>

Add Panasas System to Opennms SNMP config

Steps Needed Per Panasas System

Panasas only supports SNMP v1 so you’ll need to add the following configuration to the /opt/opennms/etc/snmp-config.xml file:

<definition version="v1" retry="3">
<specific>10.7.18.16</specific>
<specific>10.10.4.2</specific>
</definition>

Panasas Poller Package

The file /opt/opennms/etc/poller-configuration.xml handles the services that Opennms will monitor. Out of the box there is a default package, called example1, which encompasses all possible IP address’. This default package though will create some annoying polling service outages which aren’t required.

The customisations I did to this file were to add an exclusion entry in the default package for the Panasas’ entire IP range:

<package name="example1">
<filter>IPADDR != '0.0.0.0'</filter>
<exclude-range begin="10.7.18.16" end="10.7.18.250" />
<include-range begin="1.1.1.1" end="254.254.254.254" />

Then created a new panasas entry pretty much copied from example1 but only including the relevant services: ICMP, SNMP and SSH

<package name="panasas">
<filter>IPADDR != '0.0.0.0'</filter>
<include-range begin="10.7.18.16" end="10.7.18.250" />
<rrd step="300">
<rra>RRA:AVERAGE:0.5:1:2016</rra>
<rra>RRA:AVERAGE:0.5:12:1488</rra>
<rra>RRA:AVERAGE:0.5:288:366</rra>
<rra>RRA:MAX:0.5:288:366</rra>
<rra>RRA:MIN:0.5:288:366</rra>
</rrd>
<service name="ICMP" interval="300000" user-defined="false" status="on">
<parameter key="retry" value="2" />
<parameter key="timeout" value="3000" />
<parameter key="rrd-repository" value="/opt/opennms/share/rrd/response" />
<parameter key="rrd-base-name" value="icmp" />
<parameter key="ds-name" value="icmp" />
</service>
<service name="SNMP" interval="300000" user-defined="false" status="on">
<parameter key="oid" value=".1.3.6.1.2.1.1.2.0" />
</service>
<service name="SSH" interval="300000" user-defined="false" status="on">
<parameter key="retry" value="1" />
<parameter key="banner" value="SSH" />
<parameter key="port" value="22" />
<parameter key="timeout" value="3000" />
<parameter key="rrd-repository" value="/opt/opennms/share/rrd/response" />
<parameter key="rrd-base-name" value="ssh" />
<parameter key="ds-name" value="ssh" />
</service>
<downtime interval="30000" begin="0" end="300000" /><!-- 30s, 0, 5m -->
<downtime interval="300000" begin="300000" end="43200000" /><!-- 5m, 5m, 12h -->
<downtime interval="600000" begin="43200000" end="432000000" /><!-- 10m, 12h, 5d -->
<downtime begin="432000000" delete="true" /><!-- anything after 5 days delete -->
</package>

This needs to be updated with every new system

Adding the Panasas System as a node

Create a Provisioning Requisition entry, this are new names for provisioning groups, via Admin > Manage Provisioning Requisitions

In the text entry field enter in a relevant name then click on Add New Requisition button.

Click on Edit to the right of Requisition

Click on Add Node

Enter in a descriptive name in Node

Click Save

Click on the Add Interface link

Enter in the relevant IP address and description, say dir blade, leave Status and SNMP Primary as is the click Save

Click on Add Service link and add the following services

  • SSH
  • SNMP
  • ICMP

Click on Done

The system will be added and data collection and graphing will begin. You can speed this up by clicking on the Synchronize button.

Human Readable Resource Graph Entries

Out of the box the system will list each Panasas node resource graph groups by the contents of the label section of the resourceType section in the corresponding data collection xml document. Each instance within said resource group, say a blade or a blade set, will be listed by what is returned by the resourceLabel part of the resourceType section.

What is then shown on the web page might not be as identifying as liked. For example the blade set configuration, by default the resource graph group is listed as panBSetEntry and any instances, i.e. bladesets, are listed as OIDs:

<resourceType name="panBSetEntry" label="panBSetEntry" resourceLabel="${index}">

This is not very user friendly, and it’s even worse for per blade resource graphs.

To improve on this the key component, resourceLabel is changed, using the Opennms wiki page as reference, to ${string(index)}. This returns for Bladeset and Volume resource groups a satisfactory and mappable list.

<resourceType name="panBSetEntry" label="Panasas Blade Sets" resourceLabel="${string(index)}">
<resourceType name="panVolEntry" label="panVolEntry" resourceLabel="${string(index)}">

resource_group_panasas01

This hasn’t worked for the actual blades as this returns a slightly more usable but still improvable Hardware serial number.

For example for the Director Blades, the resourceType is panPerfDirectorEntry. If we look at that entry in the corresponding MIB file we see the INDEX is set to the hardware serial number:

panPerfDirectorEntry OBJECT-TYPE
SYNTAX PanPerfDirectorEntry
MAX-ACCESS not-accessible
STATUS current
DESCRIPTION
"A row in panPerfDirectorTable."
INDEX { panHwNodeHwSN }
::= { panPerfDirectorTable 1 }

Updating the INDEX entry to:

INDEX { panHwNodeHwSN, panHwNodeName }

Then also updating the IMPORTS section from:

IMPORTS
<SNIP>
panHwNodeHwSN
FROM PANASAS-HW-MIB-V1

To:

IMPORTS
<SNIP>
panHwNodeHwSN, panHwNodeName
FROM PANASAS-HW-MIB-V1

Then edit the refering MIB PANASAS-HW-MIB-V1 and make sure panHwNodeName is set as an index. In the organisation part at the top, in this example it’s part of the panHwNode > panHwNodeTable > panHwNodeEntry section:

panHwNodeName        [Index]

And then again in the definition area where the index is specified, in this example panHwNodeEntry:

panHwNodeEntry OBJECT-TYPE
SYNTAX PanHWNodeEntry
MAX-ACCESS not-accessible
STATUS current
DESCRIPTION
"An entry in panHwNodeTable."
INDEX { panHwNodeHwSN, panHwNodeName }
::= { panHwNodeTable 1 }

Via the GUI go into MIB compiler, right click on the compiled MIB and choose Generate Data Collection. This will OVERWRITE the current data collection XML file currently in /opt/opennms/etc/datacollection/ so if you have customisations backup the file first.

You will then have to recreate within the GUI the Data collection group and any defined System Definitions related to the MIB.

You do not have to over write your graphs and loose any customisations, nor do you have to restart OpenNMS for this to take effect.

I’VE STILL NOT GOT BLADES LISTED BY THEIR NAME

Customised Graphs

I wanted to groups together some of the monitored metrics otherwise each page would be filled with lots of single graphs. So for example I combined the following storage blade metrics:

  • Total Disk Capacity
  • Used Disk Capacity
  • Available Disk Capacity
  • Reserved Disk Capacity

I commented out all the individual sections in the /opt/opennms/etc/snmp-graph.properties.d/PANASAS-PERFORMANCE-MIB-V1-graph.properties file and created a new section:

reports=panPerfDirectorTable.panPerfDirectorCpuUtil, \
<SNIP>
panPerfStorageTable.StorageBladeCapacity, \
<SNIP>
report.panPerfStorageTable.StorageBladeCapacity.name=Capacity info in Giga Bytes (GB).
report.panPerfStorageTable.StorageBladeCapacity.columns=panPerfStoragCapTot,panPerfStoraCapUsed,panPerfStorCapAvail,panPerfStorCapReser
report.panPerfStorageTable.StorageBladeCapacity.type=panPerfStorageEntry
report.panPerfStorageTable.StorageBladeCapacity.description=Capacity info in Giga Bytes
report.panPerfStorageTable.StorageBladeCapacity.command=--title="Capacity info in Giga Bytes (GB)" \
DEF:totalcap={rrd1}:panPerfStoragCapTot:AVERAGE \
DEF:usedcap={rrd2}:panPerfStoraCapUsed:AVERAGE \
DEF:availcap={rrd3}:panPerfStorCapAvail:AVERAGE \
DEF:resvcap={rrd4}:panPerfStorCapReser:AVERAGE \
LINE1:totalcap#ff0000:"Total disk capacity" \
GPRINT:totalcap:AVERAGE:"Avg\\: %8.2lf %s" \
GPRINT:totalcap:MIN:"Min\\: %8.2lf %s" \
GPRINT:totalcap:MAX:"Max\\: %8.2lf %s\n" \
LINE1:usedcap#0000ff:"Used disk capacity" \
GPRINT:usedcap:AVERAGE:"Avg\\: %8.2lf %s" \
GPRINT:usedcap:MIN:"Min\\: %8.2lf %s" \
GPRINT:usedcap:MAX:"Max\\: %8.2lf %s\n" \
LINE1:availcap#00ff00:"Available disk capacity" \
GPRINT:availcap:AVERAGE:"Avg\\: %8.2lf %s" \
GPRINT:availcap:MIN:"Min\\: %8.2lf %s" \
GPRINT:availcap:MAX:"Max\\: %8.2lf %s\n" \
LINE1:resvcap#ffff00:"Reserved disk capacity" \
GPRINT:resvcap:AVERAGE:"Avg\\: %8.2lf %s" \
GPRINT:resvcap:MIN:"Min\\: %8.2lf %s" \
GPRINT:resvcap:MAX:"Max\\: %8.2lf %s\n"

Then restarted the Opennms services.

3 thoughts on “Monitoring Panasas with Opennms”

  1. While OpenNMS looks powerful, it is the most complicated one when it comes to configuration.

    I’ve never dealt with SNMP before, but i got a device from a company called Packet Power (power monitoring) that has SNMP interface, i was able to successfully poll custom data from the device through the vendor’s MIB using PRTG network monitor, the interface is so easy.

    With open NMS, there are so many confusing things, stuff you can do multiple ways, here is my problem:
    How on Earth i am going to scan/poll/obtain snmp data from a device with a custom MIB ? here is what i’ve done so far:

    1. upload MIB,compiled with no errors
    2. generate data collection, with system definitions for kind of a device within the MIB.
    3. added snmp interface (Home =>Admin=> Configure SNMP by IP)
    4. added prequesition (Home =>Admin Provisioning=> Requisitions)

    the node appears on node info menu, with snmp attributes polled right, but i don’t know how to tell that node to scan points from that MIB or data collection configured

    any idea ?

    cheers

    1. Found any solution? I’m in the same situation as you at the time of this comment.

      1. At work we’ve moved away from Panasas and I soon stopped using OpenNMS after the blog post as it was taking too much effort and I couldn’t fix what I wanted to, so sorry can’t help you.

Leave a Reply to nullgenome Cancel reply

Your email address will not be published.