Preamble
Either curl and jq or a python script is needed to handle the data returned when polling the DSNet Manager as the data is returned in JSON format. SNMP support for querying the DSNet system doesn’t seem possible, Cleversafe is designed to reply to various API calls.
The two main API calls used for capacity reporting is:
listStoragePools.adm listVaults.adm
A more complete API which can can take arguements within the URI, which may be needed for throughput analysis, is:
viewSystem.adm
An account has been created on the Manager with Operator role privilages but no rights to any Vaults, it can then call these APIs for reporting.
Simple testing of the APIs and to view the returned data run the following commands:
curl -u monitor:monitor -k "https://<DSNET manager>/manager/api/{json}/1.0/listStoragePools.adm"
curl -u monitor:monitor -k "https://<DSNET manager>/manager/api/{json}/1.0/listVaults.adm"
This gives a full list of all vaults or storage pools in the DSNet in JSON format.
The -k switch is important as we’re using self signed certificates, also once in production remember the –silent switch so it doesn’t confuse Zabbix.
Storage Pools
To try and mangle the returned data with within a single command rather than a Python script I use the jq command to prettyfy the data so it then be manipulated in a way to be used as a Custom Parameter.
I still do not fully understand what I’m doing but after trial and error I managed to get jq to return all the configured Storage Pools’ raw capcity & utilisation:
jq '.responseData.storagePools | map(.name,.utilization,.capacity)'
This returned a list of all the configured Pools, for example:
[ "pool1", 110224329670656, 428679318671360, "pool2", 10042207012782080, 11431468335628288, "pool3", 7052230679418873, 7978074181816320, "pool4", 5027274442085489, 6846819800678400, "pool5", 37312856560, 8001889328627712 ]
The next step is to then search based on Pool name, as these are rarely created a funky discovery rule won’t be required. All we need to supply is the Manager and Storage Pool names to the custom parameter for use in the command.
Using pool1 as an example we would want to extract the first number for the used space and the second number for the total size. It seemed that the returned data need a bit more mangling than thought as I found it difficult to extract a single number with no white spaces or commas:
grep -A 2 pool1 | tr -s '\n' ',' | awk -F, '{print $2}' | awk '{print $1}'
Which returns the number:
110224329670656
So initial Cleversafe Pool custom parameters would be:
UserParameter=clsf_pool_size_total[*], curl --silent -u monitor:monitor -k "https://$1/manager/api/{json}/1.0/listStoragePools.adm" | jq '.responseData.storagePools | map(.name,.utilization,.capacity)' | grep -A 2 $2 | tr -s '\n' ',' | awk -F, '{print $$3}' | awk '{print $$1}'
UserParameter=clsf_pool_size_used[*], curl --silent -u monitor:monitor -k "https://$1/manager/api/{json}/1.0/listStoragePools.adm" | jq '.responseData.storagePools | map(.name,.utilization,.capacity)' | grep -A 2 $2 | tr -s '\n' ',' | awk -F, '{print $$2}' | awk '{print $$1}'
Remember to restart the Zabbix agent:
systemctl restart zabbix-agent
To call the parameter correctly we would use in the web GUI:
Name: Cleversafe Pool <item> <pool name> Type: Zabbix Agent Key: <custom parameter name>[<manager name>,<pool name>] Host Interface: 127.0.0.1:10050 Type of Info: Numeric (Unsigned) Data Type: Decimal Update Interval: 600
Vaults
For Vaults it’s a bit more complex as they can be created and removed more often than pools especially if people create/delete them as they see fit for testing etc. So it seems prudent to create a discovery rule.
Discovery Rule
The following command returns all the current Vaults as individual items:
for i in `curl --silent -u monitor:monitor -k "https://<DSNET manager>/manager/api/{json}/1.0/listVaults.adm" | jq '.responseData.vaults | map(.name)' | tr -d '"[][:space:]' | tr -s ',' '\n'`; do echo $i; done
Hoewever Zabbix is expecting JSON formatted return and even though I’ve tried to supply raw or adjusted JSON formatted lists using jq Zabbix didn’t like it (or maybe I didn’t wait enough).
So I’ve created a bash script that takes the DSNet manager’s name and formats the list returned into something that Zabbix likes, setting a Zabbix variable to the Vault name. I used the following help page section as a template.
EDIT:
Turns out my first bit of code actually dropped one of the discovered vaults. I found this out when using the same set up for Pool discovery. Below is the updated code to also correctly deal with a single vault/pool result as well:
#!/bin/bash
vaultarray=( $(curl --silent -u monitor:monitor -k "https://$1/manager/api/{json}/1.0/listVaults.adm" | jq '.responseData.vaults | map(.name)' | tr -d '"[][:space:]' | tr -s ',' ' ') )
numberofvaults=$(echo ${#vaultarray[@]})
whenlast=2
index=0
printf "{\n\t\"data\":[\n\n"
if [ $numberofvaults -eq 1 ]
then
printf "\t{ \"{#VAULTNAME}\":\"${vaultarray[$index]}\"}\n"
else
while [ $whenlast -le $numberofvaults ]
do
printf "\t{ \"{#VAULTNAME}\":\"${vaultarray[$index]}\"},\n"
index=$(( $index + 1 ))
whenlast=$(( $whenlast + 1 ))
done
printf "\t{ \"{#VAULTNAME}\":\"${vaultarray[$index]}\"}\n"
fi
printf "\n\t]\n"
printf "}\n"
It is placed in the zabbix agent’s directory on the monitoring server:
/etc/zabbix/zabbix_agentd.d
User defined parameter is then added to the Cleversafe config file:
/etc/zabbix/zabbix_agentd.d/userparameter_cleversafe.conf
This simply calls the script (in the same directory) and expects the managers hostname:
UserParameter=clsf-vault_disco[*], /etc/zabbix/zabbix_agentd.d/cleversafe_vault_disco.sh $1
A discovery rule is created with the following details, use the format:
<parameter name>[{HOST.HOST}]
Name: <name of rule>
Type: Zabbix agent
Key: clsf-vault_disco[{HOST.HOST}]
Update Interval: 3600
Item Prototypes
Now the Vaults are being discovered the Item Prototypes need to be defined for monitoring. There are many Vault metrics which can be monitored:
"allottedSize": 5286376205368973, "usableSize": 3680153594157688, "usedPhysicalSizeFromStorage": 5286376205368973, "usedLogicalSizeFromStorage": 5286291682631257, "estimateUsableUsedLogicalSizeFromStorage": 3119704460183038, "estimateUsableTotalLogicalSizeFromStorage": 3680153594157688, "softQuota": 2750000000000000, "hardQuota": 3250000000000000, "allotmentQuota": null, "allotmentUsage"
After talking to support I received the following reply:
usedPhysicalSizeFromStorage = RAW usage.
usedLogicalSizeFromStorage = consumed usage.
The vault graph on Dsnet manager shows RAW usage.
As Vaults resides in a Pool and multiple Vaults can sit in one Pool the vault itself doesn’t have a capacity metric. Using the capacity metric from the Pool it resides in will not give a true picture of available capacity so that shouldn’t be used.
The only consistent way to see available space for a Vault is to use the **hardQuota**, if none is set then nothing will be recorded and we’re none the wiser.
Another issue is that the quotas metric is based on logical not raw storage usage. Some Vaults will therefore show that they’re over their hard quota if we use the usedPhysicalSizeFromStorage metric; the next best thing is to use the estimateUsableUsedLogicalSizeFromStorage metric instead.
Firstly we build the custom parameters, all we need to supply is the Manager and Vault names for use in the command:
UserParameter=clsf_vault_size_used[*], curl --silent -u monitor:monitor -k "https://$1/manager/api/{json}/1.0/listVaults.adm" | jq '.responseData.vaults | map(.name,.estimateUsableUsedLogicalSizeFromStorage,.hardQuota,.softQuota)' | grep -A 2 $2 | tr -s '\n' ',' | awk -F, '{print $$2}' | awk '{print $$1}'
UserParameter=clsf_vault_hardquota[*], curl --silent -u monitor:monitor -k "https://$1/manager/api/{json}/1.0/listVaults.adm" | jq '.responseData.vaults | map(.name,.estimateUsableUsedLogicalSizeFromStorage,.hardQuota,.softQuota)' | grep -A 2 $2 | tr -s '\n' ',' | awk -F, '{print $$3}' | awk '{print $$1}'
UserParameter=clsf_vault_softquota[*], curl --silent -u monitor:monitor -k "https://$1/manager/api/{json}/1.0/listVaults.adm" | jq '.responseData.vaults | map(.name,.estimateUsableUsedLogicalSizeFromStorage,.hardQuota,.softQuota)' | grep -A 2 $2 | tr -s '\n' ',' | awk -F, '{print $$4}' | awk '{print $$1}'
Once there configured an item prototype can be created for each parameter using the discovery rule defined variable and the DSNet’s hostname variable:
Name: {#VAULTNAME} <metric>
Type: Zabbix agent
Key: <custom param>[{HOST.HOST},{#VAULTNAME}]
Type of info: Numeric (unsigned)
Data type: Decimal
Update interval: 600
Graph Prototype
Create a graph as normal adding the newly created item prototypes, again using the discovery rule’s variable:
Name: Cleversafe Vault {#VAULTNAME} utilisation
Y axis MIN value: Fixed
In the items section click on **Add**.
In the pop up window add the item prototypes created earlier.
Click on the **Add** button.