Updated Hyper-V Zabbix monitoring

 

 In the past I have already posted some informations about monitoring Hyper-V via Zabbix.

In the last weeks I did enhance the monitoring solution with more details and visualisation. 

With the 2.10.0 release, we did add these new features:

• Checkpoint monitoring and alerting, including replication restorepoint handling
• Per VM memory and CPU usage
• VLAN Mode of network cards
• More VM details (TPM etc.)
• Enhanced VM host screen
• Also some work has gone into optimizing resource usage by the monitoring solution, be we had to make some tradeoffs for frequency versus server load

 

Some screenshots:

VM dashboard:

 VM replication dashboard:

See here for the full changelog.

As always, you find the latest files on the github repository 

Monitoring Hyper-V with Zabbix

Zabbix is a opensource monitoring solution, which helps you keep track of the health of your IT systems.

You can host it on your own infrastructure or use one of the many hosted zabbix SAAS offerings.

A very helpful feature is, to be able to write your own monitoring scripts and integrate them with dashboards and alerts.

Beside a core set of templates and scripts, there also exists a big list of templates and github repositories, which help you monitor almost everything.

If you don't find the perfect template for your use case, then you can still write your own scripts and templates and then integrate them in your Zabbix infrastructure.

For VMWare there is very good support out of the box (At least for the commercial VMWare products).

For Hyper-V from Microsoft there is no such support built it.

For this reason, I have contributed to a project, which was adding hyper-v monitoring to Zabbix.

As the project owner did give up the project, I did tyke over the project and did smal improvements over the years. But as the requirements did start to rise, I had to rewrite the whole thing from scratch.

This way I could start with a clean base and also use all the new features added in Zabbix 4.x - 7.0. Two topics have been very usefull in that part: Calculated/Dependent items and the JSON + JS post- and pre- processing.

The result is a brand new monitoring script and associated templates for Zabbix 7.x and up. You can find the solutions in this github repository.

If you use Zabbix and Hyper-V, give it a try and comment on this post or give feedback via github. I'm interested to hear, what works good and where I could improve the monitoring.

On a technical note, I did heavily use Claude for this new implementation, I will follow up with my learnings of the AI usage for this scenario. As always, there is good and there is bad to tell about, just stay tuned for the next article.

Install Zabbix agent on TrueNAS Scale

In my previous posts I did give instructions on how to monitor TrueNAS CORE (or FreeNAS) via Zabbix.

Unfortunatly this does not work for TrueNAS SCALE, as it's Linux based, but this is no real distro where you can install packages.

But you are not out of options, you can still monitor TrueNAS SCALE via SNMP.

There even exists a Zabbix template for this.

https://github.com/tr1plus/zabbix_truenas_SCALE_snmp/tree/main

What you have to do:

• Download the template from the Github link above
• Import the Template into your Zabbix Server
• Create the TureNAS host with SNMP interface (and secrets) enabled
• On the TrueNAS activate SNMP under System->Services
• Make sure password is correct and the SNMP service is set to "Start on boot"

Shortly after, you should see data comming into Zabbix.

If not, then check if your firewall(s) have Port 161 open for the SNMP monitoring.

Happy Monitoring 

 

 

Reactivate Zabbix agent in TrueNAS after update

When you have installed the Zabbix agent as mentioned in my previous post, then you will notice that Zabbix gets deactivated wqhen you install TrueNAS updates.

To get the agent running again, you only need to reactivate it in rc.conf

1. Enable daemon
   echo 'zabbix_agentd_enable="YES"' >> /etc/rc.conf

2. Start daemon:
   /etc/rc.d/zabbix_agentd start
3. Make config files persistent and survive reboots:
   cp /etc/rc.conf /conf/base/etc/ 
   cp /etc/rc.d/zabbix_agentd /conf/base/etc/rc.d/
mkdir /conf/base/etc/zabbix_agentd
cp /etc/zabbix_agentd.* /conf/base/etc/

Of course it's a good moment to install the latest zabbix agent too.

For this take the latest version matching your server version and copy the /bin and /sbin to /usr/local and restart the agent.

If someone has an idea, on how to make the "Enable daemon" step survive a minor TrueNAS upgrade, I would be glad for some hints ;)

Install Zabbix agent on TrueNAS

 In my previous article from 2018 I did mention how to install teh Zabbix agent on FreeNAS.

Since then two years are passed and FreeNAS has been replaced by TrueNAS provides the same functionality in a new version with a new name.

The steps for installation are remained the same, but you won't need to build the agent package yourself, you can use the packages provided from Zabbix for FreeBSD.

So lets see what you have to do now:

1. Download the binary package from the Zabbix webpage at https://www.zabbix.com/de/download_agents?version=4.0+LTS&release=4.0.25&os=FreeBSD&os_version=11.2&hardware=amd64&encryption=GnuTLS&packaging=Archive

It's no problem that FreeBSD 12.x is not listed, the 11.2 packages still work

2. Unpack the /bin and /sbin folders into /usr/local of your TrueNAS server

3. Unpack the /conf/* into /etc

4. Create a use and group zabbix so your agent wont run as root, you need to do this via GUI, otherwiese the account will be gone after reboot

5. Edit your /etc/zabbit_agentd:conf to match your needs

6. Enable daemon

echo 'zabbix_agentd_enable="YES"' >> /etc/rc.conf  

 

Create /etc/rc.d/zabbix_agentd:

#!/bin/sh

# PROVIDE: zabbix_agentd
# REQUIRE: DAEMON
# KEYWORD: shutdown
#
# Add the following lines to /etc/rc.conf.local or /etc/rc.conf to
# enable zabbix_agentd:
#
# zabbix_agentd_enable (bool): Set to NO by default.  Set it to YES to
#         enable zabbix_agentd.
#

. /etc/rc.subr

name="zabbix_agentd"
rcvar=zabbix_agentd_enable
start_precmd="zabbix_precmd"
required_files="/etc/zabbix_agentd.conf"

# read configuration and set defaultsc
load_rc_config "$name"
: ${zabbix_agentd_enable="NO"}
#: ${zabbix_agentd_pre:=/etc/${name}.pre.sh}

zabbix_agentd_conf="/etc/zabbix_agentd.conf"

if [ ! -z "$zabbix_agentd_conf" ] ; then
  zabbix_agentd_flags="${zabbix_agentd_flags} -c ${zabbix_agentd_conf}"
  required_files=${zabbix_agentd_conf}
fi

zabbix_precmd()
{
  if [ ! -z "$zabbix_agentd_pre" ] ; then
    if [ -e $zabbix_agentd_pre ] ; then
      . $zabbix_agentd_pre
    fi
  fi
}

command="/usr/local/sbin/${name}"

run_rc_command "$1"  run_rc_command "$1"

 Make executable:

1. chmod +x /etc/rc.d/zabbix_agentd
   

2. Start daemon:

   /etc/rc.d/zabbix_agentd start
   

3. Make config files persistent and survive reboots:

   cp /etc/rc.conf /conf/base/etc/
   cp /etc/rc.d/zabbix_agentd /conf/base/etc/rc.d/
   mkdir /conf/etc/zabbix_agentd
   cp /etc/zabbix_agentd.* /conf/base/etc/
   cp /etc/zabbix_agentd.* /conf/base/etc/ 

   Zabbix configuration

   Use the template from the zabbix wiki for the host monitoring http://zabbix.org/wiki/File:Template_OS_FreeNAS.xml

Monitor letsencrypt certificates with Zabbix

Monitor letsenrcypt certificates with Zabbix

Letsenrcypt is a great free system to automatically provide ssl/tls certificates for your website(s)
Zabbix is a great free system to monitor your IT infrastructure

One of the main features of Zabbix is the ability to extend it with any kind of monitoring scripts.
So when you do use letsencrypt certificates, you can also monitor them from Zabbix.

There exists a template you can add to the Zabbix server, and the required config and script files you place on the agents.

These can be found here.

It provides these features:
- Autodiscovery of all active letsencrypt certificates
- Monitor the lifetime of the certificates
- Trigger when the certificates are about to expire (Which means your auro renew does not work correctly)
- Monitor the certbot version
- Triggers when certbor version is below 0.28

The certbot version is important, since all certbot versions below 0.28 don't support the ACME-2.0 standard which is required by letsencrypt since the 13-february 2019.

Whith this you can relax about your ssl certificates, you won't have customers calling you in the morning, that their website(s) have an invalid/expired certificate.

You can even enhance the ssl monitoring with more Zabbix templates from here.
These allow you to monitor also ftps/imaps/smtps/pop3s and other TLS/SSL secured connections.

We use these in our own environment, to make sure everything runs smoothly and that we are warned of potential problems before the customers notice them.

Install Zabbix agent on FreeNAS 11.x

There is no official Zabbix Agent binary for FreeNAS, so it has to be compiled from source on another machine. There are several options to do so, I decided to just spin up a FreeBSD Vagrant VM to build the binary.
This post is based on the post from [1] with some enhancements 

For more recent howto see here

Vagrant VM

Setup VM & Build Environment

1. Create Vagrantfile:

   Vagrant.configure("2") do |config|
     config.vm.box = "freebsd/FreeBSD-11.1-STABLE"
     config.vm.base_mac = "000000123456"
   end
   

2. Start VM: vagrant up
3. Enter VM: vagrant ssh
4. Install dependecies:

   sudo pkg install -y curl autoconf automake gettext gcc pcre
   

Compile Zabbix Agent

1. Download source: https://www.zabbix.com/download_sources#tab:40LTS

2. curl -fsSL "https://sourceforge.net/projects/zabbix/files/ZABBIX%20Latest%20Stable/3.4.14/zabbix-3.4.14.tar.gz/download" | tar zxvf -
   cd zabbix-3.4.14

   curl -fsSL "https://sourceforge.net/projects/zabbix/files/ZABBIX%20Latest%20Stable/4.0.3/zabbix-4.0.3.tar.gz/download" | tar zxvf -

   cd zabbix-4.0.9

3. Build with TLS support for encryption:

   ./configure --enable-agent --with-openssl --enable-ipv6
   sudo make install
   

4. Transfer compiled binary to FreeNAS system:

   scp /usr/local/sbin/zabbix_agentd root@$FREENAS_IP:/usr/local/sbin/
   

5. Leave VM: exit
6. Destroy VM (optional): vagrant destroy

FreeNAS

After the zabbix_agentd binary is transferred to the FreeNAS system, it’s time to SSH into it to configure the agent.

Zabbix Agent Configuration

1. Add user:

   pw groupadd zabbix
   pw useradd zabbix -c "Daemon user for Zabbix agent" -d /nonexistent -s /usr/sbin/nologin -w no -g zabbix 

   Create zabbix user and group via WebUI, otherwise they won't suvive a reboot

2. Create /etc/zabbix_agentd.conf:

   Server=$ZABBIX_SERVER_IP
   ServerActive=$ZABBIX_SERVER_IP
   Hostname=$ZABBIX_AGENT_HOSTNAME
   LogFile=/tmp/zabbix_agentd.log
   

Daemon Configuration

1. Enable daemon:

   echo 'zabbix_agentd_enable="YES"' >> /etc/rc.conf
   

2. Create /etc/rc.d/zabbix_agentd:

   #!/bin/sh
   
   # PROVIDE: zabbix_agentd
   # REQUIRE: DAEMON
   # KEYWORD: shutdown
   #
   # Add the following lines to /etc/rc.conf.local or /etc/rc.conf to
   # enable zabbix_agentd:
   #
   # zabbix_agentd_enable (bool): Set to NO by default.  Set it to YES to
   #         enable zabbix_agentd.
   #
   
   . /etc/rc.subr
   
   name="zabbix_agentd"
   rcvar=zabbix_agentd_enable
   start_precmd="zabbix_precmd"
   required_files="/etc/zabbix_agentd.conf"
   
   # read configuration and set defaultsc
   load_rc_config "$name"
   : ${zabbix_agentd_enable="NO"}
   #: ${zabbix_agentd_pre:=/etc/${name}.pre.sh}
   
   zabbix_agentd_conf="/etc/zabbix_agentd.conf"
   
   if [ ! -z "$zabbix_agentd_conf" ] ; then
     zabbix_agentd_flags="${zabbix_agentd_flags} -c ${zabbix_agentd_conf}"
     required_files=${zabbix_agentd_conf}
   fi
   
   zabbix_precmd()
   {
     if [ ! -z "$zabbix_agentd_pre" ] ; then
       if [ -e $zabbix_agentd_pre ] ; then
         . $zabbix_agentd_pre
       fi
     fi
   }
   
   command="/usr/local/sbin/${name}"
   
   run_rc_command "$1"  run_rc_command "$1"
   

3. Make executable:

   chmod +x /etc/rc.d/zabbix_agentd
   

4. Start daemon:

   /etc/rc.d/zabbix_agentd start
   

5. Make config files persistent and survive reboots:

   cp /etc/rc.conf /conf/base/etc/
   cp /etc/rc.d/zabbix_agentd /conf/base/etc/rc.d/
   cp /etc/zabbix_agentd.* /conf/base/etc/

Exit vagrant

in the ssh sesion type: exit

vagrant destroy

 Zabbix configuration

Use the template from the zabbix wiki for the host monitoring

http://zabbix.org/wiki/File:Template_OS_FreeNAS.xml

---

1. https://docs.j7k6.org/freenas-zabbix-agent/
2. https://blag.nullteilerfrei.de/2016/11/26/zabbix-3-0-agent-on-freebsd/
3. https://www.haphazard.io/blog/install-nagios-nrpe-on-freenas/

Monitor Netgear ReadyNAS with Zabbix

Zabbix template for Netgear ReadyNAS

Monitoring devices with Zabbix is a simple thing, at least when you have a matching template for your device.

For Netgear ReadyNAS devices there exist many different templates, but most are outdated and no longer support the current 6.9.x firmware.

We integrated "all" ideas found in the other templates to provide a easy to install/use template for your device.

 Features

• LLD of disks, fans, temperature sensors, volumes, partitions, inetrfaces and processors
• Creation of graph and triggers
• No need to deploy MIB files to servers/agents, we work with direct OID's

The template and install instructions can be found here on Github link

Please open a issue in Github if you need more features or have enhancements you can provide.

Extended monitoring of SSL certificates with Zabbix

In my post about two years ago I showed how to monitor SSL certificates with Zabbix.
In the meantime the scripts/templates have been enhanced, with some small corrections/bugfixes.

One notable new feature is the possiblity to monitor SSL certificates which are delivered by SNI, which means you can have multiple SSL certificates available on the same IP/Port combination.
This is a critical feature, to better use the available IPv4 addresses.

The enhanced templates and scripts are now available via Github, which allows you to open issues if something is not working in your environment or contribute to new features as well.

I'm interested to hear about your use cases and feedbacks.

Monitoring HTTPS (and other SSL) certificates with Zabbix

In this year, we had a lot of security issues with SSL certificates, mostly in HTTPS, but also in other protocols using TLS/SSL.

The most serious concern is the poodle attack on SSLv3 connections.
The recommended setting is to disable SSLv3 in all your services.

The other issue is, that SHA1 is deprecated for HTTPS certificates, especially when the certificate expiration date is after 1. january 2017.
Google (and other web browser software developper) will gradually tell you, that
the certificates are not thrustworthy or even invalid.

You will find a lot of links in the web how to secury your server systems against poodle, and also on how to renew your certificates, so they don't use SHA1 anymore.

But how to make sure all your systems are correctly configured and the certificates in a valid range?
With Zabbix this is very simple, I have made a Zabbix template available, so you can integrate monitoring and alerting of SSL certificates in Zabbix.

You can download the Zabbix Template(s) and the ssl/tls check script from this link.

1. Place the ssltls.check script in your zabbix external script folder and make it executable.
2. Then import the template(s) and assign them to your server(s).

The HTTPS Template checks on Port 443 for the SSL things we described above. In addition it also checks the expirationdate and hash of your certificate.

The FTP template checks if TLS is activated on your FTP server and then checks for correct ssl working. It does no hash and 2017 expiration checks, since this is only relevant for webbrowsers.
The POP_IMAP_SMTP template checks if TLS is activae on POP,IMAP,SMTP,SMTP Submission and the POP3/IMAPS and SMTP services. It does no hash and 2017 expiration checks, since this is only relevant for webbrowsers.

The ssl check script is based on a version from Simon Kowallik, I did extend it to
also be able to check for the digest algorithm and detect the poodle vulnerability.

Link for SSLv3 disabling on various server products