Monitor letsencrypt certificates with Zabbix

Monitor letsenrcypt certificates with Zabbix

Letsenrcypt is a great free system to automatically provide ssl/tls certificates for your website(s)
Zabbix is a great free system to monitor your IT infrastructure

One of the main features of Zabbix is the ability to extend it with any kind of monitoring scripts.
So when you do use letsencrypt certificates, you can also monitor them from Zabbix.

There exists a template you can add to the Zabbix server, and the required config and script files you place on the agents.

These can be found here.

It provides these features:
- Autodiscovery of all active letsencrypt certificates
- Monitor the lifetime of the certificates
- Trigger when the certificates are about to expire (Which means your auro renew does not work correctly)
- Monitor the certbot version
- Triggers when certbor version is below 0.28

The certbot version is important, since all certbot versions below 0.28 don't support the ACME-2.0 standard which is required by letsencrypt since the 13-february 2019.

Whith this you can relax about your ssl certificates, you won't have customers calling you in the morning, that their website(s) have an invalid/expired certificate.

You can even enhance the ssl monitoring with more Zabbix templates from here.
These allow you to monitor also ftps/imaps/smtps/pop3s and other TLS/SSL secured connections.

We use these in our own environment, to make sure everything runs smoothly and that we are warned of potential problems before the customers notice them.

Using letsencrypt certificates with Collabnet Subversion Edge

With the letsencrypt CA you can create ssl certificates to protect your http servers with a valid certificate.

These certificates are valid 90 days and the can be auto renewed, so your services won't interrupt.

For normal apache / nginx installations there are plenty of documentations on how to install and use such certificates.

Unfortunately with subversion edge, it does not just work out of the box, but it uses a apache http server for http/https communication.

So what's the problem?

Let's look at a standard subversion edge installation under debian:

• The certificates are stored in /opt/csvn/data/conf in the files server.crt and server.key
• The configuration for the ssl certificates are in the csvn_main_httpd.conf file, which is generated at each restart
• In the csvn_main_httpd.conf file we only have the two server.crt and server.key files referenced, but not the required intermediate certificate chain

The first thing to do is generate a ssl certificate for your server.
The http server root points to /var/www/html, so you can use that folder to generate your certificates

certbot certonly --webroot -w /var/www/html -d svn.example.com

This generates the certificates in /etc/letsencrypt/live/svn.example.com

To use them, you can now delete the two server.crt and server.key files in /opt/csvn/data/conf and create symbolic links to these files

server.crt -> /etc/letsencrypt/live/svn.example.com/cert.pem

server.key -> /etc/letsencrypt/live/svn.example.com/privkey.pem

But what about the SSLCertificateChainFile /etc/letsencrypt/live/svn.example.com/fullchain.pem file?
You can't add it to csvn_main_httpd.conf since this file is regenegated on each restart.
But fortunally you can add it into the https special config, where the chiper config is stored.
So just add it into the ssl_httpd.conf file

Now we have everything together and could start the https svn server.
Unfortunately this will fail with a error message like:

ERROR errors.GrailsExceptionResolver  -

 FileNotFoundException occurred when processing request: 

[POST] /csvn/
/opt/csvn/data/conf/server.crt (Permission denied). Stacktrace follows:
java.io.FileNotFoundException: /opt/csvn/data/conf/server.crt (Permission denied)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileWriter.<init>(FileWriter.java:107)

Ok, so you are running subversion as non-root as recommended. (If not, then rethink what you are doing and reinstall subversion as a standard user)

To allow the subversion user to access the ssl certificates of letsencrypt, the simplest way is to add the subversion user to the ssl-cert group, and give the group rx access to the certificates in /etc/letsencrypt/live and /etc/letsencrypt/archive

Once this is done, you should be able to start the svn server with active letsenrcypt ssl certificates.
What remains to be done is the periodic renewal of the certificates, which is widely described in the Internet.
What you have to do, is to restart the https service when a new certificate is generated. Usually you can do this in the renew script.

Extended monitoring of SSL certificates with Zabbix

In my post about two years ago I showed how to monitor SSL certificates with Zabbix.
In the meantime the scripts/templates have been enhanced, with some small corrections/bugfixes.

One notable new feature is the possiblity to monitor SSL certificates which are delivered by SNI, which means you can have multiple SSL certificates available on the same IP/Port combination.
This is a critical feature, to better use the available IPv4 addresses.

The enhanced templates and scripts are now available via Github, which allows you to open issues if something is not working in your environment or contribute to new features as well.

I'm interested to hear about your use cases and feedbacks.

Monitoring HTTPS (and other SSL) certificates with Zabbix

In this year, we had a lot of security issues with SSL certificates, mostly in HTTPS, but also in other protocols using TLS/SSL.

The most serious concern is the poodle attack on SSLv3 connections.
The recommended setting is to disable SSLv3 in all your services.

The other issue is, that SHA1 is deprecated for HTTPS certificates, especially when the certificate expiration date is after 1. january 2017.
Google (and other web browser software developper) will gradually tell you, that
the certificates are not thrustworthy or even invalid.

You will find a lot of links in the web how to secury your server systems against poodle, and also on how to renew your certificates, so they don't use SHA1 anymore.

But how to make sure all your systems are correctly configured and the certificates in a valid range?
With Zabbix this is very simple, I have made a Zabbix template available, so you can integrate monitoring and alerting of SSL certificates in Zabbix.

You can download the Zabbix Template(s) and the ssl/tls check script from this link.

1. Place the ssltls.check script in your zabbix external script folder and make it executable.
2. Then import the template(s) and assign them to your server(s).

The HTTPS Template checks on Port 443 for the SSL things we described above. In addition it also checks the expirationdate and hash of your certificate.

The FTP template checks if TLS is activated on your FTP server and then checks for correct ssl working. It does no hash and 2017 expiration checks, since this is only relevant for webbrowsers.
The POP_IMAP_SMTP template checks if TLS is activae on POP,IMAP,SMTP,SMTP Submission and the POP3/IMAPS and SMTP services. It does no hash and 2017 expiration checks, since this is only relevant for webbrowsers.

The ssl check script is based on a version from Simon Kowallik, I did extend it to
also be able to check for the digest algorithm and detect the poodle vulnerability.

Link for SSLv3 disabling on various server products

SMTP SSL in Exchange 2007/2010

In exchange 2007 and higher, the default connector are configured to accept passwords only when using a secured connection (Like SSL or TLS).
A good idea
, but...

The SSL option is only available for POP and IMAP, on SMTP you miss this option, here you can only configure a TLS secured connection.

Not a problem.... do you think?
When you still have users accessing your mail server via POP3/SMTP with MS Outlook 2003 (Not uncommon these days) then they don't have the option to use SMTP with TLS, only SMTP with SSL is available...

Other products from the same company not offering SMTP with TLS are: Outlook Express, Windows Mail....

So you will either have to upgrade all clients to a new MS Office version, or downgrade security on your exchange server and allow the SMTP connector to also accept password over unsafe connections.

Or, probably the best thing to do: Just switch to a real mail user agent like thunderbird.....