Reactivate Zabbix agent in TrueNAS after update

When you have installed the Zabbix agent as mentioned in my previous post, then you will notice that Zabbix gets deactivated wqhen you install TrueNAS updates.

To get the agent running again, you only need to reactivate it in rc.conf

1. Enable daemon
   echo 'zabbix_agentd_enable="YES"' >> /etc/rc.conf

2. Start daemon:
   /etc/rc.d/zabbix_agentd start
3. Make config files persistent and survive reboots:
   cp /etc/rc.conf /conf/base/etc/ 
   cp /etc/rc.d/zabbix_agentd /conf/base/etc/rc.d/
mkdir /conf/base/etc/zabbix_agentd
cp /etc/zabbix_agentd.* /conf/base/etc/

Of course it's a good moment to install the latest zabbix agent too.

For this take the latest version matching your server version and copy the /bin and /sbin to /usr/local and restart the agent.

If someone has an idea, on how to make the "Enable daemon" step survive a minor TrueNAS upgrade, I would be glad for some hints ;)

ERROR: (gcloud.iam.service-accounts.keys.create) FAILED_PRECONDITION: Precondition check failed.

When you try to create a key for a service account in google cloud via the gcloud iam service-accounts keys create command, it can occur that you receive this error ERROR: (gcloud.iam.service-accounts.keys.create) FAILED_PRECONDITION: Precondition check failed.

Unfortunally searching for this error does not show up much, and adding --verbosity=debug does not turn up more information.

The reason for this error is, that a single service account can have a maximum of (currently) 10 active keys.

If you try to create one more, then you will receive the above error message.

The only way to work arround the problem is to remove older keys, which are (hopefully) no longer used.

You can do this either via the gui or the gcloud iam service-accounts keys delete command

 

Found the solution thanks to https://gist.github.com/maelvls/c23558b717422c4c648f4258a7f2fb1 since in the google documentation there is no reference to a limit on the number of keys per service account

Install Zabbix agent on TrueNAS

 In my previous article from 2018 I did mention how to install teh Zabbix agent on FreeNAS.

Since then two years are passed and FreeNAS has been replaced by TrueNAS provides the same functionality in a new version with a new name.

The steps for installation are remained the same, but you won't need to build the agent package yourself, you can use the packages provided from Zabbix for FreeBSD.

So lets see what you have to do now:

1. Download the binary package from the Zabbix webpage at https://www.zabbix.com/de/download_agents?version=4.0+LTS&release=4.0.25&os=FreeBSD&os_version=11.2&hardware=amd64&encryption=GnuTLS&packaging=Archive

It's no problem that FreeBSD 12.x is not listed, the 11.2 packages still work

2. Unpack the /bin and /sbin folders into /usr/local of your TrueNAS server

3. Unpack the /conf/* into /etc

4. Create a use and group zabbix so your agent wont run as root, you need to do this via GUI, otherwiese the account will be gone after reboot

5. Edit your /etc/zabbit_agentd:conf to match your needs

6. Enable daemon

echo 'zabbix_agentd_enable="YES"' >> /etc/rc.conf  

 

Create /etc/rc.d/zabbix_agentd:

#!/bin/sh

# PROVIDE: zabbix_agentd
# REQUIRE: DAEMON
# KEYWORD: shutdown
#
# Add the following lines to /etc/rc.conf.local or /etc/rc.conf to
# enable zabbix_agentd:
#
# zabbix_agentd_enable (bool): Set to NO by default.  Set it to YES to
#         enable zabbix_agentd.
#

. /etc/rc.subr

name="zabbix_agentd"
rcvar=zabbix_agentd_enable
start_precmd="zabbix_precmd"
required_files="/etc/zabbix_agentd.conf"

# read configuration and set defaultsc
load_rc_config "$name"
: ${zabbix_agentd_enable="NO"}
#: ${zabbix_agentd_pre:=/etc/${name}.pre.sh}

zabbix_agentd_conf="/etc/zabbix_agentd.conf"

if [ ! -z "$zabbix_agentd_conf" ] ; then
  zabbix_agentd_flags="${zabbix_agentd_flags} -c ${zabbix_agentd_conf}"
  required_files=${zabbix_agentd_conf}
fi

zabbix_precmd()
{
  if [ ! -z "$zabbix_agentd_pre" ] ; then
    if [ -e $zabbix_agentd_pre ] ; then
      . $zabbix_agentd_pre
    fi
  fi
}

command="/usr/local/sbin/${name}"

run_rc_command "$1"  run_rc_command "$1"

 Make executable:

1. chmod +x /etc/rc.d/zabbix_agentd
   

2. Start daemon:

   /etc/rc.d/zabbix_agentd start
   

3. Make config files persistent and survive reboots:

   cp /etc/rc.conf /conf/base/etc/
   cp /etc/rc.d/zabbix_agentd /conf/base/etc/rc.d/
   mkdir /conf/etc/zabbix_agentd
   cp /etc/zabbix_agentd.* /conf/base/etc/
   cp /etc/zabbix_agentd.* /conf/base/etc/ 

   Zabbix configuration

   Use the template from the zabbix wiki for the host monitoring http://zabbix.org/wiki/File:Template_OS_FreeNAS.xml

Use perdition as ssl offload proxy for imap / pop and managesieve

 In modern setups you often have an ingress controller, which does the ssl termination of the connections and then routes the traffic to the correct backend(s)

For kubernetes and http(s), often nginx is used for that task.

When you wish to do the same for imap and/or pop, then it also possible to use nginx for this.

https://docs.nginx.com/nginx/admin-guide/mail-proxy/mail-proxy/

But the main drawback is, that you need to implement some authentication and routing based on an http request.Also the managesieve protocol isn't supported.

If you don't wish to do the hassle with this, or you haven't the required infos to do it, then perdition can handle this be used as a full imap/pop/managesieve proxy.

The basic setup is quite simple, but has a few things to be aware of, when using it with ssl and/or ipv6.

When you enable ssl, then you have to specify the certificate files.

Usually something like this:

ssl_ca_file /etc/postfix/ssl/ssl-root.ca
ssl_cert_file /etc/postfix/ssl/myserver.crt
ssl_key_file /etc/postfix/ssl/myserver.key

When you then start perdition, it will probably log some warning about not beeing able to read the DH parameters from the certificate file. 

could not read DH params from cert file

Modern OpenSSL configurations require Diffie-Hellman values to generate secure keys in the exchange.

If your certificate does not have these embedded in it, you can generate them yourself and add them to the certificate.

openssl dhparam -out dhparams.pem 4096

Then just append the content of the dhparams.pem file to your .crt file and perdition has the required DH values.

The second thing you might struggle with, when you start perdition on an IPv6 enable host, then perdition will only bind the IPv4 address and not to the IPv6 address.

Unfortunally the documentation is lacking in this area, as how to bind the IPv6 ports too.

The correct syntax is to use this in the perdition.imap4s etc. files:

bind_address 88.xx.xx.xx,"[2a01:xxx:xxx:xxx::xxx]"

Please note that you must specify both IPvç and IPv6 addresses, and also that the IPv6 "[::]" will not work.

You have to specify the IPv6 address and enclose it in "[...]", including the " characters.

Configuring perdition as managesieve proxy is also not very well documented.

Specifying the sieve capabilities is rather tricky, here a wroking example:

capability \
"\"IMPLEMENTATION\" \"Cyrus timsieved\"  "\
"\"SIEVE\" \"comparator-i;octet "\
"comparator-i;ascii-numeric "\
"fileinto "\
"reject "\
"vacation "\
"imapflags "\
"notify "\
"envelope "\
"relational "\
"regex "\
"subaddress "\
"copy\"  "\
"\"SASL\" \"PLAIN\""

 

The important things to note in this:

Use \" to delimit the capabilities, and use two spaces to delimit the capability lines .

Have a look at the base config file perdition.conf as a staring point

If you have clients using K9 mail (And probably other too), then you might remove all "AUTH=..." settings from the imap capability string.

https://lists.vergenet.net/pipermail/perdition-users/2011-August/002547.html

Please also see my upcomming post  on monitoring perdition with Zabbix

Automatically remove write protection from USB Disks in Backup Exec

Veritas BackupExec has a default, which flags disks which have been offline for more than 32 days as write protected.

BackupExec has expiration dates for each backup you have done. Once this date is reached, BackupExec does free up the disk space and deletes the old backups.

The idea behind this is, that these are disks which contain older backups, which should not automatically be purged from disk.Otherwise, when you connect the disk with the monthly backup of january on the server in june, for some restore, then BackupExec would remove these old backup sets.

That's usually not what you wish to have, this is why in the default this 32 day rule exists.

Of course you can tell BackupExec to either not set this write protection at all, or only for some longer time period. There is also a dangerous option, to allow BackupExec to delete all expired backup sets.

The default is usually fine, until you reinsert your january 2019 disk in january 2020. Then the disk will be write protected and you will have to remove the write protection first, or your backup jobs will fail.

Since this is a manual action you have to do on a regular basis, you can also automate it with some scripts.

With powershell und the windows task scheduler you can remove the write protection of the connected drives.

One important thing to note is, you must schedule the powershell script via task scheduler, if you define it as an "Run before job" in BackupExec, then the job will not start since it sees no writable disk in then system, and also does not start the "pre job run script"

This is the powershell script, which remove the write protection on all online disks in BackupExec, store it in a location on C:\...., for example as "c:\Program Files\Veritas\Backup Exec\Scripts\TurnWriteProtectedOff.ps1"

<# This script does remove the virtaul write protection on any online disks
   This way USB disks which had not been online for a long time wil be overwritable again
   2020 a.schild@aarboard.ch
#>
Import-Module "c:\Program Files\Veritas\Backup Exec\Modules\BEMCLI\BEMCLI.Scripts.psm1"

$disks= Get-BEStorageDevice
foreach ($d in $disks) {
    if ($d.Servers.IsOnline -eq "true") {
        # $d | Format-Table Name
        $d | Set-BEDiskStorageDevice -VirtualWriteProtectionEnabled $false
    }
}

In your task scheduler you then schedule it to be run 5-10 minutes before the regular backup job.

As command specify your powershell.exe as "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"

In the script argugemt you pass:

 -File "C:\Program Files\Veritas\Backup Exec\Scripts\TurnWriteProtectedOff.ps1"

And define it to be run always, with no network resources and with highest priority.

Startup problems with php-fpm and ldap user backend on Debian

Startup problems with php-fpm and ldap user backend on Debian

Do you use php-fpm together with an ldap backend for user authentication, then you probably have seen this message in your boot logs:

[26-Sep-2019 14:58:34] ERROR: [pool XXXXXX] cannot get uid for user 'XXXXXXXX'

When using the php5.6-fpm / php7.x-fpm versions of the PHP fpm process can cause startup problems, when the users are provided via the nslcd service.

The reason is, that the ldap user backend is starting after the php fpm process(es)

Fortunately this is easy to solve, just edit your /etc/init.d/php7.3-fpm init script and add the user backend service(s) to the Required-Start section

It looks like this:
# Required-Start:    $remote_fs $network

So to add nslcd (and recommended nscd too) results in this line

# Required-Start:    $remote_fs $network nslcd nscd


To enable the new dependencies of your service starts, just disable the service and reenable it.

systemctl disable php7.3-fpm 
systemctl enable php7.3-fpm

After this, you can reboot your server and the fpm processes should startup just fine
 

Chromium 75.0.3770.xx does submit html forms twice

When you have customers which use chromium on Ubuntu 18.04 LTS, then you might see duplicate form posts in your applications.

This is a very nasty bug introduced somewhere between the 74.0.3729.169 and 75.0.3770.80 releases.
Here a few references to the problem, which seems to be still "work in progress"

• https://stackoverflow.com/questions/56488687/duplicate-requests-after-chrome-update-to-version-75-0-3770-80-official-build
   
• https://bugs.chromium.org/p/chromium/issues/detail?id=977882
   
• https://support.google.com/chrome/thread/8616084?hl=en

 This does not help the customers, since we need a solution now, and not in a few days/weeks.

On ubuntu the simplest is to roll back to the last working version 74.0.3729.169

How to do this:

Login in the console of your system(s) and do this:

cd /tmp
rm chromium-*

wget -q http://launchpadlibrarian.net/424897539/chromium-browser_74.0.3729.169-0ubuntu0.18.04.1_amd64.deb
wget -q http://launchpadlibrarian.net/424897541/chromium-codecs-ffmpeg-extra_74.0.3729.169-0ubuntu0.18.04.1_amd64.deb
wget -q http://launchpadlibrarian.net/424897535/chromium-browser-l10n_74.0.3729.169-0ubuntu0.18.04.1_all.deb


dpkg -i chromium-*.deb
rm chromium-*
 
apt-mark hold chromium-browser
apt-mark hold chromium-browser-l10n
apt-mark hold chromium-codecs-ffmpeg-extra


This installs the 74er version and prevents future upgrades to install bad versions.
Of course once a working 75 or 76 release is available, you have to unhold the packages