Installing/running Debian stretsch inside kvm debian jessie

Installing or upgrading to a Debian stretch (Debian 9) edition should be a "no problem" case for most platforms.

Unfortunally running stretch in a KVM/QEMU environment based on Debian jessie isn't "just" working.

We did try to upgrade a existing Debian Jessie VM to stretch with the usual steps.
The upgrade itself works very fine and flawless as known from Debian systems.
But after rebooting the system, we did start getting errors in the guest, most of the time it did not even boot correctly.

In the host logfile we did see these messages in syslog:

kvm: zapping shadow pages for mmio generation wraparound
vcpu0 unhandled rdmsr: 0x34

In the log of the guest, we did see these stack traces:

Jun 19 15:40:18 sv54 kernel: [    0.000000] Linux version 4.9.0-3-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.30-2 (2017-06-12)
Jun 19 15:40:18 sv54 kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-3-amd64 root=UUID=fb8d9106-4421-4e85-af6f-8c0561bb2b25 ro quiet
Jun 19 15:40:18 sv54 kernel: [    0.000000] ------------[ cut here ]------------
Jun 19 15:40:18 sv54 kernel: [    0.000000] WARNING: CPU: 0 PID: 0 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/kernel/fpu/xstate.c:593 fpu__init_system_xstate+0x53b/0x981
Jun 19 15:40:18 sv54 kernel: [    0.000000] XSAVE consistency problem, dumping leaves
Jun 19 15:40:18 sv54 kernel: [    0.000000] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.9.0-3-amd64 #1 Debian 4.9.30-2
Jun 19 15:40:18 sv54 kernel: [    0.000000]  0000000000000000 ffffffff87f28634 ffffffff88803e08 0000000000000000
Jun 19 15:40:18 sv54 kernel: [    0.000000]  ffffffff87c76eae 000000000000000a ffffffff88803e60 0000000000000340
Jun 19 15:40:18 sv54 kernel: [    0.000000]  ffffffff88803e90 ffffffff88803e9c 0000000000000100 ffffffff87c76f2f
Jun 19 15:40:18 sv54 kernel: [    0.000000] Call Trace:
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87f28634>] ? dump_stack+0x5c/0x78
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87c76eae>] ? __warn+0xbe/0xe0
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87c76f2f>] ? warn_slowpath_fmt+0x5f/0x80
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87c72754>] ? xfeature_size+0x5a/0x78
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88947243>] ? fpu__init_system_xstate+0x53b/0x981
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87f586e6>] ? msr_clear_bit+0x36/0xa0
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff889468dc>] ? fpu__init_system+0x208/0x30b
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88942fea>] ? setup_arch+0xb8/0xcc6
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87d7a24e>] ? printk+0x57/0x73
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88938120>] ? early_idt_handler_array+0x120/0x120
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88938bbf>] ? start_kernel+0xab/0x463
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88938120>] ? early_idt_handler_array+0x120/0x120
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88938408>] ? x86_64_start_kernel+0x14c/0x170
Jun 19 15:40:18 sv54 kernel: [    0.000000] ---[ end trace 44d08096f31f4f03 ]---

and

Jun 19 15:40:18 sv54 kernel: [    0.387100] WARNING: CPU: 2 PID: 1 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/include/asm/fpu/internal.h:368 fpu__clear+0x179/0x1b0
Jun 19 15:40:18 sv54 kernel: [    0.387101] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.387103] CPU: 2 PID: 1 Comm: swapper/0 Tainted: G        W       4.9.0-3-amd64 #1 Debian 4.9.30-2
Jun 19 15:40:18 sv54 kernel: [    0.387103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Jun 19 15:40:18 sv54 kernel: [    0.387104]  0000000000000000 ffffffff87f28634 0000000000000000 0000000000000000
Jun 19 15:40:18 sv54 kernel: [    0.387105]  ffffffff87c76eae ffff9ec076158040 ffff9ec076158b40 ffff9ec072329400
Jun 19 15:40:18 sv54 kernel: [    0.387106]  ffffffff88869da0 ffff9ec072354400 ffff9ec076158758 ffffffff87c30fc9
Jun 19 15:40:18 sv54 kernel: [    0.387107] Call Trace:
Jun 19 15:40:18 sv54 kernel: [    0.387110]  [<ffffffff87f28634>] ? dump_stack+0x5c/0x78
Jun 19 15:40:18 sv54 kernel: [    0.387112]  [<ffffffff87c76eae>] ? __warn+0xbe/0xe0
Jun 19 15:40:18 sv54 kernel: [    0.387113]  [<ffffffff87c30fc9>] ? fpu__clear+0x179/0x1b0
Jun 19 15:40:18 sv54 kernel: [    0.387114]  [<ffffffff87e0914c>] ? flush_old_exec+0x5bc/0x6b0
Jun 19 15:40:18 sv54 kernel: [    0.387116]  [<ffffffff87e5df52>] ? load_elf_binary+0x3c2/0x1600
Jun 19 15:40:18 sv54 kernel: [    0.387117]  [<ffffffff87e087f0>] ? search_binary_handler+0xa0/0x1c0
Jun 19 15:40:18 sv54 kernel: [    0.387118]  [<ffffffff87e5d734>] ? load_script+0x204/0x230
Jun 19 15:40:18 sv54 kernel: [    0.387118]  [<ffffffff87dfe0da>] ? __check_object_size+0xfa/0x1d8
Jun 19 15:40:18 sv54 kernel: [    0.387119]  [<ffffffff87e093c8>] ? copy_strings.isra.25+0x188/0x450
Jun 19 15:40:18 sv54 kernel: [    0.387120]  [<ffffffff87e087f0>] ? search_binary_handler+0xa0/0x1c0
Jun 19 15:40:18 sv54 kernel: [    0.387121]  [<ffffffff87e09f0a>] ? do_execveat_common.isra.37+0x5aa/0x790
Jun 19 15:40:18 sv54 kernel: [    0.387123]  [<ffffffff881f8d20>] ? rest_init+0x80/0x80
Jun 19 15:40:18 sv54 kernel: [    0.387124]  [<ffffffff87e0a118>] ? do_execve+0x28/0x30
Jun 19 15:40:18 sv54 kernel: [    0.387125]  [<ffffffff881f8d70>] ? kernel_init+0x50/0x100
Jun 19 15:40:18 sv54 kernel: [    0.387126]  [<ffffffff882064f5>] ? ret_from_fork+0x25/0x30
Jun 19 15:40:18 sv54 kernel: [    0.387127] ---[ end trace 44d08096f31f4f07 ]---
Jun 19 15:40:18 sv54 kernel: [    0.387363] ------------[ cut here ]------------
Jun 19 15:40:18 sv54 kernel: [    0.387365] WARNING: CPU: 2 PID: 1 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/include/asm/fpu/internal.h:353 fpu__copy+0x140/0x190
Jun 19 15:40:18 sv54 kernel: [    0.387366] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.387367] CPU: 2 PID: 1 Comm: init Tainted: G        W       4.9.0-3-amd64 #1 Debian 4.9.30-2
Jun 19 15:40:18 sv54 kernel: [    0.387367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Jun 19 15:40:18 sv54 kernel: [    0.387368]  0000000000000000 ffffffff87f28634 0000000000000000 0000000000000000
Jun 19 15:40:18 sv54 kernel: [    0.387369]  ffffffff87c76eae ffff9ec0723e7ec0 ffff9ec076158b40 ffff9ec0723e7e80
Jun 19 15:40:18 sv54 kernel: [    0.387370]  0000000000000000 0000000000000000 00007f7cfecd79d0 ffffffff87c30a90
Jun 19 15:40:18 sv54 kernel: [    0.387371] Call Trace:
Jun 19 15:40:18 sv54 kernel: [    0.387373]  [<ffffffff87f28634>] ? dump_stack+0x5c/0x78
Jun 19 15:40:18 sv54 kernel: [    0.387374]  [<ffffffff87c76eae>] ? __warn+0xbe/0xe0
Jun 19 15:40:18 sv54 kernel: [    0.387375]  [<ffffffff87c30a90>] ? fpu__copy+0x140/0x190
Jun 19 15:40:18 sv54 kernel: [    0.387376]  [<ffffffff87c74490>] ? copy_process.part.33+0x1a0/0x1c00
Jun 19 15:40:18 sv54 kernel: [    0.387377]  [<ffffffff87dfe0da>] ? __check_object_size+0xfa/0x1d8
Jun 19 15:40:18 sv54 kernel: [    0.387379]  [<ffffffff87f56988>] ? strncpy_from_user+0x48/0x160
Jun 19 15:40:18 sv54 kernel: [    0.387379]  [<ffffffff87e0744d>] ? cp_new_stat+0x14d/0x180
Jun 19 15:40:18 sv54 kernel: [    0.387381]  [<ffffffff87c760d3>] ? _do_fork+0xe3/0x3f0
Jun 19 15:40:18 sv54 kernel: [    0.387381]  [<ffffffff87e074b9>] ? SYSC_newstat+0x39/0x60
Jun 19 15:40:18 sv54 kernel: [    0.387383]  [<ffffffff87c03b1c>] ? do_syscall_64+0x7c/0xf0
Jun 19 15:40:18 sv54 kernel: [    0.387384]  [<ffffffff8820632f>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jun 19 15:40:18 sv54 kernel: [    0.387384] ---[ end trace 44d08096f31f4f08 ]---
Jun 19 15:40:18 sv54 kernel: [    0.387420] ------------[ cut here ]------------
Jun 19 15:40:18 sv54 kernel: [    0.387422] WARNING: CPU: 2 PID: 1 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/include/asm/fpu/internal.h:353 __switch_to+0x66a/0x6c0
Jun 19 15:40:18 sv54 kernel: [    0.387422] ------------[ cut here ]------------
Jun 19 15:40:18 sv54 kernel: [    0.387423] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.387425] WARNING: CPU: 3 PID: 0 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/include/asm/fpu/internal.h:368 __switch_to+0x415/0x6c0
Jun 19 15:40:18 sv54 kernel: [    0.387426] CPU: 2 PID: 1 Comm: init Tainted: G        W       4.9.0-3-amd64 #1 Debian 4.9.30-2
Jun 19 15:40:18 sv54 kernel: [    0.387427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Jun 19 15:40:18 sv54 kernel: [    0.387427] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.387428]  0000000000000000 ffffffff87f28634 0000000000000000 0000000000000000
Jun 19 15:40:18 sv54 kernel: [    0.387430]  ffffffff87c76eae ffff9ec0762401c0 ffff9ec076158040 ffff9ec076240cc0
Jun 19 15:40:18 sv54 kernel: [    0.387431]  ffff9ec0762401c0 0000000000000002 ffff9ec076158a80 ffffffff87c24aaa
Jun 19 15:40:18 sv54 kernel: [    0.387432] Call Trace:
Jun 19 15:40:18 sv54 kernel: [    0.387432] ---[ end trace 44d08096f31f4f09 ]---

After some research we did find out, that there is a bug in the QEMU/KVM packages as provided by jessie stable repositories.

To solve the problem, you have to install more recent QEMU/KVM packages, and the best way to get them for stretch is to add the jessie backports to your sources.list

deb http://ftp.debian.org/debian jessie-backports main

After the usual apt-get update just install the qemu-kvm and related packages in the 2.8xxx version, instead of the 2.1xxxx releases.

After a reboot of your physical server, you can now enjoy debian stretch in your guests

Purge and remove backup files in Bareos / Bacula

Bareos / Bacula are very powerfull backup solutions for enterprises

When using backup to disk, one of the more nasty maintenance tasks, is removing old backup files.
To help you with this, you can use this script to remove expired backup files, and also backup files which contain error-only backups.

It is based on the code from Kjetil and has been modified to also woth with bareos 15 and mysql databases.

Patches and enhancements are welcome on the github platform.

Using letsencrypt certificates with Collabnet Subversion Edge

With the letsencrypt CA you can create ssl certificates to protect your http servers with a valid certificate.

These certificates are valid 90 days and the can be auto renewed, so your services won't interrupt.

For normal apache / nginx installations there are plenty of documentations on how to install and use such certificates.

Unfortunately with subversion edge, it does not just work out of the box, but it uses a apache http server for http/https communication.

So what's the problem?

Let's look at a standard subversion edge installation under debian:

• The certificates are stored in /opt/csvn/data/conf in the files server.crt and server.key
• The configuration for the ssl certificates are in the csvn_main_httpd.conf file, which is generated at each restart
• In the csvn_main_httpd.conf file we only have the two server.crt and server.key files referenced, but not the required intermediate certificate chain

The first thing to do is generate a ssl certificate for your server.
The http server root points to /var/www/html, so you can use that folder to generate your certificates

certbot certonly --webroot -w /var/www/html -d svn.example.com

This generates the certificates in /etc/letsencrypt/live/svn.example.com

To use them, you can now delete the two server.crt and server.key files in /opt/csvn/data/conf and create symbolic links to these files

server.crt -> /etc/letsencrypt/live/svn.example.com/cert.pem

server.key -> /etc/letsencrypt/live/svn.example.com/privkey.pem

But what about the SSLCertificateChainFile /etc/letsencrypt/live/svn.example.com/fullchain.pem file?
You can't add it to csvn_main_httpd.conf since this file is regenegated on each restart.
But fortunally you can add it into the https special config, where the chiper config is stored.
So just add it into the ssl_httpd.conf file

Now we have everything together and could start the https svn server.
Unfortunately this will fail with a error message like:

ERROR errors.GrailsExceptionResolver  -

 FileNotFoundException occurred when processing request: 

[POST] /csvn/
/opt/csvn/data/conf/server.crt (Permission denied). Stacktrace follows:
java.io.FileNotFoundException: /opt/csvn/data/conf/server.crt (Permission denied)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileWriter.<init>(FileWriter.java:107)

Ok, so you are running subversion as non-root as recommended. (If not, then rethink what you are doing and reinstall subversion as a standard user)

To allow the subversion user to access the ssl certificates of letsencrypt, the simplest way is to add the subversion user to the ssl-cert group, and give the group rx access to the certificates in /etc/letsencrypt/live and /etc/letsencrypt/archive

Once this is done, you should be able to start the svn server with active letsenrcypt ssl certificates.
What remains to be done is the periodic renewal of the certificates, which is widely described in the Internet.
What you have to do, is to restart the https service when a new certificate is generated. Usually you can do this in the renew script.

Java API for nextCloud/ownCloud

Java API for nextCloud

Currently the nextCloud and ownCloud solutions have a very big drive in the market. One of the main reason is, that you have control over your data.

When you look at recent events and news, then we can confirm this.
In our company we have been using ownCloud/nextCloud since version 5.x and have a long positive history with the solution.
We also provide managed nextCloud solutions, for sharing data with your customers for example, as backup back end and many other use cases.

The use case

To integrate nextCloud in your business processes, you sometimes need to automate things a bit more than what is included out of the box.
If possible we do this with shell scripts, but for complexer work flows, this isn't enough.
In those cases we use the full power of server side java applications.
Unfortunately the API of nextCloud is not fully REST/Webdav, it has some parts (Mainly file sharing and provisioning) which work with a XML style interface.

The java integration

To be able to use these API also from java applications, we have created a API library which exposes the important parts for simple usage in java applications.
To give back something to the open source community, we have decided to publish the library under the GPL license, so it can be used by other applications.
You can find the library source on github, and feedback and additions to the api are appreciated.

Happy coding

Extended monitoring of SSL certificates with Zabbix

In my post about two years ago I showed how to monitor SSL certificates with Zabbix.
In the meantime the scripts/templates have been enhanced, with some small corrections/bugfixes.

One notable new feature is the possiblity to monitor SSL certificates which are delivered by SNI, which means you can have multiple SSL certificates available on the same IP/Port combination.
This is a critical feature, to better use the available IPv4 addresses.

The enhanced templates and scripts are now available via Github, which allows you to open issues if something is not working in your environment or contribute to new features as well.

I'm interested to hear about your use cases and feedbacks.

Install Symantec Endpoint Protection on Debian Jessie

Syamntec endpoint protection is not only shipping for Windows systems, but also for OS-X and Linux systems.
Installing it on a debian jessie server does require some manual steps to have all features enabled.

The first step is to create a package in your SEP installation, simplest with a web downloadlink which you can later use to fetch the package via wget.

1. Install Java 8
2. Download JCE from http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
3. Install the cryptografic files into "/usr/lib/jvm/java-8-oracle/jre/lib/security"
4. Ony 64-bit systems enable i386 packages and install glibc in 32bit edition
   dpkg --add-architecture i386
   apt-get update
   apt-get install libc6:i386
5. On 64-bit systems install the linux-headers to allow compilation of the realtime scan kernel modules
   apt-get install linux-headers-amd64
6. Download your SEP package from your server wget http://<your-sep-server>/EmailInstallPackages/xxxxxxxxxxxx/sep/SymantecEndpointProtection.zip
7. Unzip zip package
8. bash ./install.sh -i  

Installing Java 8 on Debian Jessie

Installation Oracle Java 8 on Debian Jessie

Debian 8 alias Jessie ships with OpenJDK 7 which is fine in many cases. But sometimes you need a more recent version.
In that case you can use the ubuntu ppa archives as install source.

Just type these commands in the console of your Debian system and it should install just fine, also providing automatic security upgrades as they become available.

echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" > /etc/apt/sources.list.d/webupd8team-java.list
echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" >> /etc/apt/sources.list.d/webupd8team-java.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886
apt-get update
apt-get install oracle-java8-installer
java -version