Proxmox 4.1 kernel panic: downgrade DRBD resources from DRBD 9 to 8.4

I've upgraded my Proxmox environment to the latest 4.1, featuring the brand new (and still in preview state) DRBD 9.

Proxmox documentation is not so clear about it's preview state and, also, users have no choice about the DRBD version to use; Proxmox 4.1 only has DRBD 9, no choice.

That said, once completed upgrade I found that DRBD9 was the cause of host node crashes (I mean the whole host crashed with a kernel panic, together with all hosted VMs).
The issue has been reported by me and another user here:
https://forum.proxmox.com/threads/kernel-panic-with-proxmox-4-1-13-drbd-...
http://www.gossamer-threads.com/lists/drbd/users/27685

It seems that the only way-out is to downgrade DRBD resources to version 8.4.
I've done it and I had no issues after that (my environment is running flawlessy since a week ago).

NOTE: since Proxmox doesn't support 8.4 anymore, you have to build it from sources and replace the original one installed from Proxmox repositories.

System configuration: two nodes Proxmox 4.1, LVM on DRBD resource r0, DRBD 9.0, kernel pve-kernel-4.2.8-1-pve.

Downgrade running kernel from DRBD 9.x to DRBD 8.4

This part describes the procedure to downgrade from a running DRBD 9.x to DRBD 8.4 on the same kernel version.
If you're going to upgrade your kernel (already running a DRBD 8.4 module) to a newer version please follow the chapter below.

We'll have to downgrade one node at a time, let's start from NodeA.

Initialization

Define a variable containing the kernel version you're compiling for. If this is the first downgrade (still running the bundled DRBD 9.0), KVER must be set to current kernel version:

export KVER=`uname -r`

Install build tools, DRBD sources and kernel headers

apt-get install build-essential flex
apt-get install pve-headers-$KVER
cd /usr/src
wget http://oss.linbit.com/drbd/8.4/drbd-8.4.7-1.tar.gz
wget http://oss.linbit.com/drbd/drbd-utils-8.9.6.tar.gz
tar zxvf drbd-8.4.7-1.tar.gz
tar zxvf drbd-utils-8.9.6.tar.gz

Build DRBD module and utils

NOTE: configure scripts will automatically use the $KVER variable defined above.

cd /usr/src/drbd-8.4.7-1
make clean
cd drbd
make

you can now (optionally) strip the binaries and make them smaller

strip --strip-unneeded drbd.ko

now build the userland utils

cd /usr/src/drbd-utils-8.9.6
./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc --without-83support --with-84support --without-manual --with-distro=debian
make clean
make

and (optionally) strip the binaries to make them smaller

strip --strip-unneeded drbdadm-84
strip --strip-unneeded drbdsetup-84
strip --strip-unneeded drbdmeta

Move all the VMs to NodeB and shutdown resource

Move all of the VMs using the resource we're going to downgrade to the other node, then demote and deactivate resource.

drbdadm secondary r0
drbdadm down r0

Replace the bundled DRBD 9.x module with our own 8.4 version

rmmod drbd_transport_tcp
rmmod drbd
cd /lib/modules/$KVER/kernel/drivers/block/drbd
mv drbd.ko drbd.ko-9.0.0
mv drbd_transport_tcp.ko drbd_transport_tcp.ko-9.0.0
cp /usr/src/drbd-8.4.7-1/drbd/drbd.ko .
modprobe drbd

Replace DRBD 9.x tools with version 8.4

cd /usr/sbin
mv drbdadm drbdadm-9.0.0
mv drbdmeta drbdmeta-9.0.0
mv drbd-overview drbd-overview-9.0.0
mv drbdsetup drbdsetup-9.0.0
cp /usr/src/drbd-utils-8.9.6/user/v84/drbdadm-84 .
ln -s drbdadm-84 drbdadm
cp /usr/src/drbd-utils-8.9.6/user/v84/drbdsetup-84 .
ln -s drbdsetup-84 drbdsetup
cp /usr/src/drbd-utils-8.9.6/user/v9/drbdmeta .
cp /usr/src/drbd-utils-8.9.6/scripts/drbd-overview.pl drbd-overview

NOTE: in my setup I've also had to edit resource configuration file and comment out lines with "node-id: xxx;" parameters

DRBD utils warning

DRBD utils now emit a warning like this:

DRBD module version: 8.4.7
   userland version: 8.9.6
please don't mix different DRBD series.

That's because drbd-utils-8.9.6 looks forward to 9.x series so they warn if used with previous ones.
It is safe to ignore it but, if you don't feel comfortable, you can suppress the message by defining an environment variable:

export DRBD_DONT_WARN_ON_VERSION_MISMATCH=1

The relevant source file is in drbdadm_main.c, line 3597:

if (!getenv("DRBD_DONT_WARN_ON_VERSION_MISMATCH"))
    warn_on_version_mismatch();

Then each time you run drbdadm, the version mismatch warning won't be shown anymore.
You can add it to /etc/environment to have it defined for each new shell.

Downgrade resource metadata

sudo drbdadm create-md r0

It will ask you a confirmation before downgrading metadata from v90 to v84 format.

Restart DRBD service

/etc/init.d/drbd restart

Check resource status

The resource should be up and running (and resyncing), otherwise bring it up.

drbdsetup status --verbose
drbdadm up r0

Downgrade the other node

You can follow the same guide to downgrade the other host or, if both nodes share the same kernel and hardware, you could simply copy binary compiled files over it and install them, no need to install build tools.

Kernel updates

Each new kernel you'll install in the future needs a downgraded DRBD 8.4 module to be built for it.
The procedure is almost identical with some small changes.

Install new kernel & headers

Install the new kernel and its headers (i.e. version 4.99.99)

sudo apt-get install pve-kernel-4.99.99-pve pve-headers-4.99.99-pve

Define a variable containing the target kernel version.
Since we're compiling for a kernel not currently running, KVER must be set manually:

export KVER=4.99.99-pve

If there's a newer DRBD version available, you could update its sources too:

cd /usr/src
wget http://oss.linbit.com/drbd/8.4/drbd-X.X.X.tar.gz
wget http://oss.linbit.com/drbd/drbd-utils-Y.Y.Y.tar.gz
tar zxvf drbd-X.X.X-1.tar.gz
tar zxvf drbd-utils-Y.Y.Y.tar.gz

Build DRBD module and utils

NOTE: configure scripts will automatically use the $KVER variable defined above, but making the module need to specify the KDIR parameter manually.

Build drbd module:

cd /usr/src/drbd-8.4.7-1
make clean
cd drbd
make KDIR=/usr/src/linux-headers-$KVER

you can now (optionally) strip the binaries to make them smaller

strip --strip-unneeded drbd.ko

NOTE: building drbd-8.4.7-1 on some 4.4 kernels could fail with the error: drbd_bitmap.c:1033:60: error: ‘__GFP_WAIT’ undeclared (first use in this function)
You need to apply a small patch (see here: http://www.engisoftcloud.com/2016/04/07/instalacion-drbd-en-amazon-linux...):
UPDATE: kernel 4.4.13-2-pve compiled successfully without this patch

find /usr/src/drbd-8.4.7-1 -type f -exec sed -i  s/__GFP_WAIT/__GFP_RECLAIM/g {} \;

Then build drbd-tools:

NOTE: this step is optional if DRBD tools sources were not updated.

cd /usr/src/drbd-utils-8.9.6
./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc --without-83support --with-84support --without-manual --with-distro=debian
make clean
make

and (optionally) strip the binaries to make them smaller

strip --strip-unneeded drbdadm-84
strip --strip-unneeded drbdsetup-84
strip --strip-unneeded drbdmeta

Replace the bundled DRBD 9.x module with our own 8.4 version

cd /lib/modules/$KVER/kernel/drivers/block/drbd
mv drbd.ko drbd.ko-9.0.0
mv drbd_transport_tcp.ko drbd_transport_tcp.ko-9.0.0
cp /usr/src/drbd-8.4.7-1/drbd/drbd.ko .

Replace DRBD 9.x tools with version 8.4

cd /usr/sbin
mv drbdadm drbdadm-9.0.0
mv drbdmeta drbdmeta-9.0.0
mv drbd-overview drbd-overview-9.0.0
mv drbdsetup drbdsetup-9.0.0
cp /usr/src/drbd-utils-8.9.6/user/v84/drbdadm-84 .
ln -s drbdadm-84 drbdadm
cp /usr/src/drbd-utils-8.9.6/user/v84/drbdsetup-84 .
ln -s drbdsetup-84 drbdsetup
cp /usr/src/drbd-utils-8.9.6/user/v9/drbdmeta .
cp /usr/src/drbd-utils-8.9.6/scripts/drbd-overview.pl drbd-overview

Reboot the node and check resource status

Move all the virtual machines to the other node(s) and reboot this node.
When it came back up check if DRBD resources are up and running on the newer kernel.

uname -r
--> must print new kernel version!
 
sudo drbdsetup status --verbose

Downgrade the other node

Follow this same guide to upgrade the other host kernel.
If both nodes have the same kernel and hardware, you could simply copy binary compiled files over it and install them, no need to install build tools.
Beware: copy the *-84 binaries then recreate the links as above.

Hope these instructions will help other users experiencing my same issue...

03 Aug 2016
- just updated to kernel 4.4.13-2-pve without issues, using this same instructions

20 May 2016
- added instructions to disable drbd-utils warning message

12 Apr 2016
- updated procedure (thanks to Jean-Laurent Ivars suggestions)
- added the kernel update procedure

02 May 2016
- added workarounds to build modules for kernel 4.4

05 May 2016
- added optional binary strip commands

06 May 2016
- removed sudo usage (not installed on proxmox by default)
- utils binary update is not optional but mandatory on kernel update