SSI Project
  Contributed Code
  Mailing Lists
  Bruce's Corner
  Project List
SSI Documentation
Printer-friendly version

NOTICE: This procedure has not been verified for the 1.1.0 release.


The drbd-HowTO describes the following:
I. Failover of root file system using DRBD
II. Failover of non-root file systems using DRBD

I. Failover of root file system using DRBD

These are the steps involved in getting drbd root failover on OpenSSI.
These steps are for a fresh install. Towards the ends of the section,
you can find steps to convert an existing openSSI cluster into
a drbd-enabled root-failover cluster.

1. Install base linux (Fedora Core 2). Make sure you partition your disk
   such that it has a an extra partition with atleast 128 Mb for the
   drbd index later on.( For e.g., in addition to /root,/boot and /swp there
   will an extra partition for drbd of atleast size 128 MB)
2. Install the latest version OpenSSI available from
   The current latest version is openssi-1.1.0. Make sure you 
   enable Root-Failover during installation.
3. Install openssi-enabled drbd.
   The tarball is available from 
   The tarball contains the drbd package (drbd.code) and sample files
   for linuxrc and SSIfailover needed in steps below. Needs to be built
   against the kernel-source currently (I am working on a rpm):
   Rpm for openssi-kernel source is avalaible at*.i686.rpm
   Find the version that corresponds to the openSSI release you are using.
4. Configure drbd.conf in /etc/drbd.conf. Include the correct hostname,
   disk partitions and nodenum. The "device=" parameter should
   be your root and meta-disk the partition that was left free during
   installation during step 1.
5. Unpack ramdisk and mount it loopback(for example on /mnt/ramdisk/).
   Copy over the following :
   a. drbd.o 
        cp $LIBDIR/drbd.o /mnt/ramdisk/lib/
   b. drbd utilities: drbdsetup, drbdadmn
        cp /sbin/drbdsetup /mnt/ramdisk/bin/
        cp /sbin/drbdadmn /mnt/ramdisk/bin/ 
   c. drbd.conf 
        cp /etc/drbd.conf /mnt/ramdisk/etc/ 
   d. Use mknod to create /mnt/ramdisk/dev/nbd/0 
        mkdir /mnt/ramdisk/dev/nbd
        cd /mnt/ramdisk/dev/nbd
        mknod 0 b 43 0
   e. Similarly, use mknod to create the device files for root and meta-disk
   f. Copy the mount command
        cp -a /bin/mount /mnt/ramdisk/bin/

6. Handcraft linuxrc in ramdisk (your particular linuxrc might look
slightly different):
   a. insmod lib/drbd.o
      Corresponding patch for linuxrc will look like:
            echo "Loading mii.o module"
            insmod /lib/mii.o
            echo "Loading 8390.o module"
            insmod /lib/8390.o
            echo "Loading e100.o module"
            insmod /lib/e100.o
            echo "Loading tulip.o module"
            insmod /lib/tulip.o
      +	    echo "Loading drbd.o module"
      +     insmod /lib/drbd.o

   b. Initialize drbd, make init node the primary node.
      and then mount /dev/nbd/0 on /sysroot
      Corresponding patch for steps b,c,d in linuxrc will look like:
         if [ $? -ne "0" ]
             echo Creating root device
        -    echo mkrootdev /dev/root | nash --quiet --force
        -    mount -o defaults --ro -t ext3 /dev/root /sysroot
        +    for i in `drbdadm -s /bin/drbdsetup sh-devices`
        +    do
        +               drbdadm -s /bin/drbdsetup up $i
        +    done
        +    drbdadm -s /bin/drbdsetup wait_connect all
        +    drbdadm -s /bin/drbdsetup primary all
        +    mount -o defaults,chard --ro -t ext3 /dev/nbd/0 /sysroot

             if [ $? -ne "0" ]
                echo "ERROR: Mounting root file system failed."
                echo "Unable to continue. Halting."
        -       halt -L -f
        -       exit 1  # NOT REACHED
        +#      Use for debugging and potentially forcing drbd mount primary
        +       bash
        +#      halt -L -f
        +#      exit 1  # NOT REACHED

7. Pack ramdisk and copy it into /boot. Make sure you retain 
   a copy of the old ramdisk so that you can boot up with the older
   ramdisk if needed. 
8. Change /etc/init.d/SSIfailOver to start drbd and enable failover for
   Corresponding patch for SSIfailover will look like:
                return 0
-       DEVICE=`/sbin/findfs $FSDEV`
+#      DEVICE=`/sbin/findfs $FSDEV`
+        /etc/init.d/drbd start
+       DEVICE=/dev/nbd/0
        cfs_setroot $TYPE $DEVICE
         action $"Enabled root failover: " /bin/true
        return 0

9. In /etc/fstab, replace the UUID=... with /dev/nbd/0
10. Shut down the cluster and boot only the first node.
    The root partition should be picked up by the drbd-enabled ramdisk.
    You are now running with the root as a drbd primary. 
11.Once the first node completes booting up, boot up the second node. 
   When the second node comes up, use command
         drbdsetup /dev/nbd/0 wait_sync
   This should sync node2 with the first node. This might take a little 
   time while the partitions are being synced. It took me about 15 minutes.
         cat /proc/drbd 
   to check progress of the sync. The complete sync will show "ld:Consistent"
   Once the sync is complete, node2 is ready for failover.
12. Now if the first node crashes, the second node should fail over.

Converting an existing openssi-cluster into a drbd-enabled root failover
Converting an existing openssi-cluster into a drbd-enabled root failover
cluster is only possible at the moment if you have an extra partition
that is atleast 128Mb for the index partition. The only alternative
is to use resize2fs to convert an existing root partition so that there
is space for the 128Mb index partition. This as can be guessed is going
to be tricky to do over a running and in-use root filesystem.
If the cluster does have an extra 128 Mb space for the index partition, then
you can convert the existing cluster into a root-failover cluster. And then
start from step 3 above.
There are basically three steps involved in changing a non-failover cluster 
to a failover cluster.
1. Run  ssi-chnode to turn secondary node into a takeover CLMS master
2. Add the chard option to the root line in /etc/fstab
e.g. UUID=$UUID      /       ext3    chard,defaults,node=1:2 1       1
3. Run mkinitrd to create a new mkinitrd
4. Follow How-to to engineer ramdisk to do drbd-failover.
4. Run ssi-ksync to propogate new mkinitrd to all nodes.  

According to my experience, root fail-over through drbd has not always been 
extremely reliable, especially when the cluster had multiple fail-overs. 
In some cases I have had to recover things manually. This might be due to 
the fact that the base-drbd code is still in flux and has not been extensively 
tested in situations that reflect root-like usage.

II. Failover of non-root file systems

1. Download and install the OpenSSI enabled drbd and the patched mount
2. Edit /etc/drbd.conf to indicate the resources that need to
   to be failed over. If drbd is being layered over a device with
   an existing filesystem, use a separate meta-device.
3. Edit /etc/rc.d/ssiconfig and add the line
     drbd	M	all	Y
4. Enable drbd to start up at boot
     chkconfig --add drbd
5. If drbd is being layered over an existing filesystem, skip this step.
   Otherwise create the filesystem.
     insmod drbd
     drbdadm up r1    # where r1 is the name of the resource in drbd.conf
     drbdadm primary r1
     mkfs -t ext3 /dev/nbd/1 # where /dev/nbd/1 is the drbd 
                             # device name in drbd.conf
6. Edit /etc/fstab to enable failover
     /dev/nbd/1 <mntpt>	ext3	default,chard,node=1:2	0	0
   The node= are the numbers of the nodes which are in drbd.conf
7. Now start drbd
     service drbd start
8. Mount the filesystems
     mount -a

SourceForge Logo

HP Linux solutions

The Linux Clustering Information Center

This file last updated on Wednesday, 26-May-2004 23:15:38 UTC
privacy and legal statement