Hi,
I’m Sina Moghaddas, and this my first post as a co-author to my brother ?
In this post, I am going to show you how to setup DRBD with OCFS2.
DRBD® refers to block devices designed as a building block to form highly available (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as a network based raid-1.
OCFS2 is a general-purpose, shared-disk, clustered filesystem for Linux, which is capable of providing both high performance and high availability.
ok, here we go ?
I setup the scenario on two Debian 8.2 (3.16.0) machines with the following details:
Host 1
- Hostname: d-2-t-1
- IP: 192.168.100.110
Host 2
- Hostname: d-2-t-1
- IP: 192.168.100.120
Installation
# on both nodes apt-get install -y drbd8-utils # to simulate a block device dd if=/dev/zero of=/root/drbd.testimage bs=1024k count=1024 losetup /dev/loop1 /root/drbd.testimage
Let’s verify hosts on both nodes:
cat /etc/hosts # ... # 192.168.100.110 d-2-t-1 # 192.168.100.120 d-2-t-2
I use DRBD with 2 different methods.
Method 1: Primary/Secondary
# on both nodes cat > /etc/drbd.d/simo.res << EOF resource drbdsimo { meta-disk internal; device /dev/drbd1; syncer { verify-alg sha1; } net { allow-two-primaries; } # d-2-t-1 -> hostname !!!!!! on d-2-t-1 { disk /dev/loop1; address 192.168.100.110:7789; } # d-2-t-2 -> hostname !!!!!! on d-2-t-2 { disk /dev/loop1; address 192.168.100.120:7789; } } EOF
Now, we have to create DRBD and start it:
# on both nodes drbdadm create-md drbdsimo modprobe drbd drbdadm up drbdsimo
And specify the Primary node:
# on Node 1 -> make it Primary drbdadm -- --overwrite-data-of-peer primary drbdsimo
Now let’s check the status:
# it can check both nodes cat /proc/drbd # version: 8.4.3 (api:1/proto:86-101) # srcversion: 1A9F77B1CA5FF92235C2213 # # 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- # ns:90396 nr:0 dw:0 dr:91308 al:0 bm:5 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:958112 # [>...................] sync'ed: 9.0% (958112/1048508)K # finish: 0:02:38 speed: 6,024 (6,024) K/sec
I have the block, so let’s create the filesystem and mount it somewhere:
# on Node 1 mkfs.xfs /dev/drbd1 mount /dev/drbd1 /mnt
Keep in mind that you can’t mount it on two nodes.
Now I wanna change the Primary node:
# on node 1 (primary node) umount /dev/drbd1 drbdadm secondary drbdsimo # on node 2 (secondary) drbdadm primary drbdsimo
Method 2: Dual-primary DRBD with OCFS2
This should be implemented only with a clustered filesystem. If you do this with a non-clustered filesystem like ext2/ext3/ext4 or reiserfs, you will have a huge data corruption. Seriously!
In this method, you can mount the block on both nodes with the edit option.
Firstly, I make a script for my simulated file:
# on both Nodes echo > /etc/init.d/loop-for-drbd << EOF #!/bin/sh # # Startup script for drbd loop device setup # # chkconfig: 2345 50 50 # description: Startup script for drbd loop device setup # ### BEGIN INIT INFO # Provides: drbdloop # Required-Start: # Required-Stop: # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: set up drbd loop devices # Description: Startup script for drbd loop device setup ### END INIT INFO DRBD_FILEDATA_SRC="/root/drbd.testimage" DRBD_FILEDATA_DEVICE="/dev/loop1" LOSETUP_CMD=/sbin/losetup # Source function library . /etc/rc.d/init.d/functions start () { echo -n $"Setting up DRBD loop devices..." $LOSETUP_CMD $DRBD_FILEDATA_DEVICE $DRBD_FILEDATA_SRC echo } stop() { echo -n $"Tearing down DRBD loop devices..." $LOSETUP_CMD -d $DRBD_FILEDATA_DEVICE echo } restart() { stop start } case "$1" in start) start RETVAL=$? ;; stop) stop RETVAL=$? ;; restart) restart RETVAL=$? ;; *) echo $"Usage: $0 {start|stop}" exit 1 esac exit $RETVAL EOF chmod a+x /etc/init.d/loop-for-drbd
# on both Nodes cat > /etc/drbd.d/simo.res << EOF resource drbdsimo { meta-disk internal; device /dev/drbd1; disk /dev/loop1; syncer { rate 1000M; } net { # !!!! Do this only with a clustered filesystem. # !!!! If you do this with a non-clustered filesystem like ext2/ext3/ext4 or reiserfs, # !!!! you will have data corruption. Seriously! allow-two-primaries; # newest modifications and apply them to the node that didn’t have any changes after-sb-0pri discard-zero-changes; # decide that the secondary node is the victim and it will sync data from # the primary to the secondary automatically after-sb-1pri discard-secondary; # It tries to protect the consistency of both nodes by disconnecting the DRBD volume entirely. # You’ll have to tell DRBD which node has the valid data in order to reconnect the volume. # !!!! Use extreme caution if you find yourself in this scenario after-sb-2pri disconnect; } startup { become-primary-on both; } on d-2-t-1 { address 192.168.100.110:7789; } on d-2-t-2 { address 192.168.100.120:7789; } } EOF
Creating DRBD, staring and then verifying it:
drbdadm create-md drbdsimo modprobe drbd drbdadm up drbdsimo # on Node 1 drbdadm -- --overwrite-data-of-peer primary drbdsimo # on Node 2 cat /proc/drbd #version: 8.4.3 (api:1/proto:86-101) #srcversion: 1A9F77B1CA5FF92235C2213 # # 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----- # ns:0 nr:289116 dw:289116 dr:0 al:0 bm:17 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:759392 # [====>...............] sync'ed: 27.8% (759392/1048508)K # finish: 0:00:53 speed: 14,252 (10,708) want: 21,280 K/sec drbdadm primary drbdsimo
With the last command, I made the second node to be Primary.
Now I have a Primary/Primary DRBD!
Right now, I want to setup my clustered filesystem.
Cluster File System
After some researches, I found out that OCFS2 has a really amazing performance.
So, let’s set it up and make the configuration ?
# on Both nodes apt-get install -y ocfs2-tools cat > /etc/ocfs2/cluster.conf << EOF cluster: node_count = 2 name = web node: ip_port = 7777 ip_address = 192.168.100.110 number = 1 name = d-2-t-1 cluster = web node: ip_port = 7777 ip_address = 192.168.100.120 number = 2 name = d-2-t-2 cluster = web EOF dpkg-reconfigure ocfs2-tools # Answer "y" to "Load O2CB driver on boot" # Answer "web" to "Cluster to start on boot" # Enter for Other
Then we have to start and enable it on both nodes:
/etc/init.d/o2cb start && /etc/init.d/ocfs2 start /etc/init.d/o2cb enable
Making the filesystem:
# on Node 1 mkfs.ocfs2 -L "web" /dev/drbd1
And making a mount-point and adding it to startup, and then mount it:
mkdir /mnt/storage echo "/dev/drbd1 /mnt/storage ocfs2 noauto,noatime 0 0" >> /etc/fstab mount /dev/drbd1
Done ?
I hope this has been informative and you have enjoyed the setup!
great!! Thank you!!
I’m glad that it’s been helpful to you. 🙂
/Mohammad
Thanks you for it.
If you add a new disk on each node for drbd, don’t forget to use fdisk on it 😀
I still have a problem when rebooting : even if I wrote new entries in my fstabs, I need to mount drbd manually. Luckily, I ‘m rarely rebooting my servers.