openindiana iscsi mirror local & remote

I have two servers, each with 6 disks. The first 2 disks of each are configured as OS mirror. This leaves 4 disks unused. I want each server to have a pool, 2 local disks mirrored against 2 remote disks. This wasn’t as easy as it sounds, so here’s a process that works. I have only done this on OI 151a6. I don’t know what will happen on solaris 10, opensolaris, solaris 11, or anything else.

Edit: This configuration turned out to be unstable. In the ensuing months, the servers would randomly crash in ways that made me suspect IO errors, but couldn’t really prove it. We eliminated the local-to-remote mirror iscsi configuration, and things became stable, but not high availability.

First, set up iscsi.

Now, each system has unused: c4t2d0 c4t3d0 c4t4d0 c4t5d0
Each system will export via iscsi, all four disks: c4t2d0 c4t3d0 c4t4d0 c4t5d0
Each system will initiate a connection to all eight disks, including the ones at 127.0.0.1

When creating zpool, don’t use the local device name (c4t2d0 or whatever) because that’s not available to the other system. Also, if you use the local device name instead of the multipath device name, it seems to cause data corruption on that device, as perceived by the other system using the multipath name.

Use only the multipath iscsi target device names.

Do this on both systems, to enable targets:

sudo pkg install pkg:/network/iscsi/target
sudo svcadm enable -s svc:/system/stmf
sudo svcadm enable -s svc:/network/iscsi/target

Do this on both systems, to enable initiators:

sudo iscsiadm modify discovery --static enable

Note: I’m skipping the chap bidirectional authentication in this example. You probably want to figure that out.

On both machines:

for DISKNUM in 2 3 4 5 ; do sudo sbdadm create-lu /dev/rdsk/c4t${DISKNUM}d0 ; done
for GUID in `sudo sbdadm list-lu | grep rdsk | sed 's/ .*//'` ; do sudo stmfadm add-view $GUID ; done
sudo itadm create-target

Now set up the initiator on host1:

sudo iscsiadm add static-config iqn.2010-09.org.openindiana:xxx,127.0.0.1
sudo format -e

Make a note of the new device names. And hit Ctrl-C.
Now you know which disks are on host1. Make a note, as below, “export H1T0=…”

sudo iscsiadm add static-config iqn.2010-09.org.openindiana:xxx,192.168.7.7
sudo format -e

Make a note of the new device names. And hit Ctrl-C.
Now you know which disks are on host2. Make a note, as below, “export H2T0=…”

Now set up the initiator on host2:

sudo iscsiadm add static-config iqn.2010-09.org.openindiana:yyy,192.168.7.8
sudo iscsiadm add static-config iqn.2010-09.org.openindiana:yyy,127.0.0.1

sudo format -e

Make a note of the new device names. And hit Ctrl-C

Create the following information:

export H1T0=c5t6540CB9496F540CB9496F540CB9496F4d0
export H1T1=c5t6540CB9496F540CB9496F540CB9496F1d0
export H1T2=c5t6540CB9496F540CB9496F540CB9496F2d0
export H1T3=c5t6540CB9496F540CB9496F540CB9496F3d0

export H2T0=c5t601C57BADF301C57BADF301C57BADF34d0
export H2T1=c5t601C57BADF301C57BADF301C57BADF31d0
export H2T2=c5t601C57BADF301C57BADF301C57BADF32d0
export H2T3=c5t601C57BADF301C57BADF301C57BADF33d0

Only do this on one server:

sudo zpool create iscsiTank1 mirror $H1T0 $H2T0 mirror $H1T1 $H2T1
sudo zpool create iscsiTank2 mirror $H1T2 $H2T2 mirror $H1T3 $H2T3

Now, if you want to pass the pool over the other system, just export & import.

Ok, so.
New problem.
Natural state of problem, by default:

If I don’t export the pool before rebooting, then either the iscsi target or initiator is shutdown before the filesystems are unmounted. So the system spews all sorts of error messages while trying to go down, but it eventually succeeds. It’s somewhat important to know if it was the target or initiator that went down first – If it was the target, then only the local disks became inaccessible, but if it was the initiator, then both the local and remote disks became inaccessible, and will result in data loss.

Upon reboot, the pool fails to import, so the svc:/system/filesystem/local service fails, and of course all the other services depend on it. System comes up in maintenance mode. The whole world is a mess, you have to login at physical text console to export the pool, and reboot. But it comes up cleanly the second time.

Solve the problem:

Fetch files:

iscsi-pool-ctrl.conf
iscsi-pool-ctrl.sh
iscsi-pool-ctrl.xml

sudo mkdir /root/bin
sudo cp iscsi-pool-ctrl.sh /root/bin
sudo chmod +x /root/bin/iscsi-pool-ctrl.sh
sudo cp iscsi-pool-ctrl.conf /etc/iscsi-pool-ctrl.conf

Edit /etc/iscsi-pool-ctrl.conf

Test it by manually running some commands like:

sudo /root/bin/iscsi-pool-ctrl.sh import
sudo /root/bin/iscsi-pool-ctrl.sh import
sudo /root/bin/iscsi-pool-ctrl.sh export

The second import shows what behavior will be like upon svcadm refresh or svcadm restart

sudo svccfg import iscsi-pool-ctrl.xml
sudo svcadm enable svc:/network/iscsi/pool:default

If you are using iscsi to connect to localhost, make the initiator dependent on the target as follows:
(Otherwise, during reboots, target will die before initiator, which is bad news.)

sudo svccfg -s svc:/network/iscsi/initiator:default
svc:/network/iscsi/initiator:default> addpg iscsi-target dependency
svc:/network/iscsi/initiator:default> setprop iscsi-target/grouping = astring: "require_all"
svc:/network/iscsi/initiator:default> setprop iscsi-target/restart_on = astring: "none"
svc:/network/iscsi/initiator:default> setprop iscsi-target/type = astring: "service"
svc:/network/iscsi/initiator:default> setprop iscsi-target/entities = fmri: "svc:/network/iscsi/target:default"
svc:/network/iscsi/initiator:default> exit

sudo svcadm refresh svc:/network/iscsi/initiator:default

Comments are closed.