G'day Mate (Mk III)

« Cowboy | Main | Measles »

July 19, 2005

Insufficient metadevice database replicas ...

One day you reboot your Sun server - the one that has only two internal disks configured as mirrors using SVM (Solaris Volume Manager)* - and it says something like this:

Rebooting with command: boot
Boot device: disk:a File and args:
SunOS Release 5.8 Version Generic_108528-13 32-bit
Copyright 1983-2001 Sun Microsystems, Inc. All rights reserved.
WARNING: md: d20: /dev/dsk/c0t1d0s0 needs maintenance
WARNING: forceload of misc/md_trans failed
WARNING: forceload of misc/md_raid failed
WARNING: forceload of misc/md_hotspares failed
configuring IPv4 interfaces: hme0.
Hostname: foo
WARNING: md: d23: /dev/dsk/c0t1d0s5 needs maintenance
metainit: foo: stale databases
Insufficient metadevice database replicas located.

At this point, your boss will walk in and ask "How's it going?". This is known as SNAFU.** The important point is that you should not panic. Instead, you can boot into single user mode (the messages above will be followed with a 'Type control-d to proceed with normal startup, (or give root password for system maintenance):' and fix the problem.

Run metadb and look for the dead copies:

# metadb
flags first blk block count
a m p luo 16 1034 /dev/dsk/c0t0d0s7
a p luo 1050 1034 /dev/dsk/c0t0d0s7
a p luo 2084 1034 /dev/dsk/c0t0d0s7
W p l 16 1034 /dev/dsk/c0t1d0s7
W p l 1050 1034 /dev/dsk/c0t1d0s7
W p l 2084 1034 /dev/dsk/c0t1d0s7


Note that (in this case) there are three dead copies but they are all on the one disk. Clear all three at once:

# metadb -d c0t1d0s7

You may get a message about something being read-only. Ignore this and reboot the server:

# reboot

The server will come up normally now because there are only three copies left and they all agree (the quorum rule requires more than 50% of the copies to agree).

When the server comes up, start by replacing the dead disk. You will almost certainly need to copy a new partition map onto the disk (see the extended entry for a step by step guide). You can then re-establish the mirrors by checking the advice of the metastat command:

# metastat | grep metareplace
Invoke: metareplace d10 c0t1d0s0
Invoke: metareplace d20 c0t1d0s1
Invoke: metareplace d30 c0t1d0s3
Invoke: metareplace d60 c0t1d0s6


Now, you don't actually want to run these commands as shown. Instead, use the '-e' option to replace the disk with itself:

# metareplace -e d10 c0t1d0s0
# metareplace -e d20 c0t1d0s1
# metareplace -e d30 c0t1d0s3
# metareplace -e d60 c0t1d0s6

Next we need to repair the metadb copies by recreating them:

# metadb -a -c 3 c0t1d0s7

Note that you should recreate the same number of copies as you found previously (three in this case).

Once the system is stable (the resync's have all finished), ytou might want to think about being a little smarter with the quorum rules. In general, you probably want to put metadb copies on at least three disks but if that is not possible, some versions of disksuite (SVM) will honour a special kernel flag:

set md:mirrored_root_flag=1

If you add this line to the system file (before the "MDD root info") line, you may be able to save yourself some hassle next time :-)

[* Previously known as SDS (Solaris Disk Suite).]

[** Situation Normal, All Fouled Up.]

Posted by Ozguru at July 19, 2005 06:00 AM

Comments

Posted by: Oscar at August 2, 2005 06:40 PM

Posted by: Ozguru at August 6, 2005 12:23 AM