« Cowboy | Main | Measles »

July 19, 2005

Insufficient metadevice database replicas ...

One day you reboot your Sun server - the one that has only two internal disks configured as mirrors using SVM (Solaris Volume Manager)* - and it says something like this:

Rebooting with command: boot
Boot device: disk:a File and args:
SunOS Release 5.8 Version Generic_108528-13 32-bit
Copyright 1983-2001 Sun Microsystems, Inc. All rights reserved.
WARNING: md: d20: /dev/dsk/c0t1d0s0 needs maintenance
WARNING: forceload of misc/md_trans failed
WARNING: forceload of misc/md_raid failed
WARNING: forceload of misc/md_hotspares failed
configuring IPv4 interfaces: hme0.
Hostname: foo
WARNING: md: d23: /dev/dsk/c0t1d0s5 needs maintenance
metainit: foo: stale databases
Insufficient metadevice database replicas located.

At this point, your boss will walk in and ask "How's it going?". This is known as SNAFU.** The important point is that you should not panic. Instead, you can boot into single user mode (the messages above will be followed with a 'Type control-d to proceed with normal startup, (or give root password for system maintenance):' and fix the problem.

Run metadb and look for the dead copies:

# metadb
flags first blk block count
a m p luo 16 1034 /dev/dsk/c0t0d0s7
a p luo 1050 1034 /dev/dsk/c0t0d0s7
a p luo 2084 1034 /dev/dsk/c0t0d0s7
W p l 16 1034 /dev/dsk/c0t1d0s7
W p l 1050 1034 /dev/dsk/c0t1d0s7
W p l 2084 1034 /dev/dsk/c0t1d0s7

Note that (in this case) there are three dead copies but they are all on the one disk. Clear all three at once:
# metadb -d c0t1d0s7

You may get a message about something being read-only. Ignore this and reboot the server:

# reboot

The server will come up normally now because there are only three copies left and they all agree (the quorum rule requires more than 50% of the copies to agree).

When the server comes up, start by replacing the dead disk. You will almost certainly need to copy a new partition map onto the disk (see the extended entry for a step by step guide). You can then re-establish the mirrors by checking the advice of the metastat command:

# metastat | grep metareplace
Invoke: metareplace d10 c0t1d0s0
Invoke: metareplace d20 c0t1d0s1
Invoke: metareplace d30 c0t1d0s3
Invoke: metareplace d60 c0t1d0s6

Now, you don't actually want to run these commands as shown. Instead, use the '-e' option to replace the disk with itself:
# metareplace -e d10 c0t1d0s0
# metareplace -e d20 c0t1d0s1
# metareplace -e d30 c0t1d0s3
# metareplace -e d60 c0t1d0s6

Next we need to repair the metadb copies by recreating them:
# metadb -a -c 3 c0t1d0s7

Note that you should recreate the same number of copies as you found previously (three in this case).

Once the system is stable (the resync's have all finished), ytou might want to think about being a little smarter with the quorum rules. In general, you probably want to put metadb copies on at least three disks but if that is not possible, some versions of disksuite (SVM) will honour a special kernel flag:

set md:mirrored_root_flag=1

If you add this line to the system file (before the "MDD root info") line, you may be able to save yourself some hassle next time :-)

[* Previously known as SDS (Solaris Disk Suite).]

[** Situation Normal, All Fouled Up.]

Copying a disk layout

For this example, we have three disks: c0t0d0, c0t1d0 and c0t2d0. We want to copy the layout of c0t0d0 to c0t2d0 (bold is what you type, italics indicate comments):

# format
Searching for disks...done
0. c0t0d0
1. c0t1d0
2. c0t2d0
Specify disk (enter its number): 0
selecting c0t0d0
[disk formatted]
format> p p for partition
partition> p p for print
partition> n n for name
Enter table name (remember quotes): foo
partition> q q for quit (partition sub-menu)
format> di di for disk (d by itself is ambiguous)
Specify disk (enter its number)[0]: 2
format> p p for partition
partition> s s for select
0. foo
1. unnamed
Specify table (enter its number)[0]: 0
partition> l l for label
Ready to label disk, continue? y
partition> q q for quit (partition sub-menu)
format> q q for quit (from the format program)

At this point the two disks will be laid out the same way. Note that if you cannot see 'foo' in the list of partition tables, it means that the two disks have different sizes and therefore you cannot copy the partition map.

Posted by Ozguru at July 19, 2005 06:00 AM


your whole copy disk VTOC example via "format" can be replaced with a single line command:-

# prtvtoc /dev/dsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c5t0d0s2

the above example copies the VTOC from disk c0t0d0 to disk c5t0d0

Posted by: Oscar at August 2, 2005 06:40 PM

Thanks Oscar. Your comments have been added to this post: Copying Disk Layouts (Solaris).

Posted by: Ozguru at August 6, 2005 12:23 AM