Csdrhrt

I have a RHEL version 6 machine (appliance) configured with a RAID 6 array containing 12 disks, configured for DRBD with an identical appliance in the same physical chassis. This array is partitioned into 9 drives, one of which contains the boot sector; there is no separate local boot disk. The device has a hardware RAID controller; I'm unsure the specific model.

We attempted to hot-swap a disk, and the system went into what I can only assume was a kernel panic. This forced us to restart the machine at bare metal. After quite a bit of tinkering (extensive, but irrelevant to the question), we're able to mount the boot partition, log partition, recovery partition, etc; and all of these work fine.

However, the largest partition, the "/store" partition which contains the data we're trying to gain access to, is still partially corrupted. The disk will mount after an xfs_repair -L, however we can't get to a significant amount of the data, nor can we bring up the appliance due to PGSQL having a number of corrupted directories.

When running xfs_repair, it runs for approximately 1 hour; the most common notifications written to the screen are below, in order from most common to least common. Please note that there are enough of these the speed of scrolling is reminiscent of a script running an infinite loop; I am unable to count how many of each. Additionally, I'm quite sure I've missed some somewhere.

Phase 2:

block (##############) multiply claimed by bno space tree, state - 1

block (##############) multiply claimed by bno space tree, state - 2

block (##############) multiply claimed by cnt space tree, state - 2

block (##############) multiply claimed by cnt space tree, state - 1

inode chunk claims used block, inobt block - agno 24, bno #####, inopb 16

Phase 3:

found inodes not in the inode allocation tree

bad inode format in inode ################

bad magic number #x#### in inode #########, resetting magic number

bad (negative) size -##################### on inode #############

Somewhere in here, it throws the following, and then exits:

xfs_repair: dinode.c:768: process_bmbt_reclist_int: Assertion `i < *numrecs' failed.

Aborted (core dumped)

Does anyone have any idea how I can get the xfs_repair to progress past this point?