Hi
I've been following this thread for a while - just thought I'd add some of my own experiences here.
Our setup here is similar to what you describe - we've got a StoreOnce software device that's been created on RHEL 6.4, running on an XFS filesystem. The XFS filesystems have been created on LUNs presented over a FC SAN.
For various reasons (which we've now resolved) we had a few issues where housekeeping activities on the underlying storage devices caused I/O pauses, which then caused some of the OS filesystems to become read-only, and the only way out of this was a hard reboot of the server.
This clearly caused significant issues with the StoreOnce device, similar to that which you have described above.
I've been using the recovery and verification methods that have been posted here along with instructions from Support with some success, but like you I'm a little concerned about the overall integrity of the data store. It seems that the only way to completely verify the StoreOnce datastore is to verify the media individually - I've got between 70 and 100TB of data in the store - you can imagine how long this is going to take!
What I really would like is a way to verify the datastore directly, rather than via the individual media. Verifying this way would mean checking between 8 and 14 TB rather than the full data set.
The other thing I'd like to see is that if there *is* a corrupt block, it's flagged, and the next time a backup is taken, the block gets rewritten. There's no reason to completely discard ~20 copies of a backup if you still have a valid source copy you can use...
I've also got a few operational issues - the fact that they put configuration data inside the store right alongside the user data, and the lack of good, consistent logging from the StoreOnce processes, but those are issues for another day...